Principal Platform Engineer — Data Private Cloud (Kubernetes/OpenShift)
Role summary
This Principal Platform Engineer role focuses on architecting, engineering, and enabling large-scale, multi-tenant enterprise data platforms. The ideal candidate will have extensive experience building and operating data lakes, lakehouses, or data warehouses that support analytics, data engineering, AI/ML, and regulatory workloads for thousands of users. The role emphasizes platform architecture, scalability, security, and operational excellence, rather than application or pipeline development. Key responsibilities include designing secure-by-design platforms, implementing multi-tenant isolation, capacity planning, and leveraging platform engineering principles with tools like Terraform and GitOps. A strong background in Kubernetes (5+ years) and 7+ years of overall engineering experience is required.
Principal Engineer –Data Platforms (Enterprise Data Platforms, Lakehouse, Multi‑Tenant Architectures)
Core Role Expectation
This role is a hands-on Principal Engineer responsible for architecting, engineering, and enablement of large-scale, multi-tenant enterprise data platforms. The ideal candidate has deep experience building enterprise data lakes, lakehouses, or data warehouses that support data analytics, data engineering, AI/ML, and regulatory workloads across hundreds or thousands of users.
This isnot an application or pipeline-only role. The focus is on platform architecture, scalability, security, and operational excellence for shared enterprise data platforms.
Required Technical & Leadership Skillset
Enterprise Data Platform Architecture & Engineering
- Extensive experience designing, engineering, and operating enterprise-scale data platforms, including data lakes, lakehouses, or data warehouses
- Proven experience leading large, multi-tenant data platforms serving multiple lines of business with strict isolation, governance, and performance controls
- Deep understanding of data platform reference architectures, including:
- Lakehouse patterns (compute/storage separation, open table formats)
- Shared services vs. tenant-owned workloads
- Platform-as-a-product operating models
- Demonstrated ownership of end-to-end platform lifecycle: architecture, build, migration, operations, and modernization
Multi-Tenancy, Scale & Performance
- Hands-on experience designing and enforcing multi-tenant isolation.
- Expertise in capacity planning, workload isolation, quota management, and performance optimization at enterprise scale
- Experience supporting mixed workloads (batch, interactive SQL, streaming, ML/AI) on shared platforms
Data Platform Technologies (Hands-On)
- Strong hands-on expertise with modern data platform ecosystems, such as:
- Compute & Processing: Spark (including Spark at scale), distributed processing frameworks
- Query & Analytics: Trino/Presto or similar distributed SQL engines
- Table Formats & Storage: Iceberg (or similar), Iceberg Rest Catalogue, object storageand enterprise storage platforms
- Metadata, Catalog & Governance: DataHub, Apache Atlas, Hive Metastore, or equivalent
- Experience designing and operating production-grade data services, not just proof-of-concepts
Platform Engineering & Automation
- Strong background in platform engineering principles applied to data platforms:
- Infrastructure as Code (Terraform or equivalent)
- Automated environment provisioning and repeatability
- GitOps or declarative deployment models
- Experience standardizing and industrializing data platforms to support self-service consumption at scale
Security, Governance & Compliance
- Demonstrated experience building secure-by-design data platforms in regulated environments
- Hands-on knowledge of:
- Authentication and authorization models (enterprise IAM integration)
- Fine-grained access controls and data entitlements
- Auditability, lineage, and compliance controls
- Proven ability to partner with Security, Risk, Compliance, and Audit teams to meet regulatory requirements (e.g., SOX, PCI, data privacy)
Technical Leadership & Influence
- Recognized technical leader capable of:
- Setting data platform strategy and standards across the enterprise
- Making architecture decisions that balance scalability, cost, risk, and time-to-market
- Mentoring senior engineers and influencing platform adoption across teams
- Experience leading complex platform migrations or modernizations (e.g., legacy data platforms to modern lakehouse architectures)
Data Platform Components (Platform Enablement)
You provide leadership for the platform that runs these technologies, not the pipelines or applications built on them:
- Compute: Spark on K8s, Kyuubi, JupyterHub
- Query/Analytics: Trino, Superset
- Orchestration: Airflow on Kubernetes
- Catalog/Governance: Gravitino, DataHub, Ranger
- Storage: Iceberg, S3/NetApp, PostgreSQL
- Messaging/Search: Kafka, OpenSearch
Required Qualifications
- 7+ years of Engineering experience, or equivalent demonstrated through one or a combination of the following: work experience, training, military experience, education
- 5+ years of hands-on experience with Kubernetes in production environments (OpenShift Container Platform strongly preferred)
- Proven track record designing and operating large-scale data platforms in enterprise environments
Preferred / Differentiating Experience
- Experience in financial services or other highly regulated enterprises
- Prior ownership of enterprise data platform transformations
- Contributions to open-source data or platform ecosystems
- Background in platform product thinking or developer experience for data platforms
- Experience supporting AI/ML workloads on shared enterprise data platforms