Data Engineering Lead-7
Role summary
Seeking a Data Engineering Lead with 4-15 years of experience to define data architecture, build and operate robust data pipelines on cloud platforms (AWS, Azure, GCP), and ensure data quality and governance. The role requires strong skills in Python, SQL, Spark (PySpark), and tools like Prefect/Airflow, dbt, Snowflake/Databricks, and AWS data services. Responsibilities include data modeling, pipeline design, performance optimization, security enforcement, and enabling AI/LLM use cases. The lead will also provide strategic technical leadership, manage stakeholder communication, identify risks, and drive operational excellence through automation and best practices. Experience with CI/CD and containerization is essential.
New York, New York 10001 Posted March 29th, 2026
Looking for more job opportunities? Click here!
Job Type: Full Time
Job Category: IT
Job Description
Role : Data Engineering Lead
Location: New York, NY
FTE ONLY
Job Description
Must Have Technical/Functional Skills
AWS Data Engineering Services (EMR/Glue, Redshift, Aurora, S3, Lambda), Spark, Python, Collibra, Snowflake/Databricks, Tableau.
Roles & Responsibilities
Ingest and model data from APIs, files/SFTP, and relational sources; implement layered architectures (raw/clean/serving) using PySpark/SQL and dbt, Python.
Design and operate pipelines with Prefect (or Airflow), including scheduling, retries, parameterization, SLAs, and well documented runbooks.
Build on cloud data platforms, leveraging S3/ADLS/GCS for storage and a Spark platform (e.g., Databricks or equivalent) for compute; manage jobs, secrets, and access.
Publish governed data services and manage their lifecycle with Azure API Management (APIM) authentication/authorization, policies, versioning, quotas, and monitoring.
Enforce data quality and governance through data contracts, validations/tests, lineage, observability, and proactive alerting.
Optimize performance and cost via partitioning, clustering, query tuning, job sizing, and workload management.
Uphold security and compliance (e.g., PII handling, encryption, masking) in line with firm standards.
Collaborate with stakeholders (analytics, AI engineering, and business teams) to translate requirements into reliable, production ready datasets.
Enable AI/LLM use cases by packaging datasets and metadata for downstream consumption, integrating via Model Context Protocol (MCP) where appropriate.
Continuously improve platform reliability and developer productivity by automating routine tasks, reducing technical debt, and maintaining clear documentation.
4–15 years of professional data engineering experience.
Strong Python, SQL, and Spark (PySpark) skills, and/or Kafka.
Snowflake (Snowpipe, Tasks, Streams) as a complementary warehouse.
Databricks (Delta formats, workflows, cataloging) or equivalent Spark platforms.
Hands-on experience building ETL/ELT with Prefect (or Airflow), dbt, Spark, and/or Kafka.
Experience onboarding datasets to cloud data platforms (storage, compute, security, governance).
Familiarity with Azure/AWS/GCP data services (e.g., S3/ADLS/GCS; Redshift/BigQuery; Glue/ADF).
Git-based workflows CI/CD and containerization with Docker (Kubernetes a plus).
Generic Managerial Skills, If any
Strategic Technical Leadership: Defining data architecture, evaluating new technologies, and setting technical standards for AWS-based pipelines
Stakeholder Communication: Bridging the gap between technical teams and business stakeholders, gathering requirements, and reporting progress
Risk Management: Proactively identifying potential bottlenecks in data workflows, security risks, or scalability issues
Operational Excellence: Implementing automation, optimizing costs, and maintaining high data quality standards.
Required Skills
DEVOPS ENGINEER
SENIOR EMAIL SECURITY ENGINEER