Pyspark Data Engineer with Databricks

Position Title : Pyspark Data Engineer with Databricks

Location : New York, NY (Onsite/Hybrid)

Experience : 8+ Years

Employee Type : Full Time with Benefits

Note :- Must be comfortable to attend In Person Interview at New York Location

Job Description

We are looking for a hands-on mid–senior level
PySpark Data Engineer with Databricks
who can design, build, and own production-grade data pipelines and platform components. This role requires strong expertise in Python/PySpark, Databricks, and Snowflake, with a focus on building scalable, cost‑efficient, and reliable data systems that support both analytics and machine learning use cases.

Key Responsibilities

- Design, develop, and maintain
end‑to‑end ETL/ELT pipelines using Python
and
PySpark on Databricks
.
- Optimize
Spark jobs for performance, scalability, and cost-efficiency
in production environments.
- Implement
data quality frameworks
including validation, reconciliation, and anomaly detection.
- Build and manage
orchestration workflows
(Airflow / Databricks Workflows / equivalent).
- Implement
pipeline monitoring, logging, alerting, and observability
for reliable operations.
- Develop and operationalize
ML workflows using MLflow
(experiment tracking, model registry, packaging, deployment).
- Build scalable
data ingestion and data modeling solutions
for analytics and ML use cases.
- Collaborate with data scientists, platform teams, engineering stakeholders, and business partners.

Required Skills & Qualifications

- 8+ years of experience in
data engineering
with strong hands‑on work in
PySpark and Python
.
- Deep experience with
Databricks
, Spark optimization, cluster tuning, and performance troubleshooting.
- Strong experience working with
Snowflake
or similar cloud data warehouses.
- Practical knowledge of
workflow orchestration tools
and dependency management.
- Solid understanding of
data modeling
, ingestion frameworks, and distributed systems architecture.
- Hands‑on experience implementing
CI/CD
for data and ML pipelines.
- Strong experience with
MLflow
for managing the ML lifecycle.
- Excellent communication skills with the ability to work across engineering and business teams. Desired Skills

Nice-to-Have Skills

- Exposure to
AI/LLM use cases
, vector search, or RAG pipelines.
- Familiarity with
Java-based services
or microservices architecture.
- Knowledge of data governance, cataloging, and security practices.