Databricks - Lead Data Engineer

We Are Hiring: Databricks Lead Data Engineer – Director Equivalent Role

Location:
Atlanta, USA

Work Model:
Hybrid –
3 to 4 days in office per week (mandatory)

Eligibility:
US Citizens and Green Card (GC) holders only

How to Apply

If you are interested in this position and have the required skills, please send across your resume at:

**contact@pavestechnologies.com
; hr@pavestechnologies.com ;
Rakesh.k@pavestechnologies.com**

Paves Technologies is seeking a highly experienced
Databricks Lead Data Engineer – Lead Level (Director Equivalent Role)
to drive enterprise-scale data architecture, governance, and advanced analytics initiatives on Azure Cloud. This is a senior leadership role requiring deep Databricks expertise, strong data modeling capabilities, and hands-on architectural ownership across PySpark based distributed systems.

Role Overview

The ideal candidate will bring
10-12 + years of overall data engineering experience
, including strong hands-on expertise with
Azure Databricks, PySpark, Python, and Azure Cloud data services
. You will define architecture standards, lead modernization initiatives, and implement scalable
Medallion Architecture (Bronze, Silver, Gold layers)
to support enterprise analytics and business intelligence.

Key Responsibilities

- Lead end-to-end architecture and implementation of enterprise-scale data platforms using
Azure Databricks on Azure Cloud
.
- Design and implement
Medallion Architecture (Bronze, Silver, Gold layers)
using Delta Lake best practices.
- Build scalable
PySpark-based ETL/ELT pipelines
across ingestion (Bronze), transformation (Silver), and curated analytics (Gold) layers.
- Develop advanced data transformations using
Python, PySpark, Spark SQL, and advanced SQL constructs
.
- Architect robust
data models (dimensional, star schema, normalized models)
aligned to analytics and reporting needs.
- Drive adoption of advanced Databricks capabilities including
Unity Catalog, Declarative Pipelines, Delta Lake optimization, and governance frameworks
.
- Establish best practices for
partitioning strategies, file compaction, Z-ordering, caching, broadcast joins, and query optimization
.
- Define and standardize reusable
Azure Cloud data platform tools, templates, CI/CD frameworks, and infrastructure automation
.
- Work across Azure ecosystem components such as
Azure Data Factory (ADF), Azure Data Lake Storage (ADLS), Azure DevOps, networking, and security services
.
- Ensure high standards for
data quality, RBAC, lineage tracking, governance, and production stability
.
- Provide architectural leadership and mentorship to data engineering teams.

Required Experience & Skills

- 10–12+ years of overall experience
in Data Engineering.
- Minimum
3+ years of strong hands-on Databricks experience
.
- Mandatory Certifications:
- Databricks Certified Data Engineer Associate
- Databricks Certified Data Engineer Professional
- Deep hands-on expertise in
PySpark, Python programming, and distributed Spark processing
.
- Strong experience designing and implementing
Medallion Architecture (Bronze/Silver/Gold layers)
.
- Advanced knowledge of
Data Modeling, Data Analysis, and complex SQL (window functions, CTEs, execution plan tuning)
.
- Strong understanding of
Delta Lake architecture, schema evolution, partition strategies, performance optimization, and data governance
.
- Well-versed in enterprise
Azure Cloud data platforms, reusable accelerators, CI/CD templates, and governance standards
.
- Proven experience architecting scalable, secure, cloud-native data solutions.
- Strong leadership, stakeholder management, and executive communication skills.

How to Apply

If you are interested in this position and have the required skills, please send across your resume at:

contact@pavestechnologies.com
;
**hr@pavestechnologies.com ;
Rakesh.k@pavestechnologies.com**