Sr. Azure Data engineer

Job Title- Sr. Azure Data engineer

Location - Remote- US or Canada

Technical Skills- DataBricks, Unity Catalog, Hive -- is mandatory .

MUST HAVE - 5 + years of experience.

Role Summary

We are looking for a
hands-on Data Engineer
with strong experience in
Azure Databricks
to support the migration from
Hive Metastore to Unity Catalog
. The role focuses on
remediating Databricks notebooks
, validating
Unity Catalog–compliant access to Azure Data Lake (ADLS Gen2)
, and performing
end-to-end testing
after migration.

The engineer will also work closely with
Azure Data Factory (ADF)
pipelines used for orchestration, ensuring seamless execution post‑migration and minimal disruption during cutover.

Key Responsibilities

Unity Catalog Migration (Azure Databricks)

- Support migration from
Hive Metastore to Unity Catalog
in
Azure Databricks
.
- Validate and work with Unity Catalog objects:
- Catalogs, schemas, managed & external tables
- External locations and volumes backed by
ADLS Gen2
- Ensure compliance with Unity Catalog governance and access controls (RBAC, ACLs).

Notebook Remediation & Code Changes (Primary)

- Update Databricks notebooks to be Unity Catalog–compliant:
- Replace legacy database references with
3‑level namespace
:
- catalog.schema.table
- Refactor Spark SQL and PySpark code impacted by UC migration.
- Update storage access patterns:
- Replace legacy mount-based paths with
Unity Catalog external locations or volumes
.
- Ensure secure access to
Azure Data Lake (ADLS Gen2)
using managed identities / service principals.
- Parameterize hardcoded paths, schemas, and table references where required.
- Resolve permission-related issues caused by UC enforcement.

ADF Orchestration Validation

- Validate and support
Azure Data Factory (ADF)
pipelines that orchestrate Databricks notebooks and jobs.
- Ensure:
- Correct notebook paths and parameters post-migration
- Service principal permissions align with Unity Catalog policies
- No regression in scheduled and triggered executions
- Troubleshoot end-to-end pipeline failures spanning
ADF → Databricks → ADLS
.

Testing & Post-Migration Validation (Primary)

- Execute comprehensive testing after migration:
- Data validation & reconciliation
(row counts, aggregates, sample checks)
- Functional testing
of notebooks and Databricks jobs
- ADF pipeline execution validation
- Security & access testing
(users, groups, service principals)
- Identify and resolve performance regressions or permission-related failures.
- Support production cutover and
hypercare
activities.

okay, let me check with a few and get back

awesome

thank youuuuu

Thank You!