Senior Databricks Data Engineer

Senior Databricks Data Engineer

Toronto ON (3 to 4 Days Onsite)

Contract Role

Very strong Databricks.

Data transfer and ETL tools: Python, PySpark, Databricks, SQL, and relevant libraries for large-scale data processing

Familiarity with networking concepts, data storage interfaces, and scoping ETL requirements

Strong SQL proficiency (SQL “flavor” largely unimportant)

Understanding of underlying data models and optimization strategies (partitioning, indexing, query performance)

Ability to create and maintain ETL pipelines and workflows in Databricks; ensure source data is received on schedule and transformed data is available for Power BI and Tableau consumption

Experience with API ingestion, including connecting to external APIs, extracting data, and testing/validating for completeness and accuracy

Experience with Delta Sharing (Deltashare) for secure, real-time sharing of data across systems and teams

Familiarity with AWS cloud services, including S3 for data storage and CloudWatch for monitoring ETL pipelines and workflows

Good understanding of system Entities and Attributes, with ability to implement structural changes in the Data Model as requirements evolve

Knowledge of data replication and Change Data Capture (CDC) processes, including tools like Qlik