Data Engineer

Description:

Position Summary

We are looking for a Data Engineer to build the future of our data ecosystem. As we move from traditional SQL databases to a modern Databricks Lakehouse, you will be responsible for designing and implementing the pipelines that power our credit union’s decision-making ecosystem. You will bridge the gap between our data systems and our business stakeholders to ensure our data is governed, accessible, and actionable.

Key Responsibilities

• Cloud Data Transformation: Lead the design and implementation of end-to-end data pipelines, transitioning legacy on-premise SQL workloads into a scalable cloud environment.
• Lakehouse Architecture & Modeling: Execute a Medallion Architecture to transform raw data into curated datasets for analytics and AI use. You will build the foundational models and establish a comprehensive data dictionary to ensure transparency and adoption by business users.
• Reliable & Observable Pipelines: Deliver robust data. You will implement proactive monitoring and standardized pipeline patterns to ensure maximum stability.
• Data Quality & Compliance: Architect and manage data governance to enforce data security, PII masking, and regulatory compliance. You will establish automated data quality with automated alerting on failures.
• Cross-Functional Support: Partner with teams across the organization to translate business requirements into technical models, occasionally assisting with analytics and reports to meet immediate organizational needs.

Requirements:

Qualifications

• Databricks Mastery: Deep, hands-on experience with Delta Lake, PySpark, and Spark SQL.
• Modern Data Transport: Proficiency in handling various protocols, including REST APIs and secure file transfers.
• SQL & Python Expert: Proficiency in Python for automation and advanced data manipulation, paired with expertise in optimizing complex SQL queries.
• DevOps Literacy: Experience with version control (Git) and CI/CD best practices to ensure code quality and reliable deployments.
• Financial Domain Knowledge (Preferred): Familiarity with credit union or banking data structures (loans, shares, transactions) is a significant plus.
• The Builder Mindset: You thrive in environments where you are the first to "lay the bricks" and define the standard for those who follow.
• Quality Obsessed: You believe that data without governance and observability is just noise; you take pride in the accuracy and lineage of your work.
• AI-Enhanced Workflow: You stay ahead of the curve by actively leveraging AI tools and LLMs to accelerate development, documentation, and testing processes.