Data Bricks Data Engineer
Role summary
We are seeking an experienced Data Engineer to design, develop, and maintain complex data pipelines and ETL/ELT workflows within the Azure ecosystem. The role requires advanced proficiency in Python, PySpark, and SQL, with hands-on experience in Azure Databricks for cluster management, job scheduling, and governance. You will leverage core Azure services, containerization, and Infrastructure-as-Code (ARM templates) to build robust and secure data solutions. Collaboration with cross-functional teams, adherence to Agile methodologies, and a strong understanding of data governance are essential. Responsibilities include implementing monitoring, alerting, cost optimization, and providing production support.
Must Have Technical/Functional Skills
- Strong programming skills in Python and PySpark.
- Advanced proficiency writing SQL for analytics and ETL processes.
- Proven experience building and optimizing complex data pipelines in Azure.
- Hands-on experience with Azure Databricks: cluster management, job scheduling, workspace governance.
- Strong working knowledge of core Azure services: Storage Account, Synapse, Key Vault, VMSS, Function Apps, Web Apps, Log Analytics Workspace, service principals, and managed identities.
- Experience with container services (ACA, container instances) and containerized data workloads.
- Familiarity with Azure networking concepts and secure network integration for data platforms.
- Experience creating Azure infrastructure using ARM templates.
- Proficient with GitLab and Azure DevOps for CI/CD and source control workflows.
- Strong analytical, problem-solving, and communication skills; proven ability to work cross-functionally.
- Experience working in Agile teams and understanding of data governance frameworks.
Responsibilities
- Design, develop, and maintain end-to-end data pipelines and ETL/ELT workflows using PySpark and Python.
- Implement, optimize, and monitor large-scale data processing workloads in Azure Databricks, including cluster configuration, autoscaling, and governance.
- Build and maintain data integration and orchestration solutions using Azure services to meet performance, availability, and security requirements.
- Collaborate with data consumers, thread authors/owners, and stakeholders to gather business requirements, prioritize needs, and translate analytical objectives into technical designs.
- Implement secure data access patterns using Azure Active Directory, Managed Identities, and service principals.
- Author Infrastructure-as-Code for Azure resources (ARM templates) and deploy consistent, repeatable environments.
- Configure and operate Azure components including Storage Account, Synapse, Key Vault, VMSS, Function Apps, Web Apps, Log Analytics Workspace, Azure Container Apps / container instances, and related services.
- Collaborate with networking and security teams to design and implement Azure networking for data solutions.
- Implement monitoring, alerting, and cost optimization for data workloads (Log Analytics, metrics, and dashboards).
- Use GitLab and Azure DevOps for source control, CI/CD pipelines, and release management.
- Follow Agile/Scrum practices and participate in sprint planning, standups, and retrospectives.
- Ensure soluti ons meet data governance, lineage, and compliance requirements.
- Operations Support and Oncall Support for Production Issues and Deployments.
Base Salary Range : $125,000 to $135,000 Per Annum
TCS Employee Benefits Summary:
Discretionary Annual Incentive.
Comprehensive Medical Coverage: Medical & Health, Dental & Vision, Disability Planning & Insurance, Pet Insurance Plans.
Family Support: Maternal & Parental Leaves.
Insurance Options: Auto & Home Insurance, Identity Theft Protection.
Convenience & Professional Growth: Commuter Benefits & Certification & Training Reimbursement.
Time Off: Vacation, Time Off, Sick Leave & Holidays.
Legal & Financial Assistance: Legal Assistance, 401K Plan, Performance Bonus, College Fund, Student Loan Refinancing.
#LI-SV2
#LI-KUMARAN
Location
Seattle, WA
Job Function
TECHNOLOGY
Role
Engineer
Job Id
403103
Desired Skills
Datastage
Salary Range
$125,000-$135,000 a year
Desired Candidate Profile
Qualifications : BACHELOR OF COMPUTER SCIENCE
