Data Engineer
Role summary
We are seeking a Data Engineer to independently manage SQL databases (Oracle, Postgres), design schemas, optimize queries, and automate ETL/ELT pipelines. The role involves performance tuning, implementing data quality checks, and database administration using PSQL. You will build robust data pipelines, manage schema evolution, handle late-arriving data, and implement backfill strategies. Responsibilities include designing table structures, implementing star/snowflake schemas, utilizing data lake technologies, and managing schema changes with tools like Liquibase. Technical proficiency in advanced SQL, Python, Apache Spark, Kafka, and Airflow is essential. Experience with cloud infrastructure (Databricks, Terraform, Kubernetes) and data governance (Pytest, data lineage) is required. Collaboration with stakeholders to translate business needs into technical solutions and documenting processes is key.
Key Qualifications:
- Ability to independently manage Oracle and Postgres SQL databases by designing efficient schemas, optimizing complex SQL queries, and automating ETL/ELT pipelines
- Experience with Performance tuning (indexing, vacuuming), implementing data quality checks, managing backups/restores, and utilizing PSQL for database administration
- Past experience successfully developing data pipelines -- Independently building robust, efficient data pipelines, managing schema evolution, handling late-arriving data, and implementing backfill strategies.
- Demonstrated ability to design efficient table structures, implement star/snowflake schemas, and utilize data lake technologies
- Familiarity with database schema change management tool such as Liquibase
- Technical Proficiency: Advanced SQL skills, strong Python programming, and working with big data technologies such as Apache Spark, Kafka, and Airflow for orchestration.
- Cloud Infrastructure & DevOps: Utilizing cloud services (e.g., Databricks) and infrastructure-as-code tools like Terraform and Kubernetes.
- Data Quality & Governance: Implementing data quality checks, testing with Pytest, and managing data lineage to ensure reliability.
- Collaboration: Working with stakeholders to translate business requirements into technical solutions and documenting data processes
- Data Maintenance: Demonstrated ability to create and follow low risk processes to handle business requested data maintenance activities
- Business Context: Understands how data drives the business processes, able to collaborate with business stakeholders to design and/or modify data processes and procedures to meet operational business needs
- Experience using AI-assisted development tools
Ability to obtain a public trust clearance
Similar roles
- Senior Data EngineerExperion Technologies · Plano, Texas, United States · Hybrid
- Lead Data EngineerSmart IT Frame LLC · Los Angeles, California, United States · Hybrid
Principal Data EngineerRS21: A Data Science and Visualization Company · United States · Remote
Senior Data EngineerRaag Solutions · Bellevue, Washington, United States · Onsite- Lead Data EngineerRetail Insight Ltd · Illinois, United States · Hybrid