Data Engineer (Experienced)
Role summary
We are seeking an experienced Data Engineer to design, build, and maintain our data infrastructure, including data lakes and warehouses. Responsibilities include data ingestion, transformation, storage, ensuring data availability and quality, and collaborating with stakeholders. The role requires expertise in data pipeline development, data modeling, data lake and warehouse management, ETL, and data orchestration. Familiarity with cloud platforms, containerization, and data governance is preferred. This is a full-time, remote position.
Data Engineer
We are looking for a data engineer to join our team and help design, build, and maintain the data infrastructure that powers our organization's data initiatives (including managing a data lake and data warehouse). Your role will involve data ingestion, transformation, storage, and ensuring data availability and quality for analytics and reporting. You will also collaborate with data scientists and other stakeholders to ensure the quality, reliability, and availability of our data.
Responsibilities:
- Design, develop, and maintain robust data pipelines for ingesting, processing, and transforming data from various sources into usable formats.
- Architect and manage our data lake, ensuring it serves as a central repository for structured and unstructured data, supporting analytics and data exploration.
- Implement and manage data warehousing solutions for structured data. Ensure data is organized, indexed, and accessible for analytics.
- Create and manage data models and schemas to support data storage and retrieval needs. Optimize data structures for performance and scalability.
- Develop and maintain ETL processes to ensure data quality, consistency, and accuracy. Transform and clean data as needed for downstream consumption.
- Collaborate with cross-functional teams to integrate data from diverse sources, including databases, APIs, external systems, and data lakes.
- Establish and enforce data quality standards and practices. Implement data validation and error handling mechanisms.
- Monitor and optimize data pipelines, data lakes, data warehouses, and databases for performance, scalability, and efficiency.
- Implement data security and access control measures to protect sensitive data.
- Maintain clear and comprehensive documentation of data processes, pipelines, and infrastructure.
Requirements:
- Bachelor's or Master's degree in Computer Science, Data Engineering, or a related field.
- Proven experience as a Data Engineer or similar role, with expertise in data pipeline development, data modeling, data lake management, and data warehouse management.
- Proficiency in data engineering technologies and tools, including ETL frameworks, data warehousing solutions, and database management systems.
- Strong programming skills in relevant languages.
- Experience with data orchestration and workflow management tools
- Familiarity with cloud platforms and containerization technologies.
- Knowledge of data governance and compliance standards.
- Excellent problem-solving skills and the ability to work independently and as part of a team.
- Excellent communication skills.
Other Key Considerations
- Adept at SCRUM/Agile methodologies and environment
- Highly organized and able to work in a fast-paced environment.
- Able & willing to learn new tools, software, and processes.
- Able to take direction, listen to the needs of business managers, and interpret their needs into technical instructions.
- Effective working on teams, within, and outside of function.
- Able to work under pressure.
Job Type: Full-time
Work Location: Remote