Data Engineer

Key Responsibilities:

- Data Architecture & Design:
Develop and implement robust, scalable, and high-performance data pipelines.
- Design and manage data architectures that support efficient storage, retrieval, and processing of large datasets.
- Architect and maintain data lakes and data warehouses to support business analytics.
- Data Pipeline Development:
Build and optimize ETL processes to integrate data from various sources into the data lake or data warehouse.
- Ensure data accuracy, completeness, and consistency throughout the data pipeline.
- Design automated data processing and transformation workflows.
- Data Modeling:
Develop and implement data models that enable efficient reporting and analytics.
- Work with business analysts and data scientists to understand data needs and provide relevant data solutions.
- Cloud Platforms:
Leverage cloud technologies (AWS, Azure, Google Cloud) for data storage, processing, and analytics.
- Ensure seamless integration with cloud-native data services such as Redshift, BigQuery, Snowflake, etc.
- Performance Optimization:
Continuously monitor and optimize the performance of data pipelines, ensuring minimal latency and efficient resource utilization.
- Conduct troubleshooting and performance tuning for data-related systems.
- Collaboration & Communication:
Work closely with data scientists, analysts, and other engineering teams to ensure data is accessible, clean, and structured to meet business needs.
- Maintain clear documentation for all data engineering workflows, pipelines, and architecture.
- Data Governance & Security:
Ensure compliance with data governance and security policies.
- Implement data access controls and audit processes to ensure data security and privacy.

Qualifications & Skills:

- Education:
Bachelor's degree in Computer Science, Engineering, Information Systems, or related field. A Master's degree is a plus.
- Experience:
6+ years of hands-on experience in data engineering or a related role.
- Programming Skills:
Expertise in Python, Java, or Scala. Familiarity with SQL and NoSQL databases.
- Data Technologies:
Strong experience with ETL tools (Apache NiFi, Talend, Airflow, etc.).
- Hands-on experience with cloud-based data platforms (AWS, Azure, Google Cloud).
- Experience with data warehousing technologies (Redshift, Snowflake, BigQuery, etc.).
- Working knowledge of distributed computing systems (Hadoop, Spark).
- Data Modeling & Analysis:
Strong skills in data modeling and data analysis using SQL and other querying languages.
- Version Control & CI/CD:
Proficiency in version control (Git) and CI/CD pipelines.
- Problem Solving & Troubleshooting:
Strong analytical, troubleshooting, and debugging skills.
- Communication:
Excellent communication skills with the ability to collaborate across multiple teams and stakeholders.

Nice-to-Have:

• Experience with Machine Learning frameworks and integrations (TensorFlow, PyTorch, etc.).
• Familiarity with containerization technologies (Docker, Kubernetes).
• Knowledge of data visualization tools (Tableau, Power BI).