Big Data Developer/Data Engineer
Role summary
A Big Data Developer/Data Engineer is needed for a long-term contract in Toronto, ON. The role involves working within an Agile cross-functional team to design, develop, and maintain data ingestion flows and orchestrate the data platform. Key responsibilities include designing and implementing data pipelines, frameworks, and ETL processes, ensuring data quality, security, and performance. The developer will also optimize SQL queries, integrate Snowflake with other data sources and BI tools, monitor and troubleshoot data jobs and platform issues, and maintain comprehensive documentation. Required skills include 5+ years of experience with Big Data technologies (Hadoop, Spark, Hive), proficiency in Snowflake, strong SQL and data modeling skills, and programming experience in Scala or Java with API development. Familiarity with ETL tools, cloud platforms (preferably AWS), SCM, IaC, and CI/CD pipelines is also necessary. Knowledge of Gen AI for code generation is a plus.
Role: Big Data Developer/Data Engineer
Location: Toronto, ON
Duration: Long Term Contract
Job Description:
Work within an Agile cross functional team to design, develop and maintain data ingestion flows and evolve the platform to orchestrate them.
Design and implement data pipelines, framework and ETL processes.
Ensure data quality, security, and performance. Develop and optimize SQL queries, stored procedures, and views.
Integrate Snowflake with other data sources and BI tools.
Monitor and troubleshoot data jobs and platform issues.
Develop and maintain comprehensive documentation for data pipelines, transformations, and data models.
What do you need to succeed
5+ years of Experience with Big Data technologies used for ETL Hadoop, Spark, Hive
Well versed with Snowflake Platform.
Strong SQL skills and knowledge of data modeling.
Programming Experience with Scala or Java an API Development.
Experience with ETL tools (e.g., Informatica, Talend, Apache Airflow).
Familiarity with any of the cloud platforms (Preferably AWS).
Knowledge on Gen AI to generate code and improve developer proficiency
Knowledge of Python or other scripting languages is a plus.
Excellent problem-solving and communication skills.
Knowledge of SCM, Infrastructure-as-code, and CICD pipelines.
Nice to Have
Experience with workflow management tools like Apache Airflow
Experience with Continuous Integration tools Git Actions, GitHub, Automated Testing tools, Git, or similar tools.
Experience with DockerKubernetesContainersOCP4
Shell Scripting Bachelors or masters degree in computer science, Data Engineering and, or a related field.