Recutify Inc. Verified
Human Resources Technology (HR Tech), Artificial Intelligence, Software
Senior Data Engineer (Hadoop + GCP Dataproc)
Toronto, Ontario, CanadaHybridFull TimeSeniorPosted 1 month ago
Compensation estimateAI
See base, equity, bonus, and total comp estimates for this role — free, no credit card.
Sign up to see compensation estimatePosition: Senior Data Engineer (Hadoop + GCP Dataproc)
Location : Toronto
Mode of Work : Hybrid (4 days a week)
Role Overview:
We are looking for an experienced Senior Data Engineer with strong expertise in the Hadoop ecosystem and Google Cloud Platform, particularly GCP Dataproc. The ideal candidate will have hands-on experience in modernizing data platforms, optimizing large-scale data processing workloads, and migrating Hive-based workloads to BigQuery.
Required Qualifications:
- 6 10+ years of experience working within the Hadoop ecosystem, with deep expertise in Hive, Hive Metastore, and GCP Dataproc.
- Strong hands-on experience with Dataproc Serverless, BigQuery, Google Cloud Storage (GCS), Cloud Composer, and Cloud Logging/Monitoring.
- Solid understanding of table formats (Parquet, ORC), partitioning/bucketing strategies, and query performance optimization.
- Proven experience migrating Hive datasets, tables, and SQL queries to BigQuery, including handling syntax differences, functions, UDFs, and performance tuning.
- Strong knowledge of Security/IAM best practices, service accounts, and network/security controls; ability to operate within strict organization-level policies.
Nice-to-Have Skills :
- Familiarity with Dataproc Metastore vs. standalone Hive Metastore, and experience with Glue or other metadata/catalog services.
- Exposure to data quality frameworks and lineage tools (e.g., OpenLineage, Collibra).
- A FinOps mindset, including experience managing quotas, reservations, and cost governance for Dataproc and BigQuery workloads
