TechDoQuest Verified
Education, IT Services, Professional Training
Senior Data Engineer (Hadoop + GCP Dataproc)
Toronto, Ontario, CanadaOnsiteContractSeniorPosted 2 months ago
Compensation estimateAI
See base, equity, bonus, and total comp estimates for this role — free, no credit card.
Sign up to see compensation estimateRequired Qualifications:
- 6–10+ years of experience working within the Hadoop ecosystem, with deep expertise in Hive, Hive Metastore, and GCP Dataproc.
- Strong hands-on experience with Dataproc Serverless, BigQuery, Google Cloud Storage (GCS), Cloud Composer, and Cloud Logging/Monitoring.
- Solid understanding of table formats (Parquet, ORC), partitioning/bucketing strategies, and query performance optimization.
- Proven experience migrating Hive datasets, tables, and SQL queries to BigQuery, including handling syntax differences, functions, UDFs, and performance tuning.
- Strong knowledge of Security/IAM best practices, service accounts, and network/security controls; ability to operate within strict organization-level policies.
Nice-to-Have Skills :
- Familiarity with Dataproc Metastore vs. standalone Hive Metastore, and experience with Glue or other metadata/catalog services.
- Exposure to data quality frameworks and lineage tools (e.g., OpenLineage, Collibra).
- A FinOps mindset, including experience managing quotas, reservations, and cost governance for Dataproc and BigQuery workloads.
