Data Automation Engineer

Role Overview

We are seeking an experienced Data Automation Engineer to design, build, and optimize scalable data

ingestion pipelines for product telemetry and enterprise business applications data. This role operates in

a fastpaced environment, rapidly prototyping data pipelines to enable data exploration, analytics, and

downstream consumption.

Responsibilities

 Design and implement modern data ingestion pipelines for product telemetry and business

systems data using:

o Python and Bulk APIs

o Object storage and S3based lakehouse architectures

o Streaming ingestion techniques leveraging Kafka, Spark, and other opensource

frameworks

 Build endtoend automated pipelines from source systems to Snowflake, ensuring reliability,

scalability, and observability.

 Apply data curation techniques (advanced SQL) in Snowflake, including transformation,

standardization, and preparation of analyticsready datasets.

 Optimize ingestion performance for largescale, highvolume datasets (terabytes of data) using:

o API ratelimit handling

o Microbatching strategies

o Incremental loading and CDC (Change Data Capture) techniques

 Implement DevOps and CI/CD practices for data pipelines, including version control, deployment

automation, and environment management.

 Partner with data analysts, SMEs, and platform teams to quickly prototype data pipelines

supporting exploration and evolving requirements.

 Monitor, troubleshoot, and optimize ingestion jobs to ensure data freshness, accuracy, and

performance.

Required Skills & Experience

 12+ years of strong hands on experience in data engineering and ingestion automation in cloud or

hybrid environments.

 Proven experience ingesting product telemetry data and business application data (e.g., CRM,

ERP, financial, consumption, renewals systems).

 Advanced proficiency with Python for building ingestion pipelines, automation, orchestration, and

data validation.

 Expertise with streaming and distributed data processing technologies such as Kafka and Spark.

Experience working with S3 or lakehousestyle storage architectures.

 Strong proficiency with Snowflake, including data ingestion, transformation, and performance

optimization.

 Deep understanding of ingestion optimization techniques for large, voluminous datasets,

including rate limits, microbatching, incremental loads, and CDC.

 Experience applying DevOps practices to data pipelines, including CI/CD and operational

automation.

 Ability to work independently in a fastpaced, iterative environment and deliver working solutions

rapidly.

Desirable Skills

 Experience with additional opensource ingestion frameworks and connectors

 Exposure to eventdriven or telemetryheavy architectures at scale