Senior Data Engineer

Job description

Senior Data Engineer (Python & PySpark)Role Overview

We are seeking a seasoned Senior Data Engineer with over 8 years of experience to lead the design and development of scalable data solutions. The ideal candidate is a Python expert who has a deep mastery of the Spark ecosystem and a proven track record of building custom frameworks for complex data processing. You will be responsible for bridging the gap between raw data storage (HBase/Hive) and actionable business insights.

Key Responsibilities

Architecture & Design: Lead the design and maintenance of robust, high-performance software solutions using Python.

Framework Development: Architect and implement custom Python frameworks (e.g., Rule Engines) to automate data validation and business logic.

Data Pipeline Engineering: Develop sophisticated PySpark programs utilizing DataFrames and RDDs to execute complex business transformations.

Integration & Storage: Design data extraction strategies from HBase and manage seamless read/write operations using Hive Context objects.

API & Backend Development: Build and optimize back-end services and APIs to ensure data accessibility and system interoperability.

Performance Tuning: Monitor, troubleshoot, and optimize applications for maximum speed, security, and long-term maintainability.

Technical Qualifications

Experience: 8+ years in Software Engineering or Data Engineering.

Core Mastery: Expert-level proficiency in Python and PySpark (Mandatory).

Big Data Ecosystem: Strong hands-on experience with Apache Spark, HBase, and Hive.

Data Structures: Advanced knowledge of Spark RDDs and DataFrames for large-scale data manipulation.

Database Integration: Experience integrating Python-based services with various SQL/NoSQL databases.

Problem Solving: Proven ability to troubleshoot complex software defects in distributed computing environments