Scicom Infrastructure Services, Inc. Verified

IT Services, Business Process Outsourcing (BPO), Cloud Services, Managed Services

Databricks SME

RemoteRemoteContractPosted 1 month agoHidden Gem · Himalayas

Is this role right for you?

Upload your resume and get a skill-by-skill breakdown — see exactly where you match, where you're close, and what to highlight. Not a mystery percentage.

Get a tailored resume highlighting what this role needs.

Required skills

Databricks
Lakehouse
Well-Architected Framework
batch
streaming
analytics
ML workloads
cluster
compute
workspace

Nice to have

MLflow
MLOps
real-time analytics
multi-region
cross-account
consulting

Role summary

Seeking a Databricks SME to design, implement, and optimize enterprise Databricks Lakehouse architectures. This role involves defining reference architectures for various workloads, ensuring alignment with the Databricks Well-Architected Framework, and leading implementations of key components like Unity Catalog, Delta Lake, and DLT. Responsibilities include integrating with cloud services, establishing CI/CD pipelines, and ensuring security, governance, and compliance. The ideal candidate will have 7+ years in data engineering/architecture with 3-5 years of hands-on Databricks experience, strong Apache Spark and cloud platform skills, and excellent communication and leadership abilities. Certifications in Databricks Data Engineering or Architecture are required.

Key Responsibilities:

Architecture & Platform Design

Design enterprise Databricks Lakehouse architectures aligned with the Databricks Well-Architected Framework
Define reference architectures for batch, streaming, analytics, and ML workloads
Select and standardize cluster, compute, and workspace architectures
Design multi-workspace strategies (dev/test/prod, shared vs. isolated)
Ensure architectures meet scalability, availability, and performance requirements

Well-Architected Framework Alignment

Apply Databricks best practices across all pillars, including:

Security & Governance (Unity Catalog, IAM, data access controls)
Reliability & Resilience (job retries, checkpointing, failure isolation)
Performance Efficiency (cluster sizing, autoscaling, caching)
Cost Optimization (compute policies, workload separation, monitoring)
Operational Excellence (monitoring, automation, CI/CD, runbooks)

Implementation & Engineering

Lead Databricks workspace, cluster, and Unity Catalog implementations
Implement Delta Lake, Delta Live Tables (DLT), and Structured Streaming
Build and optimize ETL/ELT pipelines using Spark and SQL
Integrate Databricks with cloud services (S3/ADLS/GCS, IAM, Key Vault, networking)
Establish CI/CD pipelines for notebooks, jobs, and infrastructure

Security, Governance & Compliance

Implement Unity Catalog for centralized governance
Define data classification, lineage, and audit strategies
Enforce least-privilege access and secure networking patterns
Support compliance requirements (HIPAA, SOC 2, PCI, GDPR as applicable)

Operations & Optimization

Monitor platform health, performance, and cost
Troubleshoot production issues across jobs, clusters, and data pipelines
Perform workload tuning and cost-performance optimization
Define SLOs, alerts, and operational metrics

Collaboration & Leadership

Partner with Data Engineering, Analytics, ML, Platform, and Security teams
Translate business requirements into technical architectures
Provide architectural guidance and technical mentorship
Communicate risks, tradeoffs, and recommendations to leadership

Required Qualifications:

Experience

7+ years in data engineering, analytics, or platform architecture
3–5+ years hands-on Databricks experience in production environments
Proven experience applying the Databricks Well-Architected Framework
Experience designing cloud-native lakehouse architectures
Experience supporting mission-critical data platforms

Technical Skills

Databricks Lakehouse Platform
Apache Spark (PySpark / Scala / Spark SQL)
Delta Lake, Delta Live Tables, Structured Streaming
Unity Catalog (governance, lineage, access controls)
Cloud platforms: AWS, Azure, or GCP
Infrastructure as Code (Terraform strongly preferred)
CI/CD tools (GitHub Actions, Azure DevOps, GitLab, etc.)
Data formats and protocols (Parquet, JSON, Avro)

Certifications Required:

Databricks Certified Data Engineer Professional
Databricks Certified Professional Architect (or equivalent advanced certification)

Preferred / Additional Certifications

AWS Certified Solutions Architect (Associate or Professional)
Azure Solutions Architect Expert
Google Professional Data Engineer
Databricks Machine Learning Professional
Snowflake or other cloud data platform certifications

Soft Skills

Strong architectural decision-making and documentation skills
Excellent communication with technical and non-technical stakeholders
Ability to lead design reviews and architecture governance forums
Strong troubleshooting and performance-tuning mindset

Nice-to-Have Experience

MLflow and MLOps architectures
Real-time analytics and streaming pipelines
Multi-region or cross-account data architectures
Consulting or MSP delivery experience

Originally posted on Himalayas

Ready to apply?

You'll be redirected to Scicom Infrastructure Services, Inc.'s application page.