Data Engineer (Databricks)

JOB SUMMARY

We are seeking an experienced Data Engineer with strong hands-on Databricks expertise to support the design, development, and optimization of enterprise data solutions. The successful candidate will have deep experience building and maintaining ELT pipelines using native Databricks orchestration, working with large-scale data platforms, and applying modern data architecture principles.

JOB RESPONSIBILITIES

Design, develop, and maintain ELT pipelines using Databricks and related technologies.

Build and optimize large-scale data processing workflows using PySpark and SQL.

Orchestrate multi-stage data pipelines using Databricks Workflows and Jobs, and develop Delta Live Tables (DLT) pipelines with built-in data quality enforcement.

Develop and manage data models to support analytics, reporting, and business intelligence requirements.

Implement and support data Lakehouse architectures and data warehousing solutions.

Collaborate with data analysts, architects, and business stakeholders to understand and deliver on data requirements.

Ensure data quality, integrity, and governance across all data platforms.

Monitor and troubleshoot data workflows, pipeline performance, and platform issues.

Contribute to continuous improvement of data engineering standards and practices.

REQUIRED TECHNICAL SKILLSET

Strong hands-on experience with Databricks (Delta Lake, Databricks Workflows, Unity Catalog)

Strong hands-on experience with Databricks-native orchestration (Databricks Workflows, Jobs) as the primary mechanism for multi-stage pipeline orchestration and scheduling

Experience with Delta Live Tables (DLT) for pipeline development and expectation-based data quality enforcement

Proven experience building and supporting ELT pipelines at scale

Strong proficiency in PySpark and SQL

Experience with data modeling techniques and best practices

Good understanding of data Lakehouse architecture and data warehousing concepts

Experience with version control tools (e.g., Git)

Familiarity with cloud platforms (Azure preferred; AWS or GCP considered)

PREFERRED EXPERIENCE

Experience using Azure Data Factory for data ingestion from source systems, with Databricks Workflows/Jobs as the primary orchestration layer for transformation logic

Experience with Apache Airflow

Exposure to data governance frameworks and tools (e.g., Unity Catalog, Microsoft Purview)

Experience in enterprise consulting or regulated industry environments

Experience working with globally distributed teams across different time zones

CERTIFICATIONS PREFERRED

Databricks Certified Associate Developer for Apache Spark

Databricks Certified Data Engineer Associate or Professional

Microsoft Certified: Azure Data Engineer Associate (DP-203) or equivalent

Additional data or cloud certifications are a plus