Principal Databricks Architect

Architect and lead the implementation of an enterprise lakehouse on Databricks (Delta Lake, Unity Catalog, Photon, Workflows) across one or more major clouds (AWS, Azure, or GCP).

Design scalable batch and streaming data pipelines using PySpark , Spark SQL, Structured Streaming, and Delta Live Tables ; establish patterns for ingestion from operational systems, event streams, and third-party APIs.

Define and enforce platform standards for data modeling (medallion architecture), CI/CD, code quality, testing, observability, and cost optimization.

Lead the governance strategy using Unity Catalog — fine-grained access control, data lineage, audit, and PII handling — in partnership with security and compliance.

Optimize Spark workloads for performance and cost: cluster sizing, Photon, autoscaling, file layout, Z-ordering, caching, and query tuning.

Partner with ML engineers and data scientists to operationalize models using MLflow , feature stores, and model serving on Databricks.

Own the cloud infrastructure footprint for the platform: networking, IAM, secrets, encryption, and Terraform/ IaC for Databricks workspaces and supporting services.

Mentor a team of data engineers; lead architecture reviews, code reviews, and technical design sessions; raise the bar on engineering practices.

Engage with stakeholders across analytics, product, and finance to translate business needs into a roadmap for the data platform.