Databricks Platform Engineer

Required Skills

Databricks Platform

Workspace administration

Cluster management

Cluster policies

Instance pools

Job clusters

Databricks Runtime management

Workflows

Databricks Repos

Unity Catalog

Secret Scopes

Service Principals

Personal Access Tokens (PATs)

AWS

IAM

VPC fundamentals

EC2 concepts

S3

CloudWatch

KMS

Security Groups

Networking fundamentals

DevOps

Git

CI/CD

Terraform

Infrastructure as Code

Azure DevOps, GitHub Actions, or Jenkins

Monitoring

Platform monitoring

Incident management

Log analysis

Performance tuning

Cost optimization

Required Experience

Candidates should have hands-on experience with:

Managing enterprise Databricks platforms

Setting up multiple Databricks workspaces

Configuring Unity Catalog

AWS IAM integration

Cluster policy creation and governance

Runtime version upgrades

CI/CD implementation

Git integration

Platform monitoring and troubleshooting

Secrets management

Audit logging

Production support

Cost optimization initiatives

Preferred Qualifications

• Experience managing large-scale Databricks environments (100+ users).

• Experience supporting multiple business units or enterprise data platforms.

• Hands-on experience with Terraform for Databricks infrastructure provisioning.

• Knowledge of Lakehouse architecture and Delta Lake.

• Experience with FinOps practices for cloud cost optimization.

• Familiarity with data governance and security frameworks.

• Nice to Have

• Experience with AWS Organizations and Control Tower.

• Knowledge of networking concepts for private connectivity (PrivateLink, VPC endpoints).

• Experience with monitoring tools such as CloudWatch, Datadog, or Splunk.

• Exposure to Apache Spark internals.

• Experience supporting AI/ML workloads on Databricks.

• Education

• Bachelor's degree in Computer Science, Information Technology, Engineering, or a related field.

• Certifications (Preferred)

• Databricks Certified Data Engineer Professional or Databricks Certified Platform Administrator (if available).

• AWS Certified Solutions Architect – Associate/Professional.

• AWS Certified SysOps Administrator.

• Terraform Associate certification.

• Ideal Candidate Profile