Padmi

Lead Data Engineer Databricks on AWS | PySpark | Delta Lake | Unity Catalog

BangalorePosted 1 month ago
Infrastructure And DatabasesSeniorFull Time; Regular
Apply at Obrimo Technologies

Opens the source posting on shine.com

Source description

About the role

View original

We are looking for a Lead Data Engineer / Technical Anchor with strong expertise in Databricks on AWS to lead the architecture, engineering, and delivery of enterprise-scale Data Lakehouse solutions. This is a hands-on technical leadership role involving architecture, development, mentoring, and stakeholder collaboration. Required Skills DatabricksAWS (S3, EMR, Athena, Glue, CloudWatch)PySpark / Apache SparkPythonAdvanced SQLDelta Lake & Delta Live Tables (DLT)Unity CatalogMedallion Architecture (Bronze, Silver, Gold)Data EngineeringPerformance Tuning & Spark OptimizationKey Responsibilities Design and implement scalable Data Lakehouse architectures using Databricks on AWS.Develop robust batch and streaming data pipelines using PySpark, SQL, Delta Lake, and Delta Live Tables.Configure and manage Unity Catalog across multiple environments.Implement secure AWS data platform architectures using IAM, KMS, VPC, and PrivateLink.Optimize Spark performance, cluster utilization, and compute costs.Lead architecture discussions, conduct code reviews, and mentor Data Engineers.Collaborate with product owners, architects, and client stakeholders to deliver enterprise data solutions.Preferred Qualifications 8+ years of Data Engineering experience.Minimum 4+ years of hands-on experience with Databricks on AWS.Strong expertise in PySpark, Python, SQL, and AWS.Experience with Data Lakehouse architecture and enterprise-scale data platforms.Excellent communication and leadership skills. .

One address, no account. We’ll tell you when matching roles go live.

More at Obrimo Technologies

Related open roles

View all roles