Source description
About the role
Role Senior Databricks Engineer Role Summary Senior Databricks Engineer with robust expertise in developing, optimizing, and maintaining data engineering solutions on the Databricks platform. The candidate should have deep experience with Apache Spark (PySpark), Databricks notebooks, Delta Lake, and Spark performance optimization. This role is focused on Spark-native data engineering rather than ETL tool development. Must Have Skills Strong hands-on experience with Databricks notebooks, Jobs, Workflows, Repos, and Unity Catalog Solid expertise in PySpark and Spark SQL Experience building scalable Spark-based data pipelines Robust understanding of Spark DataFrames, joins, aggregations, window functions, and distributed processing Experience with Delta Lake (MERGE, OPTIMIZE, ZORDER, VACUUM, Change Data Feed) Hands-on experience in Spark job performance tuning Experience optimizing partitions, shuffles, caching, memory usage, and cluster utilization Experience developing batch and incremental data processing pipelines Strong programming skills in Python, PySpark, and SQL Experience working on Azure Databricks, AWS Databricks, or GCP Databricks Experience with Git, CI/CD, code reviews, testing, and production deployments Good to Have Skills Experience with Lakehouse and Medallion architecture Knowledge of Data Modeling and Data Governance Nice to Have Skills Experience with Delta Live Tables (DLT) Experience mentoring teams or leading technical discussions Experience with Photon and Databricks Asset Bundles Experience with Cluster Policies and Workspace Administration Location: Pune, Hyderabad, Bhopal .
More at NewVison