Senior Azure Databricks Developer

Role: Senior Azure Databricks Developer Role Summary: Senior Databricks Engineer with strong expertise in developing, optimizing, and maintaining data engineering solutions on the Databricks platform. The candidate should have deep experience with Apache Spark (PySpark), Databricks notebooks, Delta Lake, and Spark performance optimization. This role is focused on Spark-native data engineering rather than ETL tool development. Must Have Skills: Strong hands-on experience with Databricks notebooks, Jobs, Workflows, Repos, and Unity CatalogStrong expertise in PySpark and Spark SQLExperience building scalable Spark-based data pipelinesStrong understanding of Spark DataFrames, joins, aggregations, window functions, and distributed processingExperience with Delta Lake (MERGE, OPTIMIZE, ZORDER, VACUUM, Change Data Feed)Hands-on experience in Spark job performance tuningExperience optimizing partitions, shuffles, caching, memory usage, and cluster utilizationExperience developing batch and incremental data processing pipelinesStrong programming skills in Python, PySpark, and SQLExperience working on Azure Databricks, AWS Databricks, or GCP DatabricksExperience with Git, CI/CD, code reviews, testing, and production deploymentsImplement event-driven data flows and real-time data ingestion using Apache Kafka.Develop and integrate REST APIs for both data extraction and downstream data serving.Monitor, tune, and optimize PySpark workloads for maximum performance and cost efficiency.Communicate effectively with cross-functional stakeholders to gather requirements and translate them into scalable technical solutions.Mentor junior developers, guide architectural decisions, and prepare to take on team lead responsibilities in the future. as nice to have. Good to Have Skills: Experience with Lakehouse and Medallion architectureKnowledge of Data Modeling and Data Governance Nice to Have Skills: Experience with Delta Live Tables (DLT)Experience mentoring teams or leading technical discussionsExperience with Photon and Databricks Asset BundlesExperience with Cluster Policies and Workspace AdministrationDesign, build, and maintain robust batch and streaming data pipelines using Databricks and PySpark.Extract complex data from a variety of upstream sources (databases, internal/external APIs, flat files) and process it into the data lake.Architect and optimize Delta Tables, utilizing the Medallion architecture (Bronze, Silver, Gold layers) for structured data governance. Soft skills: Must have strong written and verbal communication skills, excellent articulation skills, and experience working with client teams. The candidate should have actively participated in and contributed to sprint planning and solution design. .