Databricks - Data Engineer

Role Overview: We are looking for a highly skilled and hands-on Senior Databricks Engineer to join our team. This is a client-facing role requiring strong technical expertise, ownership mindset, proactive communication, and the ability to build scalable enterprise-grade data solutions. The ideal candidate should have deep experience in Databricks, PySpark, SQL/PLSQL, modern data engineering practices, CI/CD processes, and large-scale distributed data processing. The candidate should be comfortable working in fast-paced environments, troubleshooting complex data engineering problems, and driving optimization initiatives. Key Responsibilities: Design, develop, and maintain scalable and high-performance data pipelines using Databricks and PySpark. Build and manage ingestion frameworks for batch and streaming data workloads. Develop enterprise-grade data solutions using Delta Lake and Medallion Architecture principles. Implement and manage data pipelines using DLT (Delta Live Tables) and Spark Streaming. Work extensively with Unity Catalog (UC) for governance and data access management. Optimize Spark jobs, SQL queries, and distributed processing workloads for performance and cost efficiency. Participate in CI/CD and release management processes using Git and Databricks Asset Bundles. Collaborate with business stakeholders, architects, and client teams to understand requirements and deliver robust solutions. Troubleshoot production issues, identify root causes, and implement sustainable fixes. Drive engineering best practices, code quality standards, and reusable framework development. Proactively communicate risks, blockers, and improvement opportunities. Challenge existing processes and contribute ideas to improve platform scalability, reliability, and engineering maturity. Must-Have Skills & Experience: Technical Skills: Minimum 6+ years of hands-on experience in Data Engineering (using Databricks) Strong hands-on experience with: Databricks PySpark SQL PLSQL Strong experience building large-scale and complex data pipelines. Hands-on experience with: Delta Lake Medallion Architecture Unity Catalog (UC) Spark Streaming Experience designing and developing ingestion frameworks. Strong troubleshooting and performance tuning skills in Spark/Databricks environments. Experience with CI/CD processes, Pull Request (PR) workflows, Git, and Databricks Asset Bundles. Experience working with distributed data processing and optimization techniques. Strong understanding of data engineering best practices and scalable architecture patterns. Soft Skills: Excellent commitment and ownership mindset. Strong client-facing communication and stakeholder management skills. Proactive communicator with the ability to challenge status quo constructively. Strong analytical and problem-solving abilities. Ability to work independently in fast-paced delivery environments. Nice-to-Have Skills: Experience implementing Data Quality frameworks and controls. Retail domain experience preferred. Data Modeling skills (dimensional modeling, warehouse modeling, etc.). Exposure to cloud-native data platforms (preferably Azure) and enterprise data governance practices. Experience working in Agile/Scrum delivery models. Databricks certified