Senior Databricks Support Engineer

Profile : Senior Azure Databricks Support Engineer Location : Noida Experience : 7+ Years Contact : Nikhil.vats@multiversetech.com We are seeking a Senior Azure Databricks Support Engineer with strong expertise in production support, Azure cloud, Databricks, PySpark, Python, SQL, and ETL/ELT pipelines . The ideal candidate will be responsible for managing mission-critical production environments, resolving complex incidents, supporting deployments, optimizing platform performance, and collaborating with development teams to ensure seamless transition from support to production. This role combines production operations (60%) with development and automation (40%) to improve platform reliability, scalability, and operational efficiency. Key Responsibilities Production Support Provide L3 production support for Azure Databricks workloads, data pipelines, notebooks, workflows, and clusters. Monitor production environments and proactively identify performance bottlenecks. Troubleshoot failures in ETL/ELT pipelines and Spark jobs with minimal business impact. Handle critical production incidents, perform Root Cause Analysis (RCA), and implement permanent fixes. Ensure compliance with SLA, KPI, and ITIL support processes. Participate in 24x7 on-call and rotational production support. Databricks Administration Manage Azure Databricks workspaces, clusters, jobs, libraries, and Unity Catalog. Configure autoscaling, cluster policies, security, and access controls. Optimize Spark jobs for performance, cost, and resource utilization. Monitor cluster health and recommend infrastructure improvements. Development & Production Readiness Develop and enhance PySpark, Python, and SQL code for data processing. Support migration of code from Development QA UAT Production. Review deployment packages and validate production readiness. Assist developers in resolving production defects and performance issues. Perform code optimization and support release management activities. Azure Cloud Operations Support Azure Data Factory (ADF), Azure Data Lake Storage (ADLS Gen2), Azure Key Vault, Azure Monitor, and Log Analytics. Configure monitoring dashboards, alerts, and automated notifications. Work with infrastructure, networking, and security teams to maintain platform stability. Manage environment upgrades, patches, and capacity planning. CI/CD & Automation Support Azure DevOps pipelines for automated deployments. Automate operational tasks using Python, PowerShell, or Shell scripting. Improve deployment processes through Infrastructure as Code (IaC) and automation. Maintain version control using Git. Performance & Optimization Optimize Spark execution plans, partitioning, caching, and cluster utilization. Reduce job execution time and cloud infrastructure costs. Analyze job failures and recommend best practices for performance tuning. Collaboration Work closely with Data Engineers, Data Architects, DevOps Engineers, Business Analysts, and Application Support teams. Create SOPs, knowledge base articles, runbooks, and operational documentation. Mentor junior support engineers and provide technical guidance. Participate in production release planning and post-deployment validation. Required Skills Azure Databricks Apache Spark PySpark Python SQL Azure Data Factory (ADF) Azure Data Lake Storage (ADLS Gen2) Azure Synapse Analytics Azure DevOps Git CI/CD Pipelines Unity Catalog Delta Lake Required Qualifications Bachelor's degree in Computer Science, Information Technology, or related field. 610 years of experience in Azure Data Engineering or Azure Databricks Production Support. Hands-on experience supporting enterprise-scale production environments. Strong knowledge of Spark architecture and Databricks optimization techniques. Experience in incident management, change management, and release management. Excellent troubleshooting and analytical skills.