Lead Databricks

Job Description Location: Noida, UP, India Core Technologies Apache Spark (Core, SQL, Structured Streaming) PySpark Databricks (AWS/Azure) Advanced SQL DevOps CI/CD Jenkins, Git/GitHub/Bitbucket Programming Python, SQL Java/Scala (Nice to have) Cloud Infrastructure AWS (S3, EMR, EC2, IAM, CloudWatch)/Azure- Preferred Databricks Runtime Cluster Management- Preferred Streaming Integration Apache Kafka Snowflake integration Airflow Perform Spark performance tuning DAG analysis Shuffle optimization Partition strategies Executor/memory tuning Experience in financial services, regulatory reporting, or enterprise data platforms. Handson experience in Delta Lake optimization and incremental processing strategies. Experience with Snowflake data warehousing. Databricks Certification (Professional level preferred). Strong understanding of distributed computing principles. Key Responsibilities Design and develop scalable batch and near-real-time ETL/ELT pipelines using Databricks (AWS) and Apache Spark (PySpark, Spark SQL, Structured Streaming). Modernize legacy SQL/Hive/stored procedure workflows into distributed Spark-native architectures. Perform Spark performance tuning, including: Build structured streaming pipelines using Kafka and Spark Structured Streaming. Design dimensional data models (Fact/Dimension, SCD Type 2). Orchestrate pipelines using Databricks Workflows / Apache Airflow. Integrate CI/CD pipelines using Jenkins, Git, Bitbucket/GitHub for automated deployment across DEV/UAT/PROD. Technical Leadership Lead end-to-end solution design for data platforms using Databricks (batch, streaming, ML workloads) Define architecture patterns like Lakehouse, Medallion (Bronze/Silver/Gold) Act as SME for Databricks, Spark, and data engineering best practices Team Leadership Lead and mentor a team of data engineers (typically 5-10 members) Conduct code reviews, enforce best practices, and ensure delivery quality Guide team in troubleshooting complex technical issues Stakeholder Management Collaborate with business stakeholders, architects, and product owners Translate business requirements into technical designs and sprint tasks Drive technical decisions (performance vs cost vs scalability) Delivery Governance Own end-to-end delivery of data projects Ensure adherence to Agile processes, SLAs, and governance models Performdesign reviews, estimations, and risk management Mandatory Competencies Data Science and Machine Learning - Data Science and Machine Learning - Databricks Cloud - Azure - Azure Data Factory (ADF), Azure Databricks, Azure Data Lake Storage, Event Hubs, HDInsight Big Data - Big Data - Pyspark Database - Database Programming - SQL Data Science and Machine Learning - Data Science and Machine Learning - Python Beh - Communication and collaboration .