Data Engineer Development Lead

Position Overview We are seeking a Senior Data Engineer to drive cloud data modernization and build scalable, AI-ready data platforms. This role emphasizes expertise in Databricks, PySpark, Azure Data Factory, Logic Apps, and Airflow, with a strong focus on orchestration, pipeline reliability, and end-to-end data workflow management. Key Responsibilities Design, build, and maintain scalable data pipelines using Databricks (PySpark), ADF, Logic Apps, and Airflow Develop and manage end-to-end orchestration frameworks integrating Airflow DAGs with ADF and Logic Apps Implement advanced workflow orchestration patterns (event-driven, micro-batch, hybrid scheduling) Ensure pipeline dependency management, execution reliability, and operational excellence Build high-performance ETL/ELT pipelines using Databricks and Delta Lake architecture Implement data observability, monitoring, and alerting mechanisms across workflows Optimize pipelines and workflows for performance, scalability, and cost efficiency Integrate pipelines with Azure services such as ADLS Gen2, Blob Storage, and event triggers Implement CI/CD for pipelines and workflows using GitHub Actions or equivalent Ensure data quality, governance, and security compliance Collaborate with data science teams for ML/GenAI data readiness Mentor junior engineers and drive best practices for orchestration and pipeline design Required Qualifications 10+ years of experience in data engineering Highly proficient in Databricks (PySpark, Delta Lake), Azure Data Factory (ADF), Azure Logic Apps, and Apache Airflow Strong programming skills in Python and advanced SQL Experience in building orchestrated data platforms with multi-tool integration Strong understanding of ETL/ELT patterns and orchestration frameworks Experience in cloud-native data architectures (Azure preferred) Hands-on experience with monitoring, logging, and pipeline reliability Data Engineer (Databricks +Pyspark) JD + ADF/Logic Apps (Highly Proficient) + Airflow Preferred Qualifications Experience with event-driven architecture and API integrations Exposure to AI/ML data pipelines and MLflow Knowledge of Data Lakehouse architectures and governance frameworks Azure / Databricks certifications, Must Have Skills:Design, build, and maintain scalable data pipelines using Databricks (PySpark), ADF, Logic Apps, and Airflow Build high-performance ETL/ELT pipelines using Databricks and Delta Lake architecture, Strong knowledge on Databricks, Orchestration workflow using ADF, Logic Apps, Airflow., Nice to have skills - .Net , C# Position Overview We are seeking a Senior Data Engineer to drive cloud data modernization and build scalable, AI-ready data platforms. This role emphasizes expertise in Databricks, PySpark, Azure Data Factory, Logic Apps, and Airflow, with a strong focus on orchestration, pipeline reliability, and end-to-end data workflow management. Key Responsibilities Design, build, and maintain scalable data pipelines using Databricks (PySpark), ADF, Logic Apps, and Airflow Develop and manage end-to-end orchestration frameworks integrating Airflow DAGs with ADF and Logic Apps Implement advanced workflow orchestration patterns (event-driven, micro-batch, hybrid scheduling) Ensure pipeline dependency management, execution reliability, and operational excellence Build high-performance ETL/ELT pipelines using Databricks and Delta Lake architecture Implement data observability, monitoring, and alerting mechanisms across workflows Optimize pipelines and workflows for performance, scalability, and cost efficiency Integrate pipelines with Azure services such as ADLS Gen2, Blob Storage, and event triggers Implement CI/CD for pipelines and workflows using GitHub Actions or equivalent Ensure data quality, governance, and security compliance Collaborate with data science teams for ML/GenAI data readiness Mentor junior engineers and drive best practices for orchestration and pipeline design Required Qualifications 10+ years of experience in data engineering Highly proficient in Databricks (PySpark, Delta Lake), Azure Data Factory (ADF), Azure Logic Apps, and Apache Airflow Strong programming skills in Python and advanced SQL Experience in building orchestrated data platforms with multi-tool integration Strong understanding of ETL/ELT patterns and orchestration frameworks Experience in cloud-native data architectures (Azure preferred) Hands-on experience with monitoring, logging, and pipeline reliability Data Engineer (Databricks +Pyspark) JD + ADF/Logic Apps (Highly Proficient) + Airflow Preferred Qualifications Experience with event-driven architecture and API integrations Exposure to AI/ML data pipelines and MLflow Knowledge of Data Lakehouse architectures and governance frameworks Azure / Databricks certifications, Must Have Skills:Design, build, and maintain scalable data pipelines using Databricks (PySpark), ADF, Logic Apps, and Airflow Build high-performa