DNAHL_SQL, Databricks, PySpark, Delta lake, Azure Data Factory, Logic Apps, and

Description: - Position Overview - We are seeking a Senior Data Engineer to drive cloud data modernization and build scalable, AI-ready data platforms. This role emphasizes expertise in Databricks, PySpark, Azure Data Factory, Logic Apps, and Airflow, with a strong focus on orchestration, pipeline reliability, and end-to-end data workflow management. - Key Responsibilities - Design, build, and maintain scalable data pipelines using Databricks (PySpark), ADF, Logic Apps, and Airflow - Develop and manage end-to-end orchestration frameworks integrating Airflow DAGs with ADF and Logic Apps - Implement advanced workflow orchestration patterns (event-driven, micro-batch, hybrid scheduling) - Ensure pipeline dependency management, execution reliability, and operational excellence - Build high-performance ETL/ELT pipelines using Databricks and Delta Lake architecture - Implement data observability, monitoring, and alerting mechanisms across workflows - Optimize pipelines and workflows for performance, scalability, and cost efficiency - Integrate pipelines with Azure services such as ADLS Gen2, Blob Storage, and event triggers - Implement CI/CD for pipelines and workflows using GitHub Actions or equivalent - Ensure data quality, governance, and security compliance - Collaborate with data science teams for ML/GenAI data readiness - Mentor junior engineers and drive best practices for orchestration and pipeline design - Required Qualifications - 10+ years of experience in data engineering - Highly proficient in Databricks (PySpark, Delta Lake), Azure Data Factory (ADF), Azure Logic Apps, and Apache Airflow - Strong programming skills in Python and advanced SQL - Experience in building orchestrated data platforms with multi-tool integration - Solid understanding of ETL/ELT patterns and orchestration frameworks - Experience in cloud-native data architectures (Azure preferred) - Hands-on experience with monitoring, logging, and pipeline reliability - Data Engineer (Databricks +Pyspark) JD + ADF/Logic Apps (Highly Proficient) + Airflow - Preferred Qualifications - Experience with event-driven architecture and API integrations - Exposure to AI/ML data pipelines and MLflow - Knowledge of Data Lakehouse architectures and governance frameworks - Azure / Databricks certifications Mandatory skills* SQL, Databricks, PySpark, Delta lake, Azure Data Factory, Logic Apps, and Airflow Description: - Position Overview - We are seeking a Senior Data Engineer to drive cloud data modernization and build scalable, AI-ready data platforms. This role emphasizes expertise in Databricks, PySpark, Azure Data Factory, Logic Apps, and Airflow, with a strong focus on orchestration, pipeline reliability, and end-to-end data workflow management. - Key Responsibilities - Design, build, and maintain scalable data pipelines using Databricks (PySpark), ADF, Logic Apps, and Airflow - Develop and manage end-to-end orchestration frameworks integrating Airflow DAGs with ADF and Logic Apps - Implement advanced workflow orchestration patterns (event-driven, micro-batch, hybrid scheduling) - Ensure pipeline dependency management, execution reliability, and operational excellence - Build high-performance ETL/ELT pipelines using Databricks and Delta Lake architecture - Implement data observability, monitoring, and alerting mechanisms across workflows - Optimize pipelines and workflows for performance, scalability, and cost efficiency - Integrate pipelines with Azure services such as ADLS Gen2, Blob Storage, and event triggers - Implement CI/CD for pipelines and workflows using GitHub Actions or equivalent - Ensure data quality, governance, and security compliance - Collaborate with data science teams for ML/GenAI data readiness - Mentor junior engineers and drive best practices for orchestration and pipeline design - Required Qualifications - 10+ years of experience in data engineering - Highly proficient in Databricks (PySpark, Delta Lake), Azure Data Factory (ADF), Azure Logic Apps, and Apache Airflow - Strong programming skills in Python and advanced SQL - Experience in building orchestrated data platforms with multi-tool integration - Solid understanding of ETL/ELT patterns and orchestration frameworks - Experience in cloud-native data architectures (Azure preferred) - Hands-on experience with monitoring, logging, and pipeline reliability - Data Engineer (Databricks +Pyspark) JD + ADF/Logic Apps (Highly Proficient) + Airflow - Preferred Qualifications - Experience with event-driven architecture and API integrations - Exposure to AI/ML data pipelines and MLflow - Knowledge of Data Lakehouse architectures and governance frameworks - Azure / Databricks certifications Mandatory skills* SQL, Databricks, PySpark, Delta lake, Azure Data Factory, Logic Apps, and Airflow