Lead Data Engineer

Roles and Responsibilities : Design, develop, test, deploy and maintain large-scale data pipelines using Azure Data Factory (ADF) to integrate various data sources into a centralized platform. Collaborate with cross-functional teams to gather requirements for new data pipeline projects and ensure successful implementation. Develop complex SQL queries to optimize database performance tuning on Azure Databricks for improved query execution times. Troubleshoot issues related to ADF workflows, including errors in data processing and debugging of complex logic. Job Requirements : 10-14 years of experience in Data Engineering with expertise in Azure Data Factory (ADF). Strong understanding of big data technologies such as Hadoop, Spark, Kafka etc. . Proficiency in writing complex SQL queries for performance tuning on Azure Databricks.