Senior Data Engineer

Note: - This is 6 Months Contractual role based on projects chances of extension is 90% for 12 months further with 100% remote opportunity. We are looking for a highly experienced Azure Databricks Data Engineer with 8+ years of overall experience and strong hands-on expertise in Azure Databricks, PySpark, Databricks Asset Bundles (DAB), Azure DevOps, and CI/CD. The ideal candidate will be responsible for designing and developing scalable data engineering solutions, implementing robust ETL/ELT pipelines, and automating Databricks deployments across multiple environments using DevOps best practices. Key Responsibilities Design, develop, and maintain enterprise-scale data pipelines using Azure Databricks and PySpark. Develop scalable ETL/ELT solutions using PySpark, Spark SQL, and SQL. Build and optimize Delta Lake tables and Lakehouse data processing workflows. Implement and manage Databricks Asset Bundles (DAB) for packaging, configuration, and deployment of Databricks resources. Design and maintain Azure DevOps CI/CD pipelines for Databricks deployments. Automate deployment of Databricks notebooks, jobs, workflows, and configurations across Dev, QA, UAT, and Production environments. Integrate Databricks development with Git, GitHub, or Azure Repos. Implement environment-specific configurations and deployment strategies. Work with Azure Data Lake Storage Gen2, Azure Key Vault, Azure Data Factory, and other Azure services. Develop reusable PySpark frameworks and production-ready data processing solutions. Perform Spark performance tuning, including optimization of partitions, joins, caching, file sizes, and Delta Lake operations. Implement data quality, validation, logging, monitoring, and error-handling frameworks. Troubleshoot complex production issues and provide technical solutions. Collaborate with architects, data engineers, DevOps teams, and business stakeholders. Provide technical guidance and mentoring to junior and mid-level engineers. Required Skills Must Have 8+ years of overall experience in Data Engineering / Big Data / Cloud Data Engineering. Strong hands-on experience with Azure Databricks. Strong expertise in PySpark and Apache Spark. Hands-on experience with Databricks Asset Bundles (DAB). Strong experience with Azure DevOps and CI/CD pipelines. Experience automating Databricks deployments across multiple environments. Strong knowledge of Delta Lake and Lakehouse architecture. Strong SQL skills. Experience with Git / GitHub / Azure Repos. Experience with Azure Data Lake Storage Gen2 (ADLS Gen2). Strong understanding of ETL/ELT concepts and modern data engineering practices.