Source description
About the role
- Design, develop, and maintain scalable data pipelines using Azure Databricks and PySpark. - Build ETL/ELT workflows to ingest, transform, and load data from multiple sources into Azure Data Lake and Azure SQL. - Optimize Spark jobs for performance, scalability, and cost efficiency. - Work with Delta Lake to implement reliable, high-performance data storage and processing. - Integrate Azure Databricks with Azure Data Factory, Azure Synapse Analytics, and other Azure services. - Monitor data pipelines, troubleshoot issues, and ensure high data quality and reliability. - Collaborate with data engineers, analysts, and business stakeholders to deliver data solutions. - Implement best practices for data governance, security, and CI/CD using Azure DevOps. - Create technical documentation and support production deployments. Key Skills: Azure Databricks, PySpark, Apache Spark, Python, SQL, Delta Lake, Azure Data Factory (ADF), Azure Data Lake Storage (ADLS), Azure Synapse Analytics, Azure DevOps, Git, ETL/ELT, Data Warehousing. .
More at Elabs Infotech