Source description
About the role
Design, develop, and maintain scalable data pipelines using Azure Databricks and PySpark. Build ETL/ELT workflows to ingest, transform, and load data from multiple sources into Azure Data Lake and Azure SQL. Optimize Spark jobs for performance, scalability, and cost efficiency. Work with Delta Lake to implement reliable, high-performance data storage and processing. Integrate Azure Databricks with Azure Data Factory, Azure Synapse Analytics, and other Azure services. Monitor data pipelines, troubleshoot issues, and ensure high data quality and reliability. Collaborate with data engineers, analysts, and business stakeholders to deliver data solutions. Implement best practices for data governance, security, and CI/CD using Azure DevOps. Create technical documentation and support production deployments. Key Skills: Azure Databricks, PySpark, Apache Spark, Python, SQL, Delta Lake, Azure Data Factory (ADF), Azure Data Lake Storage (ADLS), Azure Synapse Analytics, Azure DevOps, Git, ETL/ELT, Data Warehousing.
More at Elabs Infotech