Source description
About the role
RequirementsHave 10+ years of experience in the design, architecture, implementation, and optimization of data engineering solutions for large volumes (TB, PB scale) of data.Have expertise in designing and implementing end-to-end data architectures using Azure Databricks, including data ingestion pipelines, transformation logic, and data warehousing strategies for large-scale batch and real-time processing.Proven expertise in Azure services such as Azure Databricks, Azure Data Factory (ADF), ADLS, Azure Synapse, Azure SQL Database, and Azure Functions, and experience building scalable data lakes and pipelines.Strong handson experience processing large volumes of data with proficiency in PySpark, Python, Spark SQL, and workflow automation.Experience implementing robust data governance using Unity Catalog, along with security measures.Proficiency in requirements analysis, solution design, development, testing, deployment, and ongoing support, including cloud migration projects for large-scale data platforms.Handson exposure to configuring and managing Azure Databricks clusters, workspaces, and Unity Catalog for optimal performance, cost management, and scalability of terabyte-scale datasets.Good exposure to writing optimized SQL queries.Strong communication and problemsolving skills.Ability to create proofofconcepts, contribute to proposals, and participate in RFPs.Understanding of GenAI technologies and the ability to implement solutions using GenAI. .
More at Impetus