Lead-Data Bricks

Key Responsibilities Data Pipeline Development • Design and develop scalable data pipelines using Azure Databricks, Python, and PySpark. • Implement ETL/ELT workflows for structured, semi-structured, and unstructured data. • Develop and maintain data processing workflows using Delta Lake architecture. • Ensure efficient data ingestion, transformation, and loading processes. Data Management & Optimisation • Optimise performance of data pipelines and processing workloads. • Ensure data quality, reliability, and consistency across data platforms. • Write and optimise complex SQL queries for data processing and validation. • Implement data governance and access controls. Cloud & Data Platform Integration • Work with Azure Data Lake for storing and managing large datasets. • Manage data cataloging and governance using Unity Catalog. • Integrate data workflows with CI/CD pipelines for automated deployments. • Support scalable and secure cloud-based data architectures. Collaboration & Migration Projects • Work closely with cross-functional teams to deliver data-driven solutions. • Participate in data migration and modernisation initiatives. • Support troubleshooting and resolution of data pipeline issues. • Document data architecture, pipelines, and processes. Required Skills & Experience Must have 7 years of IT experience and at least 5+ years of hands-on experience in Databricks. Design and develop scalable data pipelines using Azure Databricks, SCALA, Python, PySpark, and Delta Lake. Have a good understanding of RDBMS and proficiency in writing complex SQL (1st Preference – Oracle / PLSQL). Implement ETL/ELT workflows for structured, semi-structured, and unstructured data. Optimise performance of data processing and ensure data quality standards. Work with Azure Data Lake, Unity Catalogue, and integrate with CI/CD pipelines. Experience with orchestration tools like Airflow or Azure Data Factory is a plus. Have working experience on a migration project. Relevant Experience Preferred • Hands-on experience developing data pipelines using Azure Databricks and PySpark. • Experience implementing ETL/ELT solutions for enterprise data platforms. • Experience optimizing data processing performance and data workflows. • Experience working with Azure Data Lake and cloud-based data architectures. • Experience working on data migration or modernization projects. • Experience integrating data pipelines with CI/CD workflows Qualification : Any degree equivalent to IT or relevant