Source description
About the role
Responsibilities- Develop, maintain, and optimize ETL/ELT pipelines using Databricks, PySpark, and SQL . Create, schedule, monitor, and troubleshoot Databricks Jobs, Workflows, and Pipelines . Design and implement scalable data processing solutions for large datasets. Perform data cleansing, transformation, validation, and quality checks. Collaborate with business and analytics teams to understand requirements and deliver data solutions. Support production deployments, monitoring, and issue resolution. Follow best practices for code versioning, documentation, and performance optimization. Educational Qualifications Bachelor's degree in Computer Science, Information Technology, Engineering, or a related field. Competencies Required: 2–3 years of experience in Data Engineering. Strong proficiency in SQL and PySpark . Hands-on experience with Azure Databricks . Experience in creating and managing Databricks Jobs, Workflows, and Pipelines . Good understanding of ETL/ELT processes and data warehousing concepts. Familiarity with Git/version control and basic CI/CD practices. Strong analytical and problem-solving skills. Experience - Experience with Azure Data Factory (ADF), ADLS, Synapse, or other Azure services . Exposure to Delta Lake and Lakehouse architecture. Knowledge of performance tuning and optimization of Spark workloads.
More at ICICI Lombard