Data Engineer-5-8Yrs-Delhi/NCR

Responsibilities • Design, Develop and maintain ETL pipelines using Pyspark in Azure Databricks using delta tables. • Create build from Github and release pipeline for Ingestion and Databricks using Azure Devops / Harness • Monitor Performance of ETL Jobs, resolve any issue that arose and improve the performance metrics as needed. • Diagnose system performance issue related to data processing and implement solution to address them. • Collaborate with other teams to ensure successful integration of data pipelines into larger system architecture requirement. • Maintain integrity and quality across all pipelines and environments. • Understand and follow secure coding practice to make sure code is not vulnerable In this role, you will report to the Data Engineering Team Lead. Skills • Should have Strong background in Software development with experience in ingest, transform and store data from large datasets using Pyspark in Azure Databricks with strong knowledge on distributed computing concepts. • Must have hands on experience in designing and developing ETL Pipelines in Pyspark in Azure Databricks with strong python scripting exposure like list comprehensions, Dictionary variables etc. • Must have minimum 5 years of exposure and good proficiency in data warehousing concepts. • Proficient in SQl and database Design concepts. EXL Service 10 Exchange Place, Suite 2200, Jersey City, NJ T: +1.201.748.4700 www.exlservice.com • Hands on experience with good proficiency in Delta table and delta file operations like merge, Insert override, Partition overrides etc. • Hands on Experience in CICD in Azure DevOps/Harness and ADF/Stone branch for orchestration. • Knowledge of Azure cloud computing platform with Azure Synapse, ADLS. • Knowledge on GitHub and build management. • Passion for data and experience working within a data driven organization • Excellent presentation, communication (oral & written), and relationship building skills, across all levels of management