Source description
About the role
Experience: 58 Years Location: Bangalore / Hyderabad / Chennai / Pune We are looking for an experienced Azure Data Engineer with strong expertise in Python, PySpark, Azure Databricks, and Azure Cloud Services. The ideal candidate should have hands-on experience designing and developing scalable data pipelines, processing large datasets, and supporting enterprise-grade data platforms. The role involves working closely with business stakeholders, building robust ETL solutions, and maintaining production data pipelines in a DevOps environment. Key Responsibilities - Design, develop, and optimize scalable data pipelines using Python, PySpark, and Azure Databricks. - Build and maintain batch and streaming data processing solutions. - Develop ETL/ELT workflows to ingest data from multiple structured and unstructured data sources. - Work with business and technical teams to gather requirements and translate them into data engineering solutions. - Develop data lake and data warehouse solutions using Azure services. - Monitor, troubleshoot, and support production data pipelines. - Collaborate with DevOps teams to automate deployments using CI/CD pipelines. - Implement logging, monitoring, and performance tuning for data processing jobs. - Ensure data quality, security, and governance best practices. - Participate in code reviews, documentation, and technical design discussions. Required Skills - 5+ years of experience in Data Engineering. - Strong programming skills in Python. - Hands-on experience with PySpark and Azure Databricks. - Experience developing both Batch and Structured Streaming pipelines. - Strong knowledge of SQL and exposure to NoSQL databases. - Hands-on experience with Azure Cloud services, including: - Azure Data Lake Storage (ADLS Gen2) - Azure Databricks - Azure Event Hub - Azure Event Grid - Azure SQL Data Warehouse / Azure Synapse Analytics - Azure Functions - Azure Key Vault - Good understanding of Serverless Architecture. - Knowledge of Infrastructure as Code using Terraform. - Experience with Azure DevOps and CI/CD pipelines. - Strong debugging, troubleshooting, and performance optimization skills. Preferred Skills - Azure Data Factory (ADF) - ARM Templates - Kubernetes (AKS) - Google Cloud Platform (GCP) - Machine Learning or AI exposure - Experience with data lake and modern cloud data platform architectures Preferred Candidate Profile - Robust analytical and problem-solving skills. - Excellent communication and stakeholder management abilities. - Experience working in Agile/Scrum environments. - Comfortable working in a DevOps model involving both development and production support. - Ability to independently design and deliver scalable data engineering solutions. Qualifications - Bachelor's or Master's degree in Computer Science, Information Technology, Engineering, or a related field. Skills: pyspark,azure,nosql,python Experience: 58 Years Location: Bangalore / Hyderabad / Chennai / Pune We are looking for an experienced Azure Data Engineer with strong expertise in Python, PySpark, Azure Databricks, and Azure Cloud Services. The ideal candidate should have hands-on experience designing and developing scalable data pipelines, processing large datasets, and supporting enterprise-grade data platforms. The role involves working closely with business stakeholders, building robust ETL solutions, and maintaining production data pipelines in a DevOps environment. Key Responsibilities - Design, develop, and optimize scalable data pipelines using Python, PySpark, and Azure Databricks. - Build and maintain batch and streaming data processing solutions. - Develop ETL/ELT workflows to ingest data from multiple structured and unstructured data sources. - Work with business and technical teams to gather requirements and translate them into data engineering solutions. - Develop data lake and data warehouse solutions using Azure services. - Monitor, troubleshoot, and support production data pipelines. - Collaborate with DevOps teams to automate deployments using CI/CD pipelines. - Implement logging, monitoring, and performance tuning for data processing jobs. - Ensure data quality, security, and governance best practices. - Participate in code reviews, documentation, and technical design discussions. Required Skills - 5+ years of experience in Data Engineering. - Strong programming skills in Python. - Hands-on experience with PySpark and Azure Databricks. - Experience developing both Batch and Structured Streaming pipelines. - Strong knowledge of SQL and exposure to NoSQL databases. - Hands-on experience with Azure Cloud services, including: - Azure Data Lake Storage (ADLS Gen2) - Azure Databricks - Azure Event Hub - Azure Event Grid - Azure SQL Data Warehouse / Azure Synapse Analytics - Azure Functions - Azure Key Vault - Good understanding of Serverless Architecture. - Knowledge of Infrastructure as Code using Terr
More at Zorba AI