Data Engineer (Pyspark +Databricks+AWS ) (Pune)

Skill: Data Engineer (Databricks + Pyspark) Loc: Pune Mode: 5 days WFO Shift: UK shift 2:30 to 11 PM Exp: 8-13 Years JD We are looking for a skilled Databricks Data Engineer to design, develop, and optimize scalable data pipelines and data platforms. The ideal candidate will have strong expertise in Databricks, SQL, Python, and PySpark , along with experience in implementing CI/CD practices for data engineering workflows. Primary Skills (Mandatory) 1. Databricks - Strong hands-on experience with Databricks (Data Lakehouse architecture) - Expertise in Delta Lake, Unity Catalog, and Databricks Workflows - Experience in performance tuning and cost optimization 1. SQL - Advanced SQL skills for data transformation, optimization, and querying - Experience in writing complex joins, window functions, and performance tuning 1. Python - Proficient in Python for data engineering tasks and scripting - Experience with modular code development and reusable components 1. PySpark - Strong experience in building scalable data pipelines using PySpark - Knowledge of Spark optimization techniques (partitioning, caching, etc.) 1. CI/CD - Experience implementing CI/CD pipelines for data workflows - Familiarity with tools such as Azure DevOps, GitHub Actions, Jenkins, or similar - Knowledge of version control (Git) and automated deployment of data solutions Secondary Skills - AWS - Experience working with AWS services such as S3, EC2, IAM, and EMR - Apache Airflow - Experience in workflow orchestration and scheduling using Airflow - AWS Glue - Hands-on experience in ETL processing and Glue jobs Positive to Have - Terraform - Knowledge of Infrastructure as Code (IaC) for provisioning cloud resources - Experience in automation and environment setup using Terraform Key Responsibilities - Design and develop scalable data pipelines using Databricks and PySpark - Build and optimize ETL/ELT processes for large-scale data processing - Collaborate with data architects and stakeholders to define data models and solutions - Implement CI/CD pipelines for automated deployment and testing - Monitor, troubleshoot, and optimize data workflows for performance and cost - Ensure data quality, governance, and security standards are maintained - Work closely with cross-functional teams across data, analytics, and business domains Required Experience - 8+ years of experience in data engineering or related roles - Hands-on experience in Databricks-based data platform projects - Strong understanding of distributed data processing concepts Skill: Data Engineer (Databricks + Pyspark) Loc: Pune Mode: 5 days WFO Shift: UK shift 2:30 to 11 PM Exp: 8-13 Years JD We are looking for a skilled Databricks Data Engineer to design, develop, and optimize scalable data pipelines and data platforms. The ideal candidate will have strong expertise in Databricks, SQL, Python, and PySpark , along with experience in implementing CI/CD practices for data engineering workflows. Primary Skills (Mandatory) 1. Databricks - Strong hands-on experience with Databricks (Data Lakehouse architecture) - Expertise in Delta Lake, Unity Catalog, and Databricks Workflows - Experience in performance tuning and cost optimization 1. SQL - Advanced SQL skills for data transformation, optimization, and querying - Experience in writing complex joins, window functions, and performance tuning 1. Python - Proficient in Python for data engineering tasks and scripting - Experience with modular code development and reusable components 1. PySpark - Strong experience in building scalable data pipelines using PySpark - Knowledge of Spark optimization techniques (partitioning, caching, etc.) 1. CI/CD - Experience implementing CI/CD pipelines for data workflows - Familiarity with tools such as Azure DevOps, GitHub Actions, Jenkins, or similar - Knowledge of version control (Git) and automated deployment of data solutions Secondary Skills - AWS - Experience working with AWS services such as S3, EC2, IAM, and EMR - Apache Airflow - Experience in workflow orchestration and scheduling using Airflow - AWS Glue - Hands-on experience in ETL processing and Glue jobs Positive to Have - Terraform - Knowledge of Infrastructure as Code (IaC) for provisioning cloud resources - Experience in automation and environment setup using Terraform Key Responsibilities - Design and develop scalable data pipelines using Databricks and PySpark - Build and optimize ETL/ELT processes for large-scale data processing - Collaborate with data architects and stakeholders to define data models and solutions - Implement CI/CD pipelines for automated deployment and testing - Monitor, troubleshoot, and optimize data workflows for performance and cost - Ensure data quality, governance, and security standards are maintained - Work closely with cross-functional teams across data, analytics, and business domains Required Experience - 8+ years of experience in data engineering or related ro