Source description
About the role
Job Role: Senior Data Engineer Experience: 8+ Years Location: Pune Key Responsibilities Design and develop scalable ETL and data processing pipelines on AWS. Build batch and real-time data pipelines using PySpark, Spark Streaming, and Kafka . Develop and optimize Data Lake solutions using Apache Hudi or Apache Iceberg . Build and manage data processing workflows using AWS Glue and Amazon EMR . Automate infrastructure deployment using Terraform and containerize applications using Docker and Amazon ECS . Optimize large-scale distributed data processing for performance, scalability, and reliability. Implement data cataloging and metadata management using AWS Glue Data Catalog . Collaborate with cross-functional teams to deliver secure, high-quality data engineering solutions. Troubleshoot production issues and continuously improve data platform performance. Key Skills 8+ years of experience in Data Engineering. Strong expertise in AWS Glue , PySpark , Amazon EMR , and Spark Streaming . Hands-on experience with Apache Kafka for real-time data streaming. Experience with Apache Hudi or Apache Iceberg . Strong programming skills in Python and Object-Oriented Programming (OOPS). Experience with AWS Glue ETL and Glue Data Catalog . Hands-on experience with Docker , Amazon ECS , and Terraform .
More at TekPillar