Data Platform Engineer

As a Data Platform Engineer at Epergne Solutions, your role involves designing, developing, and maintaining complex data pipelines using Python for efficient data processing and orchestration. You will collaborate with cross-functional teams to understand data requirements and architect robust solutions within the AWS environment. Your responsibilities will include implementing data integration and transformation processes, optimizing existing data pipelines/Apache Airflow, troubleshooting and resolving issues related to data pipelines, and working closely with AWS services such as S3, Glue, EMR, Redshift, and other related technologies to design and optimize data infrastructure. Additionally, you will be responsible for developing and maintaining documentation for data pipelines, processes, and system architecture, as well as staying updated with the latest industry trends and best practices related to data engineering and AWS services. Key Responsibilities: - Design, develop, and maintain complex data pipelines using Python for efficient data processing and orchestration - Collaborate with cross-functional teams to understand data requirements and architect robust solutions within the AWS environment - Implement data integration and transformation processes to ensure optimal performance and reliability of data pipelines - Optimize and fine-tune existing data pipelines/Airflow to improve efficiency, scalability, and maintainability - Troubleshoot and resolve issues related to data pipelines, ensuring smooth operation and minimal downtime - Work closely with AWS services like S3, Glue, EMR, Redshift, and other related technologies to design and optimize data infrastructure - Develop and maintain documentation for data pipelines, processes, and system architecture - Stay updated with the latest industry trends and best practices related to data engineering and AWS services Qualifications Required: - Bachelor's degree in Computer Science, Engineering, or a related field - Proficiency in Python and SQL for data processing and manipulation - Minimum 5 years of experience in data engineering, specifically working with Apache Airflow and AWS technologies - Strong knowledge of AWS services, particularly S3, Glue, EMR, Redshift, and AWS Lambda - Understanding of Snowflake is preferred - Experience with optimizing and scaling data pipelines for performance and efficiency - Good understanding of data modeling, ETL processes, and data warehousing concepts - Excellent problem-solving skills and ability to work in a fast-paced, collaborative environment - Effective communication skills and the ability to articulate technical concepts to non-technical stakeholders Preferred Qualifications: - AWS certification(s) related to data engineering or big data - Experience working with big data technologies like Snowflake, Spark, Hadoop, or related frameworks - Familiarity with other data orchestration tools in addition to Apache Airflow - Knowledge of version control systems like Bitbucket, Git Please note that this is a contract position with Epergne Solutions based in India, offering market standard remuneration. Work mode will be from the office, and preferred candidates are those who can join within 30 days or lesser notice period. As a Data Platform Engineer at Epergne Solutions, your role involves designing, developing, and maintaining complex data pipelines using Python for efficient data processing and orchestration. You will collaborate with cross-functional teams to understand data requirements and architect robust solutions within the AWS environment. Your responsibilities will include implementing data integration and transformation processes, optimizing existing data pipelines/Apache Airflow, troubleshooting and resolving issues related to data pipelines, and working closely with AWS services such as S3, Glue, EMR, Redshift, and other related technologies to design and optimize data infrastructure. Additionally, you will be responsible for developing and maintaining documentation for data pipelines, processes, and system architecture, as well as staying updated with the latest industry trends and best practices related to data engineering and AWS services. Key Responsibilities: - Design, develop, and maintain complex data pipelines using Python for efficient data processing and orchestration - Collaborate with cross-functional teams to understand data requirements and architect robust solutions within the AWS environment - Implement data integration and transformation processes to ensure optimal performance and reliability of data pipelines - Optimize and fine-tune existing data pipelines/Airflow to improve efficiency, scalability, and maintainability - Troubleshoot and resolve issues related to data pipelines, ensuring smooth operation and minimal downtime - Work closely with AWS services like S3, Glue, EMR, Redshift, and other related technologies to design and optimize data infrastructure - Develop and maint