Data Engineer Lead

As a Data Engineer, your role involves designing, building, and maintaining scalable data pipelines and enterprise data platforms. You will be responsible for developing robust ETL/ELT workflows, automating data engineering processes, and working on distributed data processing using Spark and large-scale data frameworks. Additionally, you will optimize and manage cloud-based data engineering solutions across AWS, Azure, or GCP, and implement orchestration workflows using Airflow or similar tools. Ensuring data quality, governance, security, and compliance standards will be a crucial part of your responsibilities. Collaboration with analytics, AI/ML, and business teams for data integration and delivery, supporting real-time and batch data processing requirements, troubleshooting data pipeline failures, and optimizing performance are also key aspects of your role. Key Responsibilities: - Design, build, and maintain scalable data pipelines and enterprise data platforms - Develop robust ETL/ELT workflows and automate data engineering processes - Work on distributed data processing using Spark and large-scale data frameworks - Optimize and manage cloud-based data engineering solutions across AWS, Azure, or GCP - Implement orchestration workflows using Airflow or similar tools - Ensure data quality, governance, security, and compliance standards - Collaborate with analytics, AI/ML, and business teams for data integration and delivery - Support real-time and batch data processing requirements - Troubleshoot data pipeline failures and optimize performance - Contribute to data architecture modernization and engineering best practices Qualification Required: - 8+ years of experience in Data Engineering - Strong proficiency in Python and Advanced SQL - Hands-on experience with Spark and distributed processing frameworks - Experience working with cloud platforms such as AWS, Azure, or GCP - Expertise in Airflow or other orchestration tools - Strong understanding of data pipeline automation - Experience with data governance and data quality practices - Excellent analytical, troubleshooting, and problem-solving skills Good to Have: - Experience with Kafka or Kinesis streaming platforms - Knowledge of Feature Stores and ML data infrastructure - Exposure to LLM-ready data pipelines - Understanding of MLOps concepts As part of the candidate profile, you should have a strong understanding of modern data architecture and scalable systems, the ability to work in fast-paced and collaborative environments, experience supporting AI/ML and analytics use cases, and strong communication and stakeholder management skills. As a Data Engineer, your role involves designing, building, and maintaining scalable data pipelines and enterprise data platforms. You will be responsible for developing robust ETL/ELT workflows, automating data engineering processes, and working on distributed data processing using Spark and large-scale data frameworks. Additionally, you will optimize and manage cloud-based data engineering solutions across AWS, Azure, or GCP, and implement orchestration workflows using Airflow or similar tools. Ensuring data quality, governance, security, and compliance standards will be a crucial part of your responsibilities. Collaboration with analytics, AI/ML, and business teams for data integration and delivery, supporting real-time and batch data processing requirements, troubleshooting data pipeline failures, and optimizing performance are also key aspects of your role. Key Responsibilities: - Design, build, and maintain scalable data pipelines and enterprise data platforms - Develop robust ETL/ELT workflows and automate data engineering processes - Work on distributed data processing using Spark and large-scale data frameworks - Optimize and manage cloud-based data engineering solutions across AWS, Azure, or GCP - Implement orchestration workflows using Airflow or similar tools - Ensure data quality, governance, security, and compliance standards - Collaborate with analytics, AI/ML, and business teams for data integration and delivery - Support real-time and batch data processing requirements - Troubleshoot data pipeline failures and optimize performance - Contribute to data architecture modernization and engineering best practices Qualification Required: - 8+ years of experience in Data Engineering - Strong proficiency in Python and Advanced SQL - Hands-on experience with Spark and distributed processing frameworks - Experience working with cloud platforms such as AWS, Azure, or GCP - Expertise in Airflow or other orchestration tools - Strong understanding of data pipeline automation - Experience with data governance and data quality practices - Excellent analytical, troubleshooting, and problem-solving skills Good to Have: - Experience with Kafka or Kinesis streaming platforms - Knowledge of Feature Stores and ML data infrastructure - Exposure to LLM-ready data pipelines - Understanding of MLOps concepts As part