Senior Data Engineer - Delivery Lead

Job purpose Seeking a highly skilled Senior Data Engineer to design, build, and maintain scalable data platforms, ETL processes, and AI-enabled data solutions. The ideal candidate will have strong expertise in modern data engineering practices, cloud-native technologies, DevOps automation, and generative AI integrations. This role requires hands-on experience developing robust data pipelines, managing containerized applications, and supporting enterprise-grade data infrastructure. Key responsibilities • Design, develop, and maintain scalable ETL workflows and data pipelines that support business-critical applications and analytics. • Build and manage data orchestration workflows using Apache Airflow. • Develop high-quality, production-ready solutions using Python. • Implement and maintain CI/CD pipelines leveraging GitHub Actions. • Manage source control, branching strategies, and code reviews through GitHub repositories. • Deploy, monitor, and optimize applications running on Azure Kubernetes Service (AKS). • Develop and maintain containerized solutions using Docker. • Integrate and support Generative AI solutions utilizing OpenAI and Anthropic Claude models. • Collaborate with data scientists, ML engineers, and business stakeholders to deliver reliable data products. • Ensure data quality, monitoring, observability, security, and governance standards are met. • Troubleshoot production issues and drive continuous improvement across data engineering platforms. • Document architecture, processes, and best practices for enterprise data solutions. Key competencies Essential skills • Bachelor’s or master’s degree in computer science, Engineering, Information Technology, or a related field. • 7–10 years of experience in data engineering, software engineering, or related technical disciplines. • Strong hands-on experience with: o Python o GitHub and Git-based development workflows o GitHub Actions CI/CD o Azure Kubernetes Service (AKS) o Apache Airflow o Docker o Building and maintaining enterprise-scale ETL processes and data pipelines • Experience designing and operating cloud-native, distributed data platforms. • Strong understanding of software engineering best practices, testing frameworks, and deployment automation. • Experience working with OpenAI and Claude (Anthropic) models in production environments. • Experience working with REST APIs and external data integrations. • Excellent problem-solving, debugging, and performance optimization skills. • Strong communication and stakeholder management capabilities. Nice to Have • Experience with S&P Xpressfeed data platforms. • Experience with Snowflake data warehousing and analytics solutions. • Familiarity with data governance, data quality frameworks, and monitoring tools. • Exposure to machine learning and AI/ML operations (MLOps) environments. • Experience working in Agile/Scrum development teams Skills: • Data Engineering: ETL, Data Pipelines, Data Integration, Data Orchestration, Data Quality • Cloud & DevOps: Azure Kubernetes Service (AKS), Docker, GitHub, GitHub Actions, CI/CD • Programming: Python • Workflow Orchestration: Apache Airflow • AI & Generative AI: OpenAI, Claude, LLM Integrations • Preferred Data Platforms: Snowflake, S&P Xpressfeed