Source description
About the role
Job Purpose Seeking a highly skilled Senior Data Engineer to design, build, and maintain scalable data platforms, ETL processes, and AI-enabled data solutions. The ideal candidate will have strong expertise in modern data engineering practices, cloud-native technologies, DevOps automation, and generative AI integrations. This role requires hands-on experience developing robust data pipelines, managing containerized applications, and supporting enterprise-grade data infrastructure. Key Responsibilities Design, develop, and maintain scalable ETL workflows and data pipelines that support business-critical applications and analytics. Build and manage data orchestration workflows using Apache Airflow. Develop high-quality, production-ready solutions using Python. Implement and maintain CI/CD pipelines leveraging GitHub Actions. Manage source control, branching strategies, and code reviews through GitHub repositories. Deploy, monitor, and optimize applications running on Azure Kubernetes Service (AKS). Develop and maintain containerized solutions using Docker. Integrate and support Generative AI solutions utilizing OpenAI and Anthropic Claude models. Collaborate with data scientists, ML engineers, and business stakeholders to deliver reliable data products. Ensure data quality, monitoring, observability, security, and governance standards are met. Troubleshoot production issues and drive continuous improvement across data engineering platforms. Document architecture, processes, and best practices for enterprise data solutions. Key competencies Essential Skills Bachelor's or master's degree in computer science, Engineering, Information Technology, or a related field. 7–10 years of experience in data engineering, software engineering, or related technical disciplines. Strong hands-on experience with: Python GitHub and Git-based development workflows GitHub Actions CI/CD Azure Kubernetes Service (AKS) Apache Airflow Docker Building and maintaining enterprise-scale ETL processes and data pipelines Experience designing and operating cloud-native, distributed data platforms. Strong understanding of software engineering best practices, testing frameworks, and deployment automation. Experience working with OpenAI and Claude (Anthropic) models in production environments. Experience working with REST APIs and external data integrations. Excellent problem-solving, debugging, and performance optimization skills. Strong communication and stakeholder management capabilities. Nice to Have Experience with S&P Xpressfeed data platforms. Experience with Snowflake data warehousing and analytics solutions. Familiarity with data governance, data quality frameworks, and monitoring tools. Exposure to machine learning and AI/ML operations (MLOps) environments. Experience working in Agile/Scrum development teams Skills: Data Engineering: ETL, Data Pipelines, Data Integration, Data Orchestration, Data Quality Cloud & DevOps: Azure Kubernetes Service (AKS), Docker, GitHub, GitHub Actions, CI/CD Programming: Python Workflow Orchestration: Apache Airflow AI & Generative AI: OpenAI, Claude, LLM Integrations Preferred Data Platforms: Snowflake, S&P Xpressfeed
More at Acuity Analytics