Source description
About the role
As a Senior Data Engineer, you will be responsible for designing, developing, and maintaining scalable ELT/data pipelines using Snowflake. Additionally, you will build and optimize data ingestion, transformation, and orchestration workflows, and develop and maintain Apache Airflow DAGs for scheduling and pipeline automation. Your role will involve writing efficient Python scripts for automation, validation, API integrations, and data processing, along with working on AWS services including S3, Lambda, SNS, CloudWatch, and IAM. You will be expected to develop advanced SQL queries, stored procedures, and data transformation logic, perform production support, troubleshooting, root-cause analysis, and pipeline monitoring. Furthermore, you will implement CI/CD workflows, support database migration activities, and work with healthcare datasets including claims, payer, provider, and pricing data. Ensuring compliance with HIPAA, PHI/PII handling, and healthcare data governance standards is a critical aspect of this role. Collaboration with cross-functional teams to understand business and technical requirements, as well as maintaining technical documentation and operational runbooks, will be part of your responsibilities. Key Responsibilities: - Design, develop, and maintain scalable ELT/data pipelines using Snowflake - Build and optimize data ingestion, transformation, and orchestration workflows - Develop and maintain Apache Airflow DAGs for scheduling and pipeline automation - Write efficient Python scripts for automation, validation, API integrations, and data processing - Work with AWS services including S3, Lambda, SNS, CloudWatch, and IAM - Develop advanced SQL queries, stored procedures, and data transformation logic - Perform production support, troubleshooting, root-cause analysis, and pipeline monitoring - Implement CI/CD workflows and support database migration activities - Work with healthcare datasets including claims, payer, provider, and pricing data - Ensure compliance with HIPAA, PHI/PII handling, and healthcare data governance standards - Collaborate with cross-functional teams to understand business and technical requirements - Maintain technical documentation and operational runbooks Required Qualifications & Skills: - 5+ years of hands-on Data Engineering experience - Strong expertise in Snowflake and ELT pipeline development - Advanced SQL skills including joins, CTEs, window functions, query optimization, and dynamic SQL - Strong Python programming experience - Hands-on experience with Apache Airflow - Experience with AWS services such as S3, Lambda, SNS, CloudWatch, and IAM - Strong understanding of end-to-end data pipeline architecture and orchestration - Experience in troubleshooting, production support, and pipeline optimization - Hands-on experience in Healthcare domain projects - Good understanding of PHI/PII handling and HIPAA compliance - Experience working with claims, payer, provider, or healthcare-related datasets As a Senior Data Engineer, you will be responsible for designing, developing, and maintaining scalable ELT/data pipelines using Snowflake. Additionally, you will build and optimize data ingestion, transformation, and orchestration workflows, and develop and maintain Apache Airflow DAGs for scheduling and pipeline automation. Your role will involve writing efficient Python scripts for automation, validation, API integrations, and data processing, along with working on AWS services including S3, Lambda, SNS, CloudWatch, and IAM. You will be expected to develop advanced SQL queries, stored procedures, and data transformation logic, perform production support, troubleshooting, root-cause analysis, and pipeline monitoring. Furthermore, you will implement CI/CD workflows, support database migration activities, and work with healthcare datasets including claims, payer, provider, and pricing data. Ensuring compliance with HIPAA, PHI/PII handling, and healthcare data governance standards is a critical aspect of this role. Collaboration with cross-functional teams to understand business and technical requirements, as well as maintaining technical documentation and operational runbooks, will be part of your responsibilities. Key Responsibilities: - Design, develop, and maintain scalable ELT/data pipelines using Snowflake - Build and optimize data ingestion, transformation, and orchestration workflows - Develop and maintain Apache Airflow DAGs for scheduling and pipeline automation - Write efficient Python scripts for automation, validation, API integrations, and data processing - Work with AWS services including S3, Lambda, SNS, CloudWatch, and IAM - Develop advanced SQL queries, stored procedures, and data transformation logic - Perform production support, troubleshooting, root-cause analysis, and pipeline monitoring - Implement CI/CD workflows and support database migration activities - Work with healthcare datasets including claims, payer, provider, and pricing data - Ensure
More at TekPillar