Senior Data Engineer

As a Data Engineer at Optum, you will play a crucial role in developing, enhancing, and maintaining scalable data pipelines to support enterprise analytics and reporting needs. Your responsibilities will include: - Designing and optimizing data processing workflows using Python, Spark, and Scala, while following data warehousing and data-modeling best practices - Building and supporting real-time and near-real-time data ingestion using Kafka and streaming frameworks - Performing activities related to platform stability, incident resolution, and operational support - Identifying and remediating security vulnerabilities to ensure compliance with enterprise and healthcare data-security standards - Executing tools, framework, and software version upgrades to maintain platform currency and reliability - Creating, managing, and monitoring Azure Data Factory (ADF) pipelines across development, test, and production environments - Supporting and optimizing Databricks workloads for batch and streaming use cases - Managing cloud storage and data movement using Azure Blob Storage and AZ Copy - Collaborating with cross-functional teams to troubleshoot issues, improve performance, and deliver reliable data solutions - Contributing to documentation, knowledge transfer, and continuous improvement of the data platform To be successful in this role, you will need to meet the following qualifications: - Bachelor's degree or equivalent experience - Solid experience with UNIX/Linux environments and scripting - Hands-on experience with IBM DataStage and Teradata - Experience with Apache Spark (batch and/or streaming) - Experience with Databricks and Snowflake in enterprise data platforms - Working knowledge of Airflow for workflow orchestration - Familiarity with GitHub and GitHub Copilot for source control and development productivity - Solid understanding of data warehousing concepts, data modeling, and ETL/ELT best practices - Proficiency in Python for data engineering and automation - Proven excellent communication, analytical, and problem-solving skills Preferred qualifications for the role include: - Experience supporting operations in large-scale data platforms - Hands-on experience with Azure cloud services, especially ADF, Blob Storage, and Databricks - Experience working in regulated or compliance-driven environments - Exposure to Kafka-based streaming architectures - Ability to mentor junior engineers and contribute to platform standardization efforts At Optum, you will have the opportunity to make a meaningful impact on the communities we serve by advancing health optimization on a global scale. Join us in our mission to help people live healthier lives and make the health system work better for everyone. As a Data Engineer at Optum, you will play a crucial role in developing, enhancing, and maintaining scalable data pipelines to support enterprise analytics and reporting needs. Your responsibilities will include: - Designing and optimizing data processing workflows using Python, Spark, and Scala, while following data warehousing and data-modeling best practices - Building and supporting real-time and near-real-time data ingestion using Kafka and streaming frameworks - Performing activities related to platform stability, incident resolution, and operational support - Identifying and remediating security vulnerabilities to ensure compliance with enterprise and healthcare data-security standards - Executing tools, framework, and software version upgrades to maintain platform currency and reliability - Creating, managing, and monitoring Azure Data Factory (ADF) pipelines across development, test, and production environments - Supporting and optimizing Databricks workloads for batch and streaming use cases - Managing cloud storage and data movement using Azure Blob Storage and AZ Copy - Collaborating with cross-functional teams to troubleshoot issues, improve performance, and deliver reliable data solutions - Contributing to documentation, knowledge transfer, and continuous improvement of the data platform To be successful in this role, you will need to meet the following qualifications: - Bachelor's degree or equivalent experience - Solid experience with UNIX/Linux environments and scripting - Hands-on experience with IBM DataStage and Teradata - Experience with Apache Spark (batch and/or streaming) - Experience with Databricks and Snowflake in enterprise data platforms - Working knowledge of Airflow for workflow orchestration - Familiarity with GitHub and GitHub Copilot for source control and development productivity - Solid understanding of data warehousing concepts, data modeling, and ETL/ELT best practices - Proficiency in Python for data engineering and automation - Proven excellent communication, analytical, and problem-solving skills Preferred qualifications for the role include: - Experience supporting operations in large-scale data platforms - Hands-on experience with Azure cloud services, especially