Senior Data Engineer - Azure Platforms

Role Overview: You will be the Azure Senior Data Engineer at EY, utilizing your expertise in data ingestion, processing, and storage to design and implement scalable data pipelines. Your role will involve working with various data sources and technologies in a cloud environment to ensure data quality and optimize storage solutions. Key Responsibilities: - Design, develop, and maintain scalable data pipelines using Python, Spark, and Azure for data ingestion and processing. - Utilize on-prem Oracle databases, batch files, and Confluent Kafka for data sourcing. - Implement ETL processes with AWS Glue and EMR for batch and streaming data. - Develop and maintain data storage solutions in S3, Redshift, and Oracle using Medallion Architecture. - Collaborate with cross-functional teams to understand data requirements and deliver solutions that align with business needs. - Monitor and optimize data workflows using tools like Airflow for orchestration. - Ensure data quality and integrity throughout the data lifecycle. - Implement CI/CD practices for data pipeline deployment with tools like Terraform. - Utilize monitoring and logging tools such as CloudWatch, Datadog, and Splunk to ensure system reliability and performance. - Communicate effectively with stakeholders to gather requirements and provide project status updates. Qualifications Required: - Proficiency in Python for data processing and automation. - Strong experience with Apache Spark for large-scale data processing. - Familiarity with Azure for data storage and management. - Experience with Python Pyspark and Databricks for real-time data streaming. - Knowledge of Redshift for data warehousing solutions. - Proficiency in Oracle databases for data management. - Experience with AWS Glue for ETL processes. - Familiarity with Apache Airflow for workflow orchestration. - Experience with EMR for big data processing. - Strong AWS data engineering skills are mandatory. Additional Company Details: EY is focused on building a better working world by creating new value for clients, people, society, and the planet. Through the use of data, AI, and advanced technology, EY teams help clients shape the future with confidence and address current and future challenges effectively. EY provides a wide range of services in assurance, consulting, tax, strategy, and transactions across more than 150 countries and territories, supported by a globally connected, multi-disciplinary network and diverse ecosystem partners. Role Overview: You will be the Azure Senior Data Engineer at EY, utilizing your expertise in data ingestion, processing, and storage to design and implement scalable data pipelines. Your role will involve working with various data sources and technologies in a cloud environment to ensure data quality and optimize storage solutions. Key Responsibilities: - Design, develop, and maintain scalable data pipelines using Python, Spark, and Azure for data ingestion and processing. - Utilize on-prem Oracle databases, batch files, and Confluent Kafka for data sourcing. - Implement ETL processes with AWS Glue and EMR for batch and streaming data. - Develop and maintain data storage solutions in S3, Redshift, and Oracle using Medallion Architecture. - Collaborate with cross-functional teams to understand data requirements and deliver solutions that align with business needs. - Monitor and optimize data workflows using tools like Airflow for orchestration. - Ensure data quality and integrity throughout the data lifecycle. - Implement CI/CD practices for data pipeline deployment with tools like Terraform. - Utilize monitoring and logging tools such as CloudWatch, Datadog, and Splunk to ensure system reliability and performance. - Communicate effectively with stakeholders to gather requirements and provide project status updates. Qualifications Required: - Proficiency in Python for data processing and automation. - Strong experience with Apache Spark for large-scale data processing. - Familiarity with Azure for data storage and management. - Experience with Python Pyspark and Databricks for real-time data streaming. - Knowledge of Redshift for data warehousing solutions. - Proficiency in Oracle databases for data management. - Experience with AWS Glue for ETL processes. - Familiarity with Apache Airflow for workflow orchestration. - Experience with EMR for big data processing. - Strong AWS data engineering skills are mandatory. Additional Company Details: EY is focused on building a better working world by creating new value for clients, people, society, and the planet. Through the use of data, AI, and advanced technology, EY teams help clients shape the future with confidence and address current and future challenges effectively. EY provides a wide range of services in assurance, consulting, tax, strategy, and transactions across more than 150 countries and territories, supported by a globally connected, multi-disciplinary network and diverse ecosystem partners.