Senior Associate/Manager Data Engineer Consultant

Role Overview: As a Data Engineer Consultant at EY AI & Data, you will be responsible for designing, implementing, and optimizing robust data infrastructure and pipelines. You will work with cutting-edge AWS and Databricks technologies, providing technical guidance to team members and collaborating with stakeholders on complex data engineering challenges. Key Responsibilities: - Design and implement enterprise-scale data architectures, including data lakes, warehouses, and real-time streaming platforms. - Develop and maintain ETL/ELT pipelines to process large volumes of structured and unstructured data efficiently. - Ensure data quality, governance, and security standards are maintained throughout all data engineering processes. - Hands-on development of Databricks notebooks using PySpark, Python, and SQL for ETL automation. - Create and optimize PySpark scripts for efficient data extraction, transformation, and loading from large datasets. - Provide technical mentorship to junior data engineers and analysts, lead code reviews, and establish best practices. - Monitor and optimize data pipeline performance, implement solutions for scalability and cost-effectiveness, and troubleshoot technical issues. Qualification Required: - Bachelor's Degree in Computer Science, Engineering, Data Science, or related field OR Master's Degree in Data Engineering, Computer Science, or MBA (preferred). - 5-7 years of data engineering experience with AWS-native implementations. - Hands-on experience with implementing Databricks notebooks using PySpark, Python, and SQL for ETL automation. - Proficiency in Python for custom data manipulation, validation, and error handling in ETL processes. - Experience in SQL for complex joins, aggregations, and database operations within Databricks environments. - Track record of testing, debugging, and optimizing data transformation processes for accuracy and performance. Additional Company Details: EY is committed to creating a better working world by providing new value for clients, people, society, and the planet, while building trust in capital markets. Using data, AI, and advanced technology, EY teams help clients shape the future with confidence and develop solutions for pressing current and future issues. With a globally connected network, EY offers services in assurance, consulting, tax, strategy, and transactions across 150+ countries and territories. Role Overview: As a Data Engineer Consultant at EY AI & Data, you will be responsible for designing, implementing, and optimizing robust data infrastructure and pipelines. You will work with cutting-edge AWS and Databricks technologies, providing technical guidance to team members and collaborating with stakeholders on complex data engineering challenges. Key Responsibilities: - Design and implement enterprise-scale data architectures, including data lakes, warehouses, and real-time streaming platforms. - Develop and maintain ETL/ELT pipelines to process large volumes of structured and unstructured data efficiently. - Ensure data quality, governance, and security standards are maintained throughout all data engineering processes. - Hands-on development of Databricks notebooks using PySpark, Python, and SQL for ETL automation. - Create and optimize PySpark scripts for efficient data extraction, transformation, and loading from large datasets. - Provide technical mentorship to junior data engineers and analysts, lead code reviews, and establish best practices. - Monitor and optimize data pipeline performance, implement solutions for scalability and cost-effectiveness, and troubleshoot technical issues. Qualification Required: - Bachelor's Degree in Computer Science, Engineering, Data Science, or related field OR Master's Degree in Data Engineering, Computer Science, or MBA (preferred). - 5-7 years of data engineering experience with AWS-native implementations. - Hands-on experience with implementing Databricks notebooks using PySpark, Python, and SQL for ETL automation. - Proficiency in Python for custom data manipulation, validation, and error handling in ETL processes. - Experience in SQL for complex joins, aggregations, and database operations within Databricks environments. - Track record of testing, debugging, and optimizing data transformation processes for accuracy and performance. Additional Company Details: EY is committed to creating a better working world by providing new value for clients, people, society, and the planet, while building trust in capital markets. Using data, AI, and advanced technology, EY teams help clients shape the future with confidence and develop solutions for pressing current and future issues. With a globally connected network, EY offers services in assurance, consulting, tax, strategy, and transactions across 150+ countries and territories.