Senior Big Data Engineer

As a Senior Big Data Engineer at dunnhumby, you will have the opportunity to extend and enhance the Data Engineering Team, working with a market-leading business to explore new opportunities and influence global retailers. You will be responsible for designing end-to-end data solutions, architecting scalable data pipelines, orchestrating automation, ensuring data governance and quality standards, and leading technical design sessions. Your technical expertise will include a Bachelor's or Master's degree in computer science or related field, extensive experience with programming languages like Python, Java, or Scala, proficiency in data pipeline tools such as Apache Spark, Kafka, and Airflow, and knowledge of cloud platforms like Azure or Google Cloud. You will also have experience with data governance frameworks, Agile or DevOps environments, modern data stack tools, Git, and process automation, along with a strong understanding of relational database management systems and data flow development. Key Responsibilities: - Design end-to-end data solutions, including data lakes, data warehouses, ETL/ELT pipelines, APIs, and analytics platforms. - Architect scalable and low-latency data pipelines using tools like Apache Kafka, Flink, or Spark Streaming to handle high-velocity data streams. - Design /Orchestrate end-to-end automation using orchestration frameworks such as Apache Airflow to manage complex workflows and dependencies. - Design intelligent systems that can detect anomalies, trigger alerts, and automatically reroute or restart processes to maintain data integrity and availability. - Define and implement data governance, metadata management, and data quality standards. - Lead architectural reviews and technical design sessions to guide solution development. - Partner with business and IT teams to translate business needs into data architecture requirements. - Ensure security, compliance, and regulatory requirements are addressed in all data solutions. - Evaluate and recommend improvements to existing data architecture and processes. Technical Expertise: - Bachelors or master's degree in computer science, Information Systems, Data Science, or related field. - Extensive experience with high-level programming languages - Python, Java, or Scala. - 5+ years of experience in data architecture, data engineering, or a related field. - Proficient in data pipeline tools such as Apache Spark, Kafka, Airflow, or similar. - Experience with data governance frameworks and tools (e.g., Collibra, Alation, OpenMetadata). - Strong knowledge of cloud platforms (Azure or Google Cloud), especially with cloud-native data services. - Experience working in Agile or DevOps environments. - Experience with modern data stack tools (e.g., dbt, Snowflake, Databricks). - Experience with Hive, Oozie, Airflow, HBase, MapReduce, Spark along with working knowledge of Hadoop/Spark Toolsets. - Extensive Experience working with Git and Process Automation. - In-depth understanding of relational database management systems (RDBMS) and Data Flow Development. As a Senior Big Data Engineer at dunnhumby, you will have the opportunity to extend and enhance the Data Engineering Team, working with a market-leading business to explore new opportunities and influence global retailers. You will be responsible for designing end-to-end data solutions, architecting scalable data pipelines, orchestrating automation, ensuring data governance and quality standards, and leading technical design sessions. Your technical expertise will include a Bachelor's or Master's degree in computer science or related field, extensive experience with programming languages like Python, Java, or Scala, proficiency in data pipeline tools such as Apache Spark, Kafka, and Airflow, and knowledge of cloud platforms like Azure or Google Cloud. You will also have experience with data governance frameworks, Agile or DevOps environments, modern data stack tools, Git, and process automation, along with a strong understanding of relational database management systems and data flow development. Key Responsibilities: - Design end-to-end data solutions, including data lakes, data warehouses, ETL/ELT pipelines, APIs, and analytics platforms. - Architect scalable and low-latency data pipelines using tools like Apache Kafka, Flink, or Spark Streaming to handle high-velocity data streams. - Design /Orchestrate end-to-end automation using orchestration frameworks such as Apache Airflow to manage complex workflows and dependencies. - Design intelligent systems that can detect anomalies, trigger alerts, and automatically reroute or restart processes to maintain data integrity and availability. - Define and implement data governance, metadata management, and data quality standards. - Lead architectural reviews and technical design sessions to guide solution development. - Partner with business and IT teams to translate business needs into data architecture requirements. - Ensure