Principal Data Engineer

Job Requirements

• Architect Data based solutions (foundational and analytical data product) based on business need and in line with the Mastercard technology portfolio, patterns and standards.

• Collaborate closely with finance, business and technical stakeholders to translate requirements into technical data architecture and solution.

• Handson, with the ability to write / develop complex code and also help the engineering teams.

• Exposure to building Data contracts and Data products.

• Partner with data and engineering teams to build robust data pipelines, data modeling, and ensure seamless integration across diverse systems and domains.

• Facilitate trade-off discussions with both business and technical stakeholders to balance priorities and make informed decisions.

• Ability to choose between different technical options considering all aspects of engineering principles.

• Ensure alignment between business goals and technical execution, making sure features and solutions meet business requirements and customer needs.

• Participate in sprint planning, retrospectives, and other agile ceremonies to ensure the team is aligned and delivering efficiently.

• Drive the adoption of best practices in software engineering, including code reviews, testing, and continuous integration/continuous delivery (CI/CD).

• Optimize the cost/benefit of software features and architecture, ensuring scalability, performance, and operational efficiency.

• Act as leader of Data platform / technology, providing guidance on complex technical challenges and driving resolution, ensuring solutions align with Mastercard’s engineering and data principles, and technical policies.

• Identify opportunities for process improvements, helping to streamline workflows and enhance team productivity.

• Mentor and guide engineers across various experience levels, helping them grow technically and improve their software and data engineering skills.

Work Experience

• Proven experience as data architect and engineer, both on on-prem and cloud-based data platforms (traditional to modern platforms) with a strong focus on delivering large-scale projects in an agile environment.

• Work with the engineering team to help design, develop and implement large scale, high-volume, high-performance, highly available, scalable data platform and pipelines for the Lakehouse being built on on-prem Cloudera Data Platform - CDP. Ability to handle large-scale data processing and distributed computing across massive datasets.

• Must - Hands-on experience with deep understanding and experience with modern Data Platforms – Cloudera Data Platform (CDP).

• Strong hands-on experience with PySpark

• Experience with Cloudera Data Platform (CDE, CDW, Ozone, Airflow, SDX), Apache Ranger. Deep understanding of distributed data systems and Hive Metastore

• Experience and understanding of cataloging, lineage, and governance

• Experience / understanding Open Data Contract Standard (ODCS) and its implementation

• Experience working with SQL, Iceberg open table and Parquet file format, and partitioning/bucketing strategies.

• Experience with data lifecycle management, including ingestion, ETL, pruning, modelling, and governance, within highly regulated environments.

• Experience with performance engineering, ensuring systems are built to scale and meet varying demands.

• Knowledge of security best practices and experience in ensuring the secure development of applications.

• Nice to have:

• Experience modernising enterprise finance systems or regulated environments • Knowledge of CI/CD, data engineering best practices • Understanding of financial data structures, accounting processes, or reconciliation workflows • Prior experience with financial systems, such as Oracle Financials, Oracle Fusion Cloud, and Hyperion, with experience optimising their integration into broader data ecosystems will be a plus.