Source description
About the role
Role Purpose The purpose of this role is to support the development and maintenance of enterprise data solutions, including the Data Lakehouse, Data Warehouse, and Medallion Architecture. This role will work alongside senior engineers and architects to build, test, and deploy scalable data pipelines, with a focus on tasks such as ingestion, data validation, and structured transformations. The role is a hands-on opportunity for someone early in their data engineering career to gain deep experience with Microsofts Azure Data Platform and Microsoft Fabric, working on real-world solutions under guidance and mentorship. Key Responsibilities Data Integration Support - Assist with building and maintaining data ingestion pipelines from various sources using tools like Microsoft Fabric, Azure Data Factory, and Synapse pipelines. ETL Development - Develop and maintain reusable ETL processes, supporting the Bronze, Silver, and Gold layers of the Medallion Architecture, with a focus on reliability, reusability, and simplicity. Notebook Automation Support - Help implement and schedule metadata-driven notebooks in Fabric, working with Spark under the supervision of senior engineers. Data Quality and Testing - Assist in implementing and monitoring data quality checks, validation rules, and basic lineage tracking. Documentation and Data Catalogue Support - Document data flows, transformations, and support updates to data cataloguing tools or metadata repositories. BI & Reporting Integration - Assist with connecting data models to Power BI, and help develop certified datasets and semantic layers for self-service users. Environment & DevOps Support - Learn and contribute to CI/CD processes using Azure DevOps and Git. Support code reviews and testing cycles. Learning and Development - Be an active participant in code reviews, design sessions, and platform learning. Take ownership of tasks and work through feedback constructively. Testing and Data Quality Assurance - Develop and maintain unit tests to validate pipeline logic and data transformations. Collaborate with engineering peers to continuously refine test coverage and uphold data integrity standards. Performance Indicators Success is measured by: - Successful delivery of sprints to the satisfaction of stakeholders. - Data pipeline reliability: pipelines run with minimal failures and auto-recovery. - Data latency: freshness meets defined SLAs for Bronze, Silver, and Gold layers. - Valuable insight into the business is delivered through the implementation of recent solutions. - Innovation: regular adoption of new Fabric, AI, and automation features. - Implementation of robust solutions with minimal time spent fixing bugs. Experience - 03 years experience in a data engineering, data analyst, or junior developer role. - Exposure to Azure Data tools, including Data Factory, Synapse, and Data Lake. - Basic knowledge of SQL, preferably T-SQL, and PySpark. - Understanding of what ETL/ELT means in practice. - Familiarity with version control tools, such as Git. - Interest in learning Microsoft Fabric and Medallion Architecture. - Experience working in agile teams or projects, such as group projects or internships. Knowledge Principles and Concepts The person should have knowledge of: - Implementing clear, reusable Bronze, Silver, and Gold layers in the Medallion Architecture. - Semantic models in Power BI and Fabrics integration for analytics. - Fabric Lakehouse, Data Pipelines, Dataflows Gen2, Notebooks, and Power BI integration. - Developing and managing data in Lakehouse architectures in Fabric. - Building scalable ETL/ELT pipelines with Fabrics pipeline orchestration. - Real-Time Intelligence in Fabric. - Setting up DevOps pipelines in Fabric. - SQL, including T-SQL and Spark SQL, for querying data across Fabric Lakehouse. - Python and Spark for notebook-based data transformations. - Agile and Scrum. - Clean code approach to development. - Test Driven Development. - CI/CD. - Security models within Azure. Technologies - Microsoft Fabric - Azure Data Factory - Azure Synapse - T-SQL, DDL, and DML - Azure Data Lake - Spark Pool - Python - Azure DevOps - Git - Power BI Role Purpose The purpose of this role is to support the development and maintenance of enterprise data solutions, including the Data Lakehouse, Data Warehouse, and Medallion Architecture. This role will work alongside senior engineers and architects to build, test, and deploy scalable data pipelines, with a focus on tasks such as ingestion, data validation, and structured transformations. The role is a hands-on opportunity for someone early in their data engineering career to gain deep experience with Microsofts Azure Data Platform and Microsoft Fabric, working on real-world solutions under guidance and mentorship. Key Responsibilities Data Integration Support - Assist with building and maintaining data ingestion pipelines from various sources using tools like Microsoft
More at Black Piano