Data Engineer Unity Catalog Migration

As a Senior Data Engineer specializing in Unity Catalog Migration, you will play a crucial role in migrating large-scale data sets using your expertise in data engineering. Your responsibilities will include: - Demonstrating 5-8+ years of experience in data engineering with a focus on large-scale migration projects, including a minimum of 500+ notebooks. - Showcasing strong proficiency in PySpark, encompassing RDD operations, DataFrame/Dataset APIs, and performance optimization. - Utilizing deep Unity Catalog migration experience, particularly in areas such as Hive metastore to UC, mount-to-volume patterns, and RBAC implementation. - Conducting end-to-end validation of Unity Catalog migrated notebooks, tables, volumes, and data pipelines to ensure functional correctness and data integrity. - Hands-on experience with Azure Databricks, UCX tool, and Azure Data Factory orchestration. - Developing code transformation tools, automation frameworks, or migration accelerators. - Possessing excellent documentation skills for technical patterns, runbooks, and knowledge transfer. - Collaborating effectively with client teams in agile/sprint environments. Additionally, the company values immediate joiners who can seamlessly integrate into the team and contribute effectively to the Unity Catalog migration project. As a Senior Data Engineer specializing in Unity Catalog Migration, you will play a crucial role in migrating large-scale data sets using your expertise in data engineering. Your responsibilities will include: - Demonstrating 5-8+ years of experience in data engineering with a focus on large-scale migration projects, including a minimum of 500+ notebooks. - Showcasing strong proficiency in PySpark, encompassing RDD operations, DataFrame/Dataset APIs, and performance optimization. - Utilizing deep Unity Catalog migration experience, particularly in areas such as Hive metastore to UC, mount-to-volume patterns, and RBAC implementation. - Conducting end-to-end validation of Unity Catalog migrated notebooks, tables, volumes, and data pipelines to ensure functional correctness and data integrity. - Hands-on experience with Azure Databricks, UCX tool, and Azure Data Factory orchestration. - Developing code transformation tools, automation frameworks, or migration accelerators. - Possessing excellent documentation skills for technical patterns, runbooks, and knowledge transfer. - Collaborating effectively with client teams in agile/sprint environments. Additionally, the company values immediate joiners who can seamlessly integrate into the team and contribute effectively to the Unity Catalog migration project.