Databricks Technical Lead

Role Overview: As an Azure Databricks Lead/Specialist at Hoonartek, you will be responsible for designing, implementing, and optimizing data solutions using Azure Databricks. Your expertise will be crucial in building robust data pipelines, ensuring data quality, and enhancing overall performance. Collaboration with cross-functional teams to deliver high-quality solutions aligned with business requirements is a key aspect of this role. Key Responsibilities: - Design and Develop Data Pipelines: - Create scalable data processing pipelines using Azure Databricks and PySpark. - Implement ETL (Extract, Transform, Load) processes for ingesting, transforming, and loading data from various sources. - Collaborate with data engineers and architects to ensure efficient data movement and transformation. - Hands-on experience with Delta tables, Delta live tables, Auto-loader, Lakeflow, Lakehouse, and the latest features of Databricks. - Data Quality Implementation: - Establish data quality checks and validation rules within Azure Databricks. - Monitor data quality metrics and address anomalies promptly. - Work closely with data governance teams to maintain data accuracy and consistency. - Unity Catalog Integration: - Leverage Azure Databricks Unity Catalog to manage metadata, tables, and views. - Integrate Databricks assets seamlessly with other Azure services. - Ensure proper documentation and organization of data assets. - Delta Lake Expertise: - Understand and utilize Delta Lake for ACID transactions and time travel capabilities on data lakes. - Implement Delta Lake tables for reliable data storage and versioning. - Optimize performance by leveraging Delta Lake features. - Performance Tuning and Query Optimization: - Profile and analyze query performance. - Optimize SQL queries, Spark jobs, and transformations for efficiency. - Tune resource allocation to achieve optimal execution times. - Resource Optimization: - Manage compute resources effectively within Azure Databricks clusters. - Scale clusters dynamically based on workload requirements. - Monitor resource utilization and cost efficiency. - Source System Integration: - Integrate Azure Databricks with various source systems such as databases, data lakes, and APIs. - Ensure seamless data ingestion and synchronization. - Handle schema evolution and changes in source data. - Stored Procedure Conversion in Databricks: - Convert existing stored procedures (e.g., from SQL Server) into Databricks-compatible code. - Optimize and enhance stored procedures for better performance within Databricks. - SSRS conversion experience. Qualification Required: - Education: Bachelor's degree in Computer Science, Information Technology, or a related field. Additional Company Details: Hoonartek empowers enterprises globally through intelligent, creative, and insightful services for data integration, data analytics, and data visualization. They are a leader in enterprise transformation, data engineering, and an acknowledged world-class Ab Initio delivery partner. With a focus on delivery, quality, and value, Hoonartek is increasingly becoming the choice for customers seeking a trusted partner of vision, value, and integrity. Role Overview: As an Azure Databricks Lead/Specialist at Hoonartek, you will be responsible for designing, implementing, and optimizing data solutions using Azure Databricks. Your expertise will be crucial in building robust data pipelines, ensuring data quality, and enhancing overall performance. Collaboration with cross-functional teams to deliver high-quality solutions aligned with business requirements is a key aspect of this role. Key Responsibilities: - Design and Develop Data Pipelines: - Create scalable data processing pipelines using Azure Databricks and PySpark. - Implement ETL (Extract, Transform, Load) processes for ingesting, transforming, and loading data from various sources. - Collaborate with data engineers and architects to ensure efficient data movement and transformation. - Hands-on experience with Delta tables, Delta live tables, Auto-loader, Lakeflow, Lakehouse, and the latest features of Databricks. - Data Quality Implementation: - Establish data quality checks and validation rules within Azure Databricks. - Monitor data quality metrics and address anomalies promptly. - Work closely with data governance teams to maintain data accuracy and consistency. - Unity Catalog Integration: - Leverage Azure Databricks Unity Catalog to manage metadata, tables, and views. - Integrate Databricks assets seamlessly with other Azure services. - Ensure proper documentation and organization of data assets. - Delta Lake Expertise: - Understand and utilize Delta Lake for ACID transactions and time travel capabilities on data lakes. - Implement Delta Lake tables for reliable data storage and versioning. - Optimize performance by leveraging Delta Lake features. - Performance