Source description
About the role
As an experienced professional with over 5 years of hands-on Azure Databricks experience, you will be responsible for the following key responsibilities: - Building real-time ingestion pipelines using Azure Event Hub and Databricks Structured Streaming. - Designing scalable streaming pipelines in adherence to best practices. - Implementing structured logging and metadata augmentation for efficient data processing. - Utilizing Azure Functions for XML/XSD validation and ensuring data quality. - Optimizing Spark performance within the Azure Databricks environment. - Setting up monitoring and alerting mechanisms through Azure Monitor, Databricks Jobs, and Log Analytics. - Collaborating cross-functionally to adopt new technologies and apply data engineering principles effectively. Your qualifications for this role should include: - Proficiency in SQL and Python programming languages. - Familiarity with Unity Catalog and data governance principles. - Experience with unit testing and DevOps practices for data workloads. - Strong understanding of advanced software development and DevOps concepts. - Hands-on experience with Azure DevOps/GitHub and basic Terraform/IaC knowledge. - Databricks Certified Associate/Professional Data Engineer certification would be a strong plus. Feel free to apply if you meet the requirements and are ready to contribute to a dynamic team environment focused on leveraging cutting-edge technologies for data engineering. As an experienced professional with over 5 years of hands-on Azure Databricks experience, you will be responsible for the following key responsibilities: - Building real-time ingestion pipelines using Azure Event Hub and Databricks Structured Streaming. - Designing scalable streaming pipelines in adherence to best practices. - Implementing structured logging and metadata augmentation for efficient data processing. - Utilizing Azure Functions for XML/XSD validation and ensuring data quality. - Optimizing Spark performance within the Azure Databricks environment. - Setting up monitoring and alerting mechanisms through Azure Monitor, Databricks Jobs, and Log Analytics. - Collaborating cross-functionally to adopt new technologies and apply data engineering principles effectively. Your qualifications for this role should include: - Proficiency in SQL and Python programming languages. - Familiarity with Unity Catalog and data governance principles. - Experience with unit testing and DevOps practices for data workloads. - Strong understanding of advanced software development and DevOps concepts. - Hands-on experience with Azure DevOps/GitHub and basic Terraform/IaC knowledge. - Databricks Certified Associate/Professional Data Engineer certification would be a strong plus. Feel free to apply if you meet the requirements and are ready to contribute to a dynamic team environment focused on leveraging cutting-edge technologies for data engineering.
More at Aionos