Senior Data Engineer - Azure Databricks

Role Overview: You are a Senior Data Engineer / Senior Developer with strong hands-on expertise in Azure Databricks, Azure SQL, and Azure Data Integration services. Your role requires deep technical ownership, ability to design and build scalable data pipelines, and support modern data architectures while working closely with architects, product owners, and downstream consumers. Key Responsibilities: - Design, develop, and maintain end-to-end batch and near-real-time data pipelines using Azure Databricks (PySpark/SQL). - Implement scalable data processing using Medallion Architecture (Bronze / Silver / Gold layers). - Develop and optimize complex T-SQL queries, stored procedures, and schemas on Azure SQL Database. - Perform performance tuning, query optimization, and data model refinement across Azure SQL and Databricks. - Design and monitor complex ETL / ELT pipelines using Azure Data Factory (ADF). - Build reusable PySpark frameworks for ingestion, transformation, and validation. - Implement operational logging, monitoring, and data quality checks. - Analyze existing legacy data architectures and design and implement mirrored or modernized equivalents on Azure Databricks. Security, Governance & Compliance: You have experience on secure data access using Azure Key Vault, Managed Identities, Service Principals, and VNET-integrated services. You follow enterprise data security, RBAC, and governance standards to ensure compliance with data privacy and audit requirements. DevOps & Engineering Discipline: You use Git / Azure DevOps for version control, pull requests & code reviews, and CI/CD pipelines for data workloads. Qualifications Required: - Hands-on experience on Azure SQL Database, Advanced T-SQL, Schema design, Query & performance tuning, Azure Data Factory (ADF), Complex ETL / ELT pipeline design, Monitoring and troubleshooting. - Experience with Azure Databricks, PySpark & SQL, Data Modeling, Medallion Architecture (Bronze / Silver / Gold), Dimensional and analytical models, and Data Integration. Additional Details: Nice to Have: - Experience analyzing or migrating legacy data platforms (e.g., Rosetta-style architectures). - Exposure to Insurance domain data (Claims, Policy, Underwriting, Reinsurance). - Experience with Delta Lake optimizations and cost management. - Azure / Databricks certifications. Role Overview: You are a Senior Data Engineer / Senior Developer with strong hands-on expertise in Azure Databricks, Azure SQL, and Azure Data Integration services. Your role requires deep technical ownership, ability to design and build scalable data pipelines, and support modern data architectures while working closely with architects, product owners, and downstream consumers. Key Responsibilities: - Design, develop, and maintain end-to-end batch and near-real-time data pipelines using Azure Databricks (PySpark/SQL). - Implement scalable data processing using Medallion Architecture (Bronze / Silver / Gold layers). - Develop and optimize complex T-SQL queries, stored procedures, and schemas on Azure SQL Database. - Perform performance tuning, query optimization, and data model refinement across Azure SQL and Databricks. - Design and monitor complex ETL / ELT pipelines using Azure Data Factory (ADF). - Build reusable PySpark frameworks for ingestion, transformation, and validation. - Implement operational logging, monitoring, and data quality checks. - Analyze existing legacy data architectures and design and implement mirrored or modernized equivalents on Azure Databricks. Security, Governance & Compliance: You have experience on secure data access using Azure Key Vault, Managed Identities, Service Principals, and VNET-integrated services. You follow enterprise data security, RBAC, and governance standards to ensure compliance with data privacy and audit requirements. DevOps & Engineering Discipline: You use Git / Azure DevOps for version control, pull requests & code reviews, and CI/CD pipelines for data workloads. Qualifications Required: - Hands-on experience on Azure SQL Database, Advanced T-SQL, Schema design, Query & performance tuning, Azure Data Factory (ADF), Complex ETL / ELT pipeline design, Monitoring and troubleshooting. - Experience with Azure Databricks, PySpark & SQL, Data Modeling, Medallion Architecture (Bronze / Silver / Gold), Dimensional and analytical models, and Data Integration. Additional Details: Nice to Have: - Experience analyzing or migrating legacy data platforms (e.g., Rosetta-style architectures). - Exposure to Insurance domain data (Claims, Policy, Underwriting, Reinsurance). - Experience with Delta Lake optimizations and cost management. - Azure / Databricks certifications.