Data Engineer (Databricks, Python)

Hiring for a Client - KnackForge Key Responsibilities Architect and implement enterprise-grade Lakehouse solutions using DatabricksDesign and deliver scalable batch and real-time data pipelines using Apache Spark (PySpark/SQL)Build ETL/ELT pipelines, incremental data loads, and metadata-driven ingestion frameworksImplement and optimize Databricks components: Delta Lake, Delta Live Tables, Autoloader, Structured Streaming, and WorkflowsDesign large-scale data warehousing solutions with 3NF and dimensional modelingEstablish data governance, security, and data quality frameworks, including Unity CatalogLead ML lifecycle management using MLflow and drive AI use cases (RAG, AI/BI)Manage cloud-native deployments on Microsoft Azure and integrate with enterprise systems (e.g., ServiceNow)Drive CI/CD, DevOps practices, and performance optimization of Spark workloadsProvide technical leadership, mentor teams, and ensure successful deliveryCollaborate with stakeholders to translate business requirements into scalable solutions Ideal Candidate Databricks Lead 10+ years of total experience, with 5+ years specifically in DatabricksArchitecture-level expertise must understand end-to-end design, not just engineering executionHands-on with Delta Lake, Delta Live Tables, Spark (PySpark/SQL), and Azure Data FactoryPrior experience working on Azure or Azure DevOps environmentsHas handled US/UK or international client-facing projectsComfortable managing and leading a teamBasic understanding of AI/ML tools and services is a plus Perks, Benefits & Culture The company provides free AWS and Azure certification trainingGroup medical insurance of 5 lakhs is included in the benefits package, along with meal allowances Hiring for a Client - KnackForge Key Responsibilities Architect and implement enterprise-grade Lakehouse solutions using DatabricksDesign and deliver scalable batch and real-time data pipelines using Apache Spark (PySpark/SQL)Build ETL/ELT pipelines, incremental data loads, and metadata-driven ingestion frameworksImplement and optimize Databricks components: Delta Lake, Delta Live Tables, Autoloader, Structured Streaming, and WorkflowsDesign large-scale data warehousing solutions with 3NF and dimensional modelingEstablish data governance, security, and data quality frameworks, including Unity CatalogLead ML lifecycle management using MLflow and drive AI use cases (RAG, AI/BI)Manage cloud-native deployments on Microsoft Azure and integrate with enterprise systems (e.g., ServiceNow)Drive CI/CD, DevOps practices, and performance optimization of Spark workloadsProvide technical leadership, mentor teams, and ensure successful deliveryCollaborate with stakeholders to translate business requirements into scalable solutions Ideal Candidate Databricks Lead 10+ years of total experience, with 5+ years specifically in DatabricksArchitecture-level expertise must understand end-to-end design, not just engineering executionHands-on with Delta Lake, Delta Live Tables, Spark (PySpark/SQL), and Azure Data FactoryPrior experience working on Azure or Azure DevOps environmentsHas handled US/UK or international client-facing projectsComfortable managing and leading a teamBasic understanding of AI/ML tools and services is a plus Perks, Benefits & Culture The company provides free AWS and Azure certification trainingGroup medical insurance of 5 lakhs is included in the benefits package, along with meal allowances