LEAD DATA ENGINEER - Databricks

Senior Data Engineer (Databricks + CosmosDB) Location: Bangalore (Hybrid - 3 days from client office) Experience: 8-10 Years Job Summary: We are seeking a highly skilled Senior Data Engineer with deep expertise in Azure Databricks and Cosmos DB. The ideal candidate will be responsible for designing, building, and optimizing scalable, cloud-native data platforms and data pipelines. This role requires a hands-on approach to developing high-performance data solutions that support analytics, AI/ML initiatives, real-time processing, and enterprise reporting requirements while ensuring data quality, security, and operational excellence. Responsibilities:1. Data Engineering & Pipeline Development Design, develop, and maintain scalable batch and real-time data pipelines using Databricks and Apache Spark. Build robust ETL/ELT processes to ingest, transform, and curate large volumes of structured and unstructured data. Develop reusable data engineering frameworks and accelerators. Optimize data processing jobs for performance, scalability, and cost efficiency. 2. Databricks Platform Development Build and maintain data solutions using Azure Databricks. Develop notebooks and workflows using Apache Spark RDD, Apache Spark SQL, and Python. Implement Delta Lake architectures for reliable and efficient data management. Configure and optimize Databricks clusters and workloads. Support CI/CD implementation for Databricks deployments. 3. Cosmos DB Development & Optimization Design and implement scalable solutions using Azure Cosmos DB (NoSQL). Model data effectively for NoSQL workloads. Optimize partition strategies, indexing policies, and query performance. Manage data replication, consistency models, and throughput provisioning. Integrate Cosmos DB with analytics and downstream systems. 4. Azure Data Platform Integration Develop and integrate end-to-end solutions using Azure services such as Azure Data Factory, Azure Functions, and Azure Logic Apps. 5. Real-Time & Streaming Data Processing Build streaming data pipelines using Spark Structured Streaming. Process event-driven workloads from Azure Event Hubs, Kafka Stream, or IoT sources. Implement near real-time analytics solutions integrated with Cosmos DB and Databricks. 6. DevOps & CI/CD Implement Devsecops practices for Data Engineering solutions. Automate deployment pipelines using Azure DevOps or GitHub Actions. Manage source control and release processes. Mandatory Skills: Azure Databricks Apache Spark RDD Apache Spark SQL Delta Lake Azure Cosmos DB (NoSQL) Preferred Skills: Delta Live Tables (DLT) Azure Databricks Azure Databricks Qualifications: Bachelor's degree in Computer Science, Engineering, or a related field. 8-10 years of experience in data engineering or related roles. Strong understanding of cloud engineering principles and modern data architectures. Excellent problem-solving skills and ability to work in a fast-paced environment.