Data Engineer Databricks / Python / Database

Overview: We are seeking an experienced Data Engineer with strong expertise in Databricks, Python, and Database engineering. The ideal candidate will design, build, and optimize scalable data pipelines, ensure data quality, and support advanced analytics and business intelligence initiatives across the organization. Key Responsibilities: Design, build, and maintain ETL/ELT workflows on Databricks. Develop scalable data pipelines using Python and Spark. Optimize data transformation logic for performance and cost efficiency. Work with structured and unstructured data from various sources. Develop and manage data models, schemas, and tables across databases (SQL/NoSQL). Implement data quality, validation, and monitoring frameworks. Collaborate with Data Scientists, Analysts, and Cloud Engineers to support analytics and ML workloads. Ensure data security, governance, and compliance in all solutions. Troubleshoot pipeline failures and continuously improve pipeline reliability. Integrate data from APIs, streaming sources, and cloud storage. Required Skills: Core Technical Skills:- Databricks (mandatory) – Delta Lake, Notebooks, Spark SQL, Jobs, Workflows Python – Data processing, Spark APIs, automation, optimization Database Engineering – Strong SQL, query tuning, schema design, stored procedures Apache Spark – RDDs/DataFrames, optimisation, partitioning Cloud Platforms (one or more): AWS / Azure / GCP ETL/ELT tools: ADF, Glue, Dataflow, or equivalent Experience working with large-scale datasets CI/CD for data pipelines (Git, DevOps, Jenkins, etc.)