Data Engineer - Lead

Data Engineer - LeadJob Description a) Mandatory Skills Databricks Apache Spark Python and SQL 8+ years strong hands-on experience with: Experience building scalable ETL/ELT data ingestion, Transformation and curation pipelines Strong understanding of Delta Lake (ACID, schema evolution, versioning) Experience with Lakehouse architecture (ingestion + curated layers) Ability to implement source-to-target mappings and physical schemas Experience implementing data quality, validation, and monitoring frameworks Strong understanding of distributed data processing Experience with CI/CD, Git, and pipeline automation Strong banking/financial services domain experience Ability to deliver production-grade, scalable data products end-to-end Experience working in Agile, cross-functional teams Great interpersonal and communication skills Can work independently with minimum supervision Flexible and adaptable working style to work with multiple stakeholders Open to work in UK Shift (10 AM to 9 PM IST with 8 hours of productive work) b) Good to Have Skills Exposure to Snowflake (hybrid environments) Knowledge of Scala Experience with data governance and catalog tools Exposure to workflow orchestration tools (Airflow, etc.) Experience in regulated environments Familiarity with financial domains (Payments, FX, Accounts) Experience in data product-oriented delivery approaches AI-assisted data product development experience Mandatory Competencies Data AI - Data Engineering - Data Quality Validation Big Data - Big Data - Pyspark Data AI - MLOPS - Python Database - Database Programming - SQL Data AI - Data Engineering - Apache Kafka Beh - Communication and collaboration