Walk-in || Senior Data Engineer || F2F Drive

Role & responsibilities . Data Engineering Execution Build and maintain ingestion frameworks (ADF / Databricks / Spark) Implement Bronze -> Silver transformations aligned to architecture Architect data quality checks, schema validation, and contract rules Build and optimize robust, high-throughput ELT/ETL pipelines, enabling ingestion, transformation, and curation of structured, semi structured, and unstructured data. Integrate data from multiple on premise and cloud based systems, APIs, and third-party sources. Implement complex transformations using PySpark, ensuring performance efficiency and code modularity. Build orchestration workflows in ADF, including pipelines, triggers, linked services, integration runtimes, and parameterized datasets. Familiar with using Databricks Genie. Build: Dimensional models (star/snowflake schemas), Fact tables, dimensions, surrogate keys, SCD handling Translate Silver datasets into: Analytics-ready models, Consistent KPI definitions and business logic Ensure: Consistency across domains (common dimensions, conformed models), Reusability and scalability of models Databricks & PySpark Engineering Develop scalable transformation scripts using PySpark on Databricks, applying advanced optimizations like caching, partitioning, and Delta Lake capabilities. Implement Delta Lake featuresACID transactions, schema enforcement, schema evolution, and time travelacross the data lifecycle. Perform performance tuning, handling bottlenecks related to cluster configuration, shuffle operations, joins, and parallelization. Collaborate with platform teams to manage Databricks clusters, jobs, notebooks, and CI/CD integrations. Data Governance & Quality Implement data quality checks, audit mechanisms, and validation frameworks to ensure data accuracy and consistency in Unity Catalog. Enforce: Unity Catalog standards; naming conventions, metadata policies; access controls (RBAC/ABAC). Handle changes such as Column derivation logic changes (not just schema changes), Backward compatibility and impact analysis. Partner with business / data stewards to define business rules and validate metrics and edge cases. Collaboration & Stakeholder Management Collaborate closely with customer IT and business teams to understand data requirements and deliver reliable, production-ready solutions. Work with architects, product owners, and cross-functional engineering teams to align technical delivery with business objectives. Provide guidance and mentoring to junior engineers when required. Preferred candidate profile Minimum 712 years of experience in data engineering, with strong hands on exposure to Azure data ecosystem. At least 3 years of real project experience in Databricks (Azure Data Platform (ADF, ADLS, Azure SQL) & Databricks (DLT, Delta Lake, Spark, Workflows, Unity Catalog). At least 2 years of hands-on experience building data pipelines using Azure Data Factory (ADF). At least 2 years of experience developing PySpark-based transformations in Databricks. Strong SQL programming experience, including writing complex queries, performance tuning, and handling large datasets. Knowledge of CI/CD pipelines (Azure DevOps preferred) for automated deployment of ADF/Databricks artifacts. Role & responsibilities . Data Engineering Execution Build and maintain ingestion frameworks (ADF / Databricks / Spark) Implement Bronze -> Silver transformations aligned to architecture Architect data quality checks, schema validation, and contract rules Build and optimize robust, high-throughput ELT/ETL pipelines, enabling ingestion, transformation, and curation of structured, semi structured, and unstructured data. Integrate data from multiple on premise and cloud based systems, APIs, and third-party sources. Implement complex transformations using PySpark, ensuring performance efficiency and code modularity. Build orchestration workflows in ADF, including pipelines, triggers, linked services, integration runtimes, and parameterized datasets. Familiar with using Databricks Genie. Build: Dimensional models (star/snowflake schemas), Fact tables, dimensions, surrogate keys, SCD handling Translate Silver datasets into: Analytics-ready models, Consistent KPI definitions and business logic Ensure: Consistency across domains (common dimensions, conformed models), Reusability and scalability of models Databricks & PySpark Engineering Develop scalable transformation scripts using PySpark on Databricks, applying advanced optimizations like caching, partitioning, and Delta Lake capabilities. Implement Delta Lake featuresACID transactions, schema enforcement, schema evolution, and time travelacross the data lifecycle. Perform performance tuning, handling bottlenecks related to cluster configuration, shuffle operations, joins, and parallelization. Collaborate with platform teams to manage Databricks clusters, jobs, notebooks, and CI/CD integrations. Data Governance & Quality Implement data quality checks,