Senior Data Specialist - R01565550

Job Description – Data Engineer (Databricks)

Company: Brillio Technologies Role: Data Engineer – Databricks Experience: 5–10 Years Location: India (Hybrid/3 days) Employment Type: Full-time

About Brillio

Brillio is a leading digital transformation partner delivering innovative solutions across Data, Cloud, AI, and Digital Engineering. We help global enterprises unlock value through cutting-edge technology and data-driven insights.

Role Overview

We are looking for a hands-on Data Engineer with strong Databricks expertise to design, build, and scale modern data platforms. The ideal candidate will have experience in PySpark, Delta Lake, and cloud-native data engineering and will work closely with analytics, product, and business teams to deliver high-performance data pipelines.

Key Responsibilities

Design, develop, and maintain scalable ETL/ELT data pipelines using Databricks and PySpark

Build and optimize data lakehouse architectures (Bronze–Silver–Gold layers)

Develop data ingestion frameworks from multiple sources (batch & real-time)

Implement Delta Lake features including schema evolution, ACID transactions, and optimization

Create and manage Databricks notebooks, workflows, and jobs for production pipelines

Ensure data quality, governance, and lineage across pipelines

Optimize Spark jobs for performance (partitioning, caching, tuning)

Collaborate with cross-functional teams (Data Scientists, Analysts, Architects)

Support production deployments, monitoring, and troubleshooting of data pipelines

Contribute to migration of legacy data platforms to modern cloud-based architecture

Technical Skills (Must-Have)

Strong experience in:

Databricks (Workflows, Notebooks, Jobs)

PySpark / Spark SQL

SQL & Data Warehousing concepts

Hands-on experience with:

Delta Lake / Lakehouse architecture

Data pipeline development & orchestration

Good understanding of:

Data modeling, ETL/ELT processes, SCD concepts

Cloud & Tools

Experience in at least one:

AWS (S3, Glue, EMR) / Azure (ADF, ADLS, Synapse)

Workflow tools:

Airflow / Databricks Workflows

Version control:

Git / CI-CD pipelines

Good to Have

Experience with:

Streaming frameworks (Kafka / Spark Streaming)

DBT / Snowflake / Redshift / Fabric

Exposure to:

Unity Catalog, Autoloader, Delta Live Tables

Experience Required

5–10 years of experience in Data Engineering

Minimum 2–4 years of hands-on Databricks experience

Proven experience building enterprise-scale data platforms and pipelines

Soft Skills

Strong problem-solving and analytical skills

Ability to work in a fast-paced, agile environment

Good communication and stakeholder management

Why Join Brillio

Work on cutting-edge data & AI programs

Opportunity to build large-scale data platforms

Collaborative and innovation-driven culture