Source description
About the role
Job Description – AWS Senior Data Engineer
Company: Brillio Technologies Role: Senior Data Engineer Experience: 5–10 Years Location: India (Hybrid/Remote) Employment Type: Full-time
Role Overview
We are looking for a Senior Data Engineer with strong Databricks and Big Data expertise to design, build, and optimize scalable data pipelines and platforms. The ideal candidate will have hands-on experience with Spark, PySpark, AWS ecosystem, and modern data architectures , along with exposure to APIs and real-time data processing.
Key Responsibilities
Design and build scalable data pipelines using Spark / PySpark on Databricks
Develop and optimize ETL/ELT workflows for large-scale data processing
Work with AWS EMR, S3, and Hadoop ecosystem for distributed data processing
Build and maintain data lake and data warehouse solutions
Develop and integrate APIs for data ingestion and consumption
Implement data processing workflows using Airflow / Autosys
Optimize data performance using partitioning, caching, and query tuning
Handle structured and semi-structured data from multiple sources
Ensure data quality, governance, and reliability of pipelines
Collaborate with cross-functional teams (Analytics, Product, Engineering)
Technical Skills (Must-Have)
Strong expertise in:
Apache Spark / PySpark
Databricks (mandatory)
SQL (PostgreSQL or similar)
Hands-on experience with:
AWS EMR, S3
Hadoop, Hive ecosystem
Programming skills:
Python (must-have)
Scala (good exposure)
Strong knowledge of:
UNIX / Shell scripting
ETL pipelines and data engineering fundamentals
Additional Skills
Experience with:
Elasticsearch (data storage & retrieval)
Workflow orchestration tools (Airflow, Autosys)
Exposure to:
API development & integration
Version control tools ( Git / SVN )
Basic knowledge of HTML (for web/data tasks)
Experience Required
Minimum 5+ years of Data Engineering experience
Strong hands-on experience in Databricks + Big Data stack
Proven experience working on large-scale distributed systems
Education
Bachelor’s or Master’s degree in Computer Science, Engineering, or related field
Why Join Brillio
Work on cloud-native, large-scale data platforms
Exposure to cutting-edge Databricks & AWS ecosystems
Collaborative and high-performance engineering culture
More at Brillio