Lead Data Engineer

Experience: 6–8 Years Location: Hyderabad Employment Type: Contract Open Positions: 1 Budget : Open Duration: 1 year Job Summary We are seeking a highly skilled Lead Data Engineer with 6–8 years of experience in designing and building scalable, cloud-native data platforms and high-performance data pipelines. The ideal candidate will have expertise in developing batch and real-time data processing systems that support analytics, AI/ML initiatives, and enterprise data solutions across domains such as Banking, FinTech, Consulting, and SaaS. This role requires strong technical leadership, hands-on development capabilities, and the ability to collaborate with cross-functional teams to deliver reliable and scalable data solutions. Key Responsibilities Data Engineering & Platform Development  Design, develop, and maintain end-to-end data pipelines for batch and real-time processing.  Build scalable ETL/ELT frameworks using Python and SQL.  Implement and manage workflow orchestration using tools such as Apache Airflow.  Design efficient data models, schemas, and transformation layers to support analytics and downstream applications.  Develop data ingestion pipelines from various sources, including: o APIs o Relational and NoSQL Databases o Event Streams o External Systems Streaming & Performance Optimization  Design and optimize real-time data streaming solutions using platforms such as Kafka.  Enhance pipeline performance for: o High Throughput o Low Latency o Cost Efficiency o Reliability  Implement monitoring, alerting, and logging mechanisms to ensure platform stability and SLA compliance. Cloud, DevOps & Engineering Best Practices  Develop and manage data solutions primarily on Microsoft Azure.  Leverage AWS or GCP experience where applicable.  Utilize Docker and CI/CD pipelines for deployment automation and version control.  Follow software engineering best practices, including: o Code Reviews o Automated Testing o Documentation Standards o Version Control Management  Ensure data quality, governance, validation, and compliance across all data pipelines. AI & Advanced Data Engineering  Support the creation of AI/ML-ready datasets and feature engineering pipelines.  Build and maintain Feature Stores and MLOps workflows.  Develop or integrate AI-enabled solutions, including: o LLM-based Data Pipelines o Document Processing Workflows o ETL Automation o Retrieval-Augmented Generation (RAG) Architectures  Collaborate closely with Data Scientists, ML Engineers, and Analytics teams to enable AI-driven business outcomes. Leadership & Collaboration  Translate business requirements into scalable technical solutions.  Partner with Product, Analytics, Engineering, and Business stakeholders.  Mentor junior engineers and provide technical guidance.  Participate in architecture discussions and contribute to platform design decisions.  Drive best practices and continuous improvement initiatives across the data engineering team. Required Skills & Experience Must-Have Skills  5–8+ years of experience in Data Engineering.  Strong programming expertise in Python.  Advanced SQL skills and experience with complex data transformations.  Hands-on experience building ETL/ELT pipelines.  Experience with cloud platforms: o Microsoft Azure (Preferred) o AWS o GCP  Experience with real-time data streaming technologies such as Kafka.  Strong experience with Apache Airflow or similar orchestration tools.  Solid understanding of: o Data Modeling o Data Warehousing Concepts o Data Architecture Principles Preferred Skills Data Platforms & Technologies  Snowflake  BigQuery  Amazon Redshift  Delta Lake  dbt  Apache Spark / PySpark AI & Machine Learning  MLOps  Feature Stores  MLflow  Large Language Models (LLMs)  Generative AI  Retrieval-Augmented Generation (RAG) Domain Experience  Banking  FinTech  Consulting  Enterprise SaaS Leadership  Prior experience in technical leadership roles.  Client-facing stakeholder management experience.  Experience driving architecture and solution design discussions.