Data Engineer (3-7 Years)

Sr. Data Engineer/Lead Data Engineer About the Role: Xtelify Ltd is looking for a Data Engineer to spearhead our Data Platform team. In this role, you will architect, scale, and oversee the deployment of mission-critical data pipelines and data-products handling a massive scale of 5B daily events and a concurrency of 5M active users . Our platform is built on a cloud-native modern data stack (AWS/GCP), enabling real-time reporting and deep data exploration from first principles. As a Lead, you will not only write high-performance code but also define technical roadmaps, mentor engineers, and ensure our infrastructure supports Xtelify's aggressive growth. • Experience: 4+ Years (with 3+ years dedicated to data engineering, software development, or distributed systems) • Job Location: Gurgaon Responsibilities: • Architecture & Scaling: Architect, build, and optimize robust, fault-tolerant, and high-performance data pipelines capable of processing multi-terabyte streaming data daily across Xtelify's video streaming platforms. • SLA Ownership: Define and maintain the infrastructure required to extract, consume, and analyze live streaming data within a high-availability, 99.999% SLA Big Data environment. • Data Strategy & Optimization: Enhance data models, workflows, and processing frameworks to improve performance, scalability, and cost-efficiency across both structured and unstructured datasets. • Data Governance: Drive data governance, data security, and compliance best practices across large-scale, complex data environments. • Cross-Functional Collaboration: Partner with data scientists, product teams, and business stakeholders to translate complex requirements into scalable, data-driven solutions. • Technical Leadership & Mentorship: Serve as the technical anchor for the engineering team. Provide mentorship, promote engineering best practices in pipeline and API design, and manage the full lifecycle of data services. • Innovation: Stay ahead of emerging technologies to drive innovation in analytics pipelines, real-time data serving, and infrastructure for machine learning applications. Desired Profile: • Education: B.E / B.Tech / M.E / M.Tech / M.S in Computer Science or Software Engineering from a premier institute. • Core Expertise: Deep, hands-on experience with Spark and Scala/Java with a proven track record of deploying production-grade streaming and batch applications at scale.• Streaming & Real-Time: Advanced, production-level experience with real-time and streaming data architectures such as Kafka, Flink, or Spark Streaming for low- latency data processing. • System Design & Modeling: Strong grasp of advanced data structures, algorithms, distributed system design, ETL pipeline design, and dimensional modeling. • Big Data & Cloud Ecosystem: Deep proficiency in the Hadoop ecosystem and cloud platform ecosystems (AWS/GCP) including tools like S3/GCS, EMR/Dataproc, Cloud SQL, Nifi, Trino and query optimization. • Delivery & Execution: Proven ability to work in complex, unstructured environments and take end-to-end technical ownership of complex, enterprise-wide data modules. Good to Have: • API Development: Hands-on experience building APIs using REST, GraphQL, or gRPC to serve real-time and batch data. • OLAP Engines: Experience with real-time OLAP engines like Apache Druid, Apache Pinot, or ClickHouse. • NoSQL Solutions: Exposure to high-throughput NoSQL solutions like Cassandra, MongoDB, Redis, or Elasticsearch at scale. • Cloud Infrastructure: Familiarity with Infrastructure as Code (Terraform) and containerization (Docker, Kubernetes). • AI Agents for Data Ops: Experience building intelligent AI agents or LLM-driven tools to automate BAU data operations (e.g., automated root-cause analysis for pipeline failures, self-healing workflows, semantic parsing of logs, or intelligent optimization of query costs and resource allocation).