Data Engineer

As a Senior Data Engineer/Lead Data Engineer at Xtelify Ltd, you will play a crucial role in spearheading our Data Platform team. Your primary responsibilities will involve architecting, scaling, and overseeing the deployment of mission-critical data pipelines and data-products handling a massive scale of approximately 5 billion daily events and a concurrency of 5 million active users. The platform you will be working on is built on a cloud-native modern data stack (AWS/GCP), enabling real-time reporting and deep data exploration from first principles. Key Responsibilities: - Architecture & Scaling: Architect, build, and optimize robust, fault-tolerant, and high-performance data pipelines capable of processing multi-terabyte streaming data daily across Xtelify's video streaming platforms. - SLA Ownership: Define and maintain the infrastructure required to extract, consume, and analyze live streaming data within a high-availability, 99.999% SLA Big Data environment. - Data Strategy & Optimization: Enhance data models, workflows, and processing frameworks to improve performance, scalability, and cost-efficiency across both structured and unstructured datasets. - Data Governance: Drive data governance, data security, and compliance best practices across large-scale, complex data environments. - Cross-Functional Collaboration: Partner with data scientists, product teams, and business stakeholders to translate complex requirements into scalable, data-driven solutions. - Technical Leadership & Mentorship: Serve as the technical anchor for the engineering team. Provide mentorship, promote engineering best practices in pipeline and API design, and manage the full lifecycle of data services. - Innovation: Stay ahead of emerging technologies to drive innovation in analytics pipelines, real-time data serving, and infrastructure for machine learning applications. Qualifications Required: - Experience: 4+ Years (with 3+ years dedicated to data engineering, software development, or distributed systems). - Education: B.E / B.Tech / M.E / M.Tech / M.S in Computer Science or Software Engineering from a premier institute. - Core Expertise: Deep, hands-on experience with Spark and Scala/Java with a proven track record of deploying production-grade streaming and batch applications at scale. - Streaming & Real-Time: Advanced, production-level experience with real-time and streaming data architectures such as Kafka, Flink, or Spark Streaming for low-latency data processing. - System Design & Modeling: Strong grasp of advanced data structures, algorithms, distributed system design, ETL pipeline design, and dimensional modeling. - Big Data & Cloud Ecosystem: Deep proficiency in the Hadoop ecosystem and cloud platform ecosystems (AWS/GCP) including tools like S3/GCS, EMR/Dataproc, Cloud SQL, Nifi, Trino, and query optimization. The company Xtelify Ltd is seeking a highly skilled individual who can not only write high-performance code but also define technical roadmaps, mentor engineers, and ensure that the infrastructure supports the company's aggressive growth. If you possess the required qualifications and are enthusiastic about making a significant impact in the field of data engineering, this role might be the perfect fit for you. As a Senior Data Engineer/Lead Data Engineer at Xtelify Ltd, you will play a crucial role in spearheading our Data Platform team. Your primary responsibilities will involve architecting, scaling, and overseeing the deployment of mission-critical data pipelines and data-products handling a massive scale of approximately 5 billion daily events and a concurrency of 5 million active users. The platform you will be working on is built on a cloud-native modern data stack (AWS/GCP), enabling real-time reporting and deep data exploration from first principles. Key Responsibilities: - Architecture & Scaling: Architect, build, and optimize robust, fault-tolerant, and high-performance data pipelines capable of processing multi-terabyte streaming data daily across Xtelify's video streaming platforms. - SLA Ownership: Define and maintain the infrastructure required to extract, consume, and analyze live streaming data within a high-availability, 99.999% SLA Big Data environment. - Data Strategy & Optimization: Enhance data models, workflows, and processing frameworks to improve performance, scalability, and cost-efficiency across both structured and unstructured datasets. - Data Governance: Drive data governance, data security, and compliance best practices across large-scale, complex data environments. - Cross-Functional Collaboration: Partner with data scientists, product teams, and business stakeholders to translate complex requirements into scalable, data-driven solutions. - Technical Leadership & Mentorship: Serve as the technical anchor for the engineering team. Provide mentorship, promote engineering best practices in