Senior Data Engineer

About YAL YAL is building AI-first products across voice, language, communication, discovery, and intelligent user experiences. Our systems combine data, AI, NLP, speech, product engineering, and real-time analytics to create scalable solutions for users and businesses. Role Overview We are looking for a hands-on, builder-mentality Data Engineer with 5–8 years of experience to lead our zero-to-one data platform initiatives. This role focuses heavily on production application data engineering—ensuring seamless, low-latency data flow between our live transactional systems, analytical warehouses, and AI models. You will work across multiple data domains, including product usage data, live user behavior, voice and text datasets, ML training data, and real-time event streams. Key Responsibilities Architect 0-to-1 Systems: Design, build, and scale our foundational data infrastructure and pipelines from scratch, making critical early-stage technology choices. Production Data Engineering: Manage high-throughput production application data streams. Implement Change Data Capture (CDC) and ensure seamless, low-latency synchronization between transactional databases (OLTP) and analytical systems (OLAP). AI/ML Data Infrastructure: Build robust data infrastructure to support AI/ML model training, feature engineering, evaluation, experimentation, and production monitoring. End-to-End Pipelines: Develop reliable ETL/ELT workflows for data ingestion, transformation, validation, storage, and consumption across structured, semi-structured, and unstructured data. Real-Time & Batch Processing: Build systems for real-time event streaming and batch data processing based on live product requirements. Integration & Collaboration: Work with backend and platform teams to integrate data pipelines with APIs, internal tools, dashboards, and live production services. Data Governance & Quality: Establish the first principles of data quality, consistency, observability, lineage, governance, and security across all pipelines. Cross-Functional Leadership: Collaborate with product and engineering teams to convert ambiguous business requirements into resilient, production-ready data solutions. Required Skills 5–8 years of experience in data engineering, backend engineering, or data platform engineering, with a proven track record of 0-to-1 setups in a fast-paced or startup environment. Strong programming fundamentals with deep hands-on experience in Python and SQL. Production Application Data: Extensive experience working with live application databases (e.g., PostgreSQL, MongoDB) and implementing event-driven architectures and CDC (Change Data Capture) pipelines. Data Modeling & Architecture: Strong understanding of data warehousing, lakehouse architecture, and distributed systems. Cloud & Tools: Experience with cloud platforms (AWS, GCP, or Azure) and hands-on expertise with processing frameworks/tools such as Airflow, Spark, Kafka, Flink, or dbt. Engineering Best Practices: Good understanding of APIs, backend systems, data quality checks, monitoring, logging, alerting, and pipeline reliability. Good to Have Experience with ML dataset formats, transcription pipelines, speech analytics, or conversational AI systems. Experience instrumenting product telemetry, tracking live user behavior events, and modeling scalable schemas to power product analytics and map user journeys. Experience with data annotation workflows, feedback loops, or human-in-the-loop systems. Hands-on exposure to feature engineering, including building centralized feature stores from scratch and optimizing real-time data pipelines for machine learning models.