Data Scientist - Evaluations

About the Role This is an intellectually demanding role. The Data Scientist anchors the evaluations function for this vertical designing, building, and maintaining evaluation frameworks that measure model and system quality in operational context. You are not running standard benchmarks. You are building domain-specific eval harnesses for high-stakes use cases where a wrong answer carries real consequences. You will work closely with the MLOps Engineer, the PM, and the deployment team. The evaluations you design are the mechanism by which the team determines whether what weve built is good enough to deploy and whether it stays good after deployment. What Youll Do Design and build evaluation frameworks for Sarvams AI outputs across domain-specific requirements: document comprehension, command summarisation, geospatial reasoning, enterprise workflow automation, and others as they emerge Define quality metrics in collaboration with domain experts and clients; translate operational requirements into measurable, defensible signals Run structured evaluation cycles pre- and post-deployment; build dashboards that surface model quality in production Identify failure modes, edge cases, and distribution shifts with the bias of someone looking for whats wrong, not confirming whats right Collaborate with the MLOps Engineer to operationalise eval pipelines automated, triggered by deployment events, versioned, and reproducible Build and manage domain-specific datasets for fine-tuning, evaluation, and benchmarking including human annotation workflows where needed Publish internal findings and quality reports that feed the product and engineering roadmap What Were Looking For 3 6 years in data science, ML research, or applied AI; at least 2 years working with LLMs in production contexts. Strong statistics and probability fundamentals you understand what makes an evaluation valid and what makes it misleading. Experience designing evaluation frameworks from scratch: custom metrics, inter-rater reliability, red-teaming methodologies. Python proficiency; comfort with pandas, NumPy, HuggingFace datasets, RAGAS, EleutherAI Eval Harness, LangSmith, or equivalent. Experience with prompt engineering, model fine-tuning, or RLHF in applied settings. Ability to work with unstructured domain data: PDFs, doctrine documents, transcripts, and field reports.