AI Quality Analyst

Job Title: AI Quality Analyst Location: Pune Experience: 410 Years Role Overview Ensure the quality, reliability, and robustness of AI/ML systems by designing and executing comprehensive testing strategies for non-deterministic models, including LLM-based applications. Proven experience in testing AI/ML-powered applications, with a strong understanding of how to validate non-deterministic systems. Handson experience with LLM testing tools such as RAGAs, DeepChecks, OpenAI Evals, or DeepEval is highly desirable. Expertise in creating test plans, designing test cases, and executing functional, performance, and regression tests for AI models, focusing on bias, accuracy, and robustness. Strong proficiency in Python and frameworks like PyTest for building automated test suites. Good experience testing RESTful APIs and backend services to ensure seamless data flow and integration. Familiarity with CI/CD pipelines (Jenkins, Kubernetes) and cloud platforms (GCP, AWS) is a major plus. A methodical approach to identifying, documenting, and tracking bugs in complex AI systems. Key Responsibilities Design and execute tests for AI/ML and LLM-powered applications Validate non-deterministic systems for accuracy, bias, and robustness Create and maintain test plans, test cases, and automation suites Use tools like RAGAs, DeepChecks, OpenAI Evals, DeepEval Perform functional, regression, and performance testing Test REST APIs and backend integrations Integrate testing into CI/CD pipelines (Jenkins, Kubernetes) Identify, document, and track defects in AI systems Required Qualification Proven experience in testing AI/ML applications Strong understanding of LLM testing and validation techniques Proficiency in Python and PyTest for automation Experience with API and backend testing Ability to evaluate bias, accuracy, and model performance Strong analytical and problem-solving skills Preferred Qualifications Handson experience with LLM evaluation tools (RAGAs, DeepEval, etc.) Familiarity with GCP or AWS cloud platforms Experience with CI/CD tools (Jenkins, Kubernetes) Exposure to AI quality frameworks and metrics .