AI Quality Tester

Job Description AI Quality Tester (LLM & AI Applications) Job Title: AI Quality Tester Location: Mohali 83 A(Work from Office) Employment Type: Full-Time Experience: 1 to 3 Years (Freshers with exceptional AI knowledge may also apply) About the Role We are looking for an AI Quality Tester to ensure the quality, accuracy, safety, and reliability of our AI-powered products. The ideal candidate will evaluate Large Language Models (LLMs), AI agents, Retrieval-Augmented Generation (RAG) systems, and conversational AI applications by designing test scenarios, validating outputs, and identifying areas for improvement. This role is ideal for someone passionate about Generative AI, prompt engineering, software testing, and user experience. Key Responsibilities AI & LLM Testing Evaluate AI-generated responses for factual accuracy, relevance, completeness, consistency, and tone. Test conversational AI, AI chatbots, AI assistants, and AI agents across diverse use cases. Validate AI outputs against predefined business rules and quality standards. Identify hallucinations, biased responses, prompt injection vulnerabilities, and unsafe outputs. Perform adversarial testing to uncover edge cases and model limitations. Prompt & RAG Evaluation Design and execute prompt test cases. Evaluate prompt effectiveness and recommend improvements. Test Retrieval-Augmented Generation (RAG) systems for retrieval accuracy and response grounding. Verify citation quality, source attribution, and contextual relevance. Assess AI performance under different prompting techniques. Functional & Regression Testing Perform manual testing of AI-powered web and mobile applications. Execute functional, regression, usability, and exploratory testing. Verify API responses and AI workflow integrations. Validate model updates without impacting existing functionality. Defect Reporting & Collaboration Log defects with detailed reproduction steps. Work closely with AI Engineers, Data Scientists, Product Managers, and Developers. Participate in sprint planning, feature validation, and release testing. Track issues using Jira, Azure DevOps, or similar tools. Quality Metrics & Documentation Create and maintain AI testing documentation. Develop prompt libraries and reusable test scenarios. Prepare quality reports and performance summaries. Recommend improvements for model performance and user experience. Required Skills Strong analytical and critical thinking skills. Excellent written and spoken English. Understanding of Generative AI and Large Language Models (LLMs). Hands-on experience with ChatGPT, Claude, Gemini, or similar AI tools. Knowledge of prompt engineering concepts. Ability to detect hallucinations, factual errors, inconsistencies, and unsafe outputs. Strong documentation and reporting skills. Basic understanding of manual software testing methodologies. Preferred Skills Experience in AI testing, QA, or Manual Testing. Knowledge of RAG architecture and vector databases. Familiarity with AI evaluation frameworks and benchmarks. Understanding of REST APIs and API testing using Postman. Basic knowledge of Python or SQL. Experience with Git, Jira, Azure DevOps, or TestRail. Awareness of AI safety, security, and responsible AI principles. Qualification Bachelor's degree in Computer Science, Information Technology, Artificial Intelligence, Engineering, or a related field. AI certifications from Claude (Anthropic), Microsoft, Google, AWS, Coursera, or other recognized platforms are an added advantage. What We Offer Opportunity to work on cutting-edge AI SaaS products. Exposure to Large Language Models, AI Agents, RAG, and Prompt Engineering. Collaborative work environment with AI Engineers and Product Teams. Learning and certification support. Career growth in the rapidly evolving field of Artificial Intelligence. Preferred Candidate Profile Passionate about Artificial Intelligence and emerging technologies. Curious, detail-oriented, and quality-focused. Strong problem-solving and investigative mindset. Excellent communication and teamwork skills. Able to think .