AI/LLM QA Engineer - Agentic & Multi-Agent System Testing

AI/LLM QA Engineer - Agentic and Multi-Agent System Testing Hyderabad, Telangana, India | Hybrid Full-time About the Role We are seeking a highly skilled AI/LLM QA Engineer with expertise in Agentic and Multi-Agent system testing to join our team. The ideal candidate will have a strong background in software QA/testing combined with handson experience in AI/ML or LLMbased applications. You will be responsible for validating the performance, accuracy, and robustness of nextgeneration AI systems, including orchestrationdriven workflows and autonomous agents. Requirements Experience: 4+ years Location: Hyderabad Work Mode: Hybrid Department: Engineering Employment Type: Fulltime Notice Period: 15 days Key responsibilities Design and execute test strategies for Agentic and MultiAgent systems, ensuring reliability and scalability. Perform LLM evaluation using metrics such as Exact match / Soft match, BLEU, ROUGE, BERTScore, and semantic similarity using embeddings. Develop and maintain prompt testing frameworks and validate output consistency across use cases. Implement and evaluate guardrails for responsible AI behavior (hallucinations, toxicity, bias). Build automated test pipelines for LLMdriven workflows and orchestration frameworks. Collaborate with AI/ML engineers and product teams to define test cases, benchmarks, and acceptance criteria. Identify defects, performance issues, and edge cases in AI systems. Integrate testing processes into CI/CD pipelines. Required skills and experience Handson experience in Agentic & MultiAgent Testing. Expertise in LLM evaluation frameworks and methodologies. Strong understanding of prompt engineering validation, guardrails and safety mechanisms, and output evaluation using embeddings and NLP metrics. 3+ years in Software QA/Testing. Minimum 2+ years of experience in AI/ML or LLMbased systems. Proficiency in Python or TypeScript/JavaScript. Good to have Experience with orchestration frameworks such as LangChain, LangGraph, LlamaIndex, DSPy, or OpenAI Assistants/Actions. Familiarity with CI/CD tools like GitLab and Jenkins. Knowledge of monitoring tools like Grafana and test management/tracking tools like Jira and XRay. Exposure to AI Quality Engineering practices. About the Company Our client is a forward-thinking organization specializing in innovative AI solutions and technologies. .