Senior Analyst - AI QA (R-18975)

5 - 8 years of overall experience in software engineering, data science, or AI/ML development, with at least 3+ years focused on AI/LLM/GenAI testing or agent-based systems.

Python expertise in scripting, automation, and debugging.

Strong PySpark experience in distributed testing, data validation, and pipeline testing.

Hands-on knowledge of GenAI concepts, including LLMs, prompting, context management, RAG pipelines, agent tool-calling, and multi-agent orchestration.

Experience with agent development and deployment frameworks such as LangGraph, AutoGen, CrewAI, Copilot Studio Agent SDK, and Vertex AI/OpenAI agent frameworks.

Solid understanding of agent architecture covering skills, tools, connectors, memory, guardrails, and observability.

Familiarity with modern GenAI/agent evaluation frameworks such as Langsmith evaluation, AutoGen agent-behavior assessment utilities etc. for benchmarking reliability, grounding, tool-use correctness, and multi-agent performance.

Strong foundation in functional and regression testing, scenario and edge-case testing, LLM safety and hallucination testing, and workflow validation.

Experience creating evaluation datasets and defining success criteria for AI behavior.

Good understanding of API testing frameworks, data engineering concepts, cloud workflow execution (Azure/GCP/AWS), and CI/CD pipelines for test automation.