Senior AI Quality Assurance (QA) Engineer

Job Position: Senior AI Quality Assurance (QA) Engineer Experience: 7+ Years Location: Bengaluru

Required Skills

7+ years of experience in Software Quality Assurance, Test Automation, or AI Quality Engineering. Hands-on experience evaluating LLM-powered applications, conversational AI systems, NL2SQL solutions, or similar AI workflows. Strong understanding of AI benchmarking methodologies and experience creating benchmark (golden) datasets for regression testing. Experience with AI evaluation and observability platforms such as Lang Smith, MLflow, Arize Phoenix , or equivalent. Strong understanding of AI evaluation metrics, including Precision, Recall, Precision@K, Recall@K, F1 Score, Exact Match (EM), Mean Reciprocal Rank (MRR), Execution Accuracy, latency (P50/P95/P99), token usage, and cost analysis . Strong proficiency in Python with hands-on experience building automation using Pytest , including unit testing, integration testing, API testing, exception handling, mocking, fixtures, and parameterized testing. Experience testing REST APIs and distributed backend services. Strong SQL skills with the ability to validate generated queries, execution results, schema alignment, and business logic. Strong analytical and debugging skills with the ability to perform root cause analysis across complex AI systems.