AI QA ENGINEER

Job Title: AI | QA Engineer Location: Bangalore, India (On-site) Department: AI & Automation

About the company

Opkey is the leading Cloud Application Lifecycle Management (CALM) platform for Oracle, Workday, Salesforce, Coupa, and more. It cuts the costs and risks that drag down implementations and ongoing change, helping you go live on time, get more from your cloud app investments, and reach AI readiness faster. Opkey's 20+ AI agents manage all five phases of the cloud application lifecycle—Define, Design, Configure, Test and Train.

Whether it’s a new implementation, a platform update, or business‑as‑usual change, Opkey handles it all: updates validated in hours, self‑healing tests, end‑to‑end integrations assured, configurations synced, and training updated in real time—all delivered in a single unified platform instead of a patchwork of disconnected tools.

Powered by Argus, a domain‑specific AI model trained on decades of expertise and terabytes of enterprise application data, Opkey automates configuration, testing, change impact analysis, and training across these applications—cutting manual effort by 80%, enabling 30% faster go‑lives, and slashing downtime risk by 92%

About the Role

We are looking for a skilled AI QA Engineer to join our ERP automation team at Opkey. You will work closely with AI developers to test, validate, and ensure the quality of AI-powered agents built for enterprise ERP workflows.

Key Responsibilities

Design, develop, and execute test strategies for AI agents used in ERP automation pipelines

Validate the behavior, accuracy, and reliability of AI agent outputs across various ERP scenarios

Collaborate with AI developers to understand agent architecture and identify edge cases

Build and maintain automated test frameworks tailored for LLM-based and agentic systems

Perform functional, regression, integration, and performance testing on AI-driven workflows

Monitor AI agent performance post-deployment and flag anomalies or drift in behavior

Write detailed test plans, bug reports, and QA documentation specific to AI systems

Conduct prompt testing and evaluate AI responses for correctness, consistency, and safety

Build and maintain internal QA dashboards and reporting tools for real-time test visibility

Participate in Agile/Scrum ceremonies including sprint planning, standups, retrospectives, and backlog grooming as the QA representative

Conduct thorough Root Cause Analysis (RCA) on defects, test failures, and AI agent misbehaviors and present findings to the team

Follow and contribute to SDLC and STLC processes ensuring quality gates are met at every stage of development

Required Skills & Experience

2–4 years of hands-on experience as a QA Engineer in software or automation products

Strong understanding of SDLC and STLC methodologies with experience applying them in fast-paced product environments

Hands-on experience working in Agile/Scrum teams with familiarity in tools like Jira or Azure DevOps

Proficiency in Python for writing test scripts, automation utilities, and data validators

Experience building lightweight internal tools or test dashboards using Streamlit

Solid understanding of AI agents, LLMs, RAG pipelines, and agentic frameworks (LangChain, AutoGen, CrewAI, etc.)

Experience testing API-based systems, microservices, and complex backend workflows

Proficiency with test automation tools such as Selenium, Pytest, Postman, or similar

Proven ability to perform Root Cause Analysis on complex bugs and systemic failures with clear documentation

Familiarity with ERP platforms (SAP, Oracle, Salesforce, etc.) is a strong plus

Knowledge of prompt engineering and how to test AI model behavior systematically

Experience with CI/CD pipelines and integrating automated tests into DevOps workflows

AI Certifications (Required / Preferred)

Certification in AI/ML fundamentals — Google, Microsoft (Azure AI), AWS, or IBM preferred

Certification in Generative AI or LLM-related technologies (e.g., DeepLearning.AI, Coursera, Udacity)

Any recognized certification in AI Agent development or Agentic frameworks is a strong plus