Source description
About the role
Agentic AI Application Tester
Taguig, National Capital, Philippines
Trending
Job Description
Key Accountabilities
- Definition and execution of testing and quality assurance strategies for AI‑enabled workflows
- Continuous evaluation and monitoring of system behavior in production environments
- Contribution to auditability, risk management, and continuous quality improvement
Principal Responsibilities
- Define quality criteria and testing strategies for agent workflows, covering accuracy, latency, safety, compliance, and operational risk
- Build automated evaluation harnesses to assess agent performance, including hallucination rates, tool misuse, policy violations, and task success
- Implement continuous production monitoring to detect anomalies, quality degradation, and emerging safety concerns
- Develop and maintain automated test suites using Playwright for UI testing and custom scripts for API and workflow validation
- Apply LLM evaluation frameworks to assess output quality, regression, and system drift over time
- Produce and maintain dashboards and reports that communicate quality metrics and trends to engineering and stakeholders
- Develop and maintain runbooks for common failure modes and contribute to incident response activities
- Collaborate closely with developers to improve prompts, tool definitions, and workflow designs based on test results
- Ensure testing, logging, and monitoring practices align with data privacy, audit, and regulatory requirements
Responsibilities
Qualifications
Knowledge, Skills & Experience
Essential
- Minimum 3 years’ experience in QA, test automation, or DevOps roles (or 2 years with direct experience testing AI or ML‑enabled systems)
- Strong Python skills for test automation, evaluation harnesses, and basic data analysis
- High attention to detail, with a focus on issues that materially impact reliability and user trust
- Comfort working with evolving tools, frameworks, and testing practices
- Collaborative mindset, using evidence‑based insights to influence product and engineering decisions
Technical Skills (Required)
- Programming: Python (test automation, evaluation harnesses, data analysis)
- UI Automation: Playwright (end‑to‑end workflow testing)
- AI Evaluation: Deepeval, RAGAS, Evidently.AI (LLM quality, drift, and regression analysis)
- Workflow Testing: API and agent workflow validation using custom scripts
- Monitoring: Production quality monitoring and anomaly detection
Desirable
- Pytest or equivalent testing frameworks
- SQL for querying logs, metrics, or evaluation datasets
- Prometheus, Grafana, or similar monitoring tools
- Familiarity with hallucination detection and AI safety patterns
- CI/CD pipelines and Git‑based workflows
WTW is an Equal Opportunity Employer
Apply Now
Job Info
- Job Identification202603409
- Job CategorySystems Architecture
- Posting Date05/28/2026, 11:44 PM
- Locations23/F W City Center, Taguig City, Metro Manila, 1634, PH
- Job ScheduleFull time
Similar Jobs
American English
- Deutsch
- American English
- Español
- Français
- Français canadien
- Polski
- Português do Brasil
- 简体中文
View Cookie SettingsLearn more
AcceptDecline
Are You Still With Us?
It seems you've been gone for a while. For security reasons we will end your session automatically in 03:00 unless you would like to continue working.
End SessionContinue Working
Work Summary
This summary is generated by AI Assist. Click inside the summary text box to make changes as necessary.
DiscardAdd Summary
Page Agentic AI Application Tester - WTW External Careers Site Careers loaded
More at WTW