Associate Lead - Testing (QA + MLOps)

While technology is the heart of our business, a global and diverse culture is the heart of our success. We love our people and we take pride in catering them to a culture built on transparency, diversity, integrity, learning and growth.

If working in an environment that encourages you to innovate and excel, not just in professional but personal life, interests you- you would enjoy your career with Quantiphi! Role: Lead/Associate Lead – QA + MLOps & Generative AI Experience: 10+ years Location: Mumbai/Bangalore (Hybrid)

Key Responsibilities: AI/ML & GenAI Testing Strategy (AWS Ecosystem) Define testing approaches for AI systems built on AWS services such as: Amazon SageMaker

Amazon Bedrock

AWS Lambda

Amazon API Gateway

Amazon Kinesis

AWS Glue

Amazon S3

Amazon CloudWatch

Design validation frameworks covering: Model accuracy & performance validation

Data drift & concept drift detection

Hallucination detection for LLMs

Prompt robustness testing

RAG validation (retrieval accuracy + grounding)

Bias & fairness validation

Safety & toxicity testing

MLOps Quality Engineering (AWS-Centric) Validate the end-to-end ML lifecycle including: Data ingestion & feature pipelines

Model training & hyperparameter tuning

Model versioning & registry

Deployment validation

Canary & blue/green release validation

Work with AWS-native services such as: SageMaker Pipelines

SageMaker Model Monitor

SageMaker Feature Store

Bedrock model evaluation workflows

CloudWatch-based observability

Implement CI/CD quality gates for ML pipelines integrated with AWS DevOps tools.

GenAI & Agentic AI Testing Define quality engineering approaches for: LLM-based applications using Amazon Bedrock

Prompt engineering validation

Multi-agent orchestration testing

Chatbot & Voice bot conversational testing

Intent classification validation

Conversation drift & fallback validation

API contract validation for LLM integrations

Build reusable evaluation harnesses for: BLEU / ROUGE scoring

Embedding similarity scoring

Response consistency

Safety scoring frameworks

Framework & Capability Development Design reusable AI testing accelerators

Create AWS-aligned AI test automation frameworks (Python-first)

Develop synthetic data generation strategies

Establish AI quality scorecards

Build an internal AI QA Center of Excellence

Client Engagement & Leadership Lead AI/ML quality strategy workshops

Perform AI risk & readiness assessments

Present quality architecture to CXOs

Drive QA transformation programs

Mentor QA teams on AWS-based AI testing

Own delivery for AI testing engagements end-to-end

Must have skills: Testing Expertise 8–12+ years in Quality Engineering

Strong test strategy, automation & governance experience

Experience leading QA transformation initiatives

Experience building frameworks from scratch AI/ML & GenAI Expertise

Deep understanding of ML lifecycle

Experience testing ML models (NLP preferred)

Hands-on experience validating LLM applications

Strong understanding of:

Prompt engineering

RAG architecture

Embeddings

Bias & explainability AWS AI/ML Expertise

Hands-on experience with:

Amazon SageMaker (training, deployment, monitoring)

Amazon Bedrock (LLM integration & evaluation)

S3-based data pipelines

AWS IAM (security validation)

CloudWatch monitoring

Lambda & API Gateway integrations

AWS CI/CD (CodePipeline / CodeBuild preferred)

Understanding of: Infrastructure as Code (Terraform / CloudFormation)

Observability in AI systems

Cost monitoring for ML workloads

Technical Skills Python (mandatory)

Experience with ML libraries (Scikit-learn, TensorFlow, PyTorch)

Experience with LLM frameworks (LangChain, etc.)

API & automation testing frameworks

Git-based workflows

Leadership & Communication

Strong client-facing communication

Experience leading QA teams

Ability to create strategy decks & solution proposals

Strong stakeholder management

If you like wild growth and working with happy, enthusiastic over-achievers, you'll enjoy your career with us !