AI Quality Analyst

You will play a critical role in ensuring AI systems deliver accurate, relevant, safe, and high-quality responses while helping improve overall model performance through structured feedback and evaluation. - AI Output Evaluation - Review and evaluate AI-generated responses for accuracy, relevance, completeness, and consistency. - Assess outputs generated by Large Language Models (LLMs), chatbots, virtual assistants, and AI-powered applications. - Identify factual inaccuracies, hallucinations, bias, safety concerns, and instruction-following issues. - Compare outputs across different AI models and recommend improvements. - Quality Assurance & Testing - Conduct manual and systematic testing of AI systems across various use cases. - Execute quality audits and maintain evaluation standards. - Develop and follow evaluation guidelines, scorecards, and quality frameworks. - Perform regression testing to ensure model improvements do not negatively impact existing performance. - AI Performance Analysis - Analyze trends, recurring issues, and quality gaps in AI outputs. - Generate actionable insights to improve AI response quality. - Track and report key quality metrics and performance indicators. - Support benchmarking and model comparison initiatives. - Feedback & Improvement - Provide structured feedback to AI training, product, and engineering teams. - Document quality issues and recommend corrective actions. - Assist in prompt optimization and response improvement strategies. - Contribute to AI model evaluation and continuous improvement programs. - Data Review & Validation - Review training data, evaluation datasets, and human feedback annotations. - Validate data quality and ensure compliance with established standards. - Identify inconsistencies in datasets that may impact AI performance. - Support AI quality calibration exercises and review sessions. - Documentation & Reporting - Create detailed evaluation reports and quality summaries. - Maintain documentation of testing procedures, findings, and recommendations. - Present quality insights to stakeholders and project teams. - Assist in defining quality benchmarks and acceptance criteria. Qualifications Required: - 3+ years of experience in Quality Assurance, AI Evaluation, Content Review, Data Quality, Operations Quality, or related fields. - Strong understanding of Generative AI, Large Language Models (LLMs), and AI-powered applications. - Experience evaluating content for quality, accuracy, compliance, and consistency. - Excellent analytical and critical thinking skills. - Strong written and verbal communication skills. - Ability to identify patterns, trends, and quality issues. - Experience working in remote and cross-functional environments. You will play a critical role in ensuring AI systems deliver accurate, relevant, safe, and high-quality responses while helping improve overall model performance through structured feedback and evaluation. - AI Output Evaluation - Review and evaluate AI-generated responses for accuracy, relevance, completeness, and consistency. - Assess outputs generated by Large Language Models (LLMs), chatbots, virtual assistants, and AI-powered applications. - Identify factual inaccuracies, hallucinations, bias, safety concerns, and instruction-following issues. - Compare outputs across different AI models and recommend improvements. - Quality Assurance & Testing - Conduct manual and systematic testing of AI systems across various use cases. - Execute quality audits and maintain evaluation standards. - Develop and follow evaluation guidelines, scorecards, and quality frameworks. - Perform regression testing to ensure model improvements do not negatively impact existing performance. - AI Performance Analysis - Analyze trends, recurring issues, and quality gaps in AI outputs. - Generate actionable insights to improve AI response quality. - Track and report key quality metrics and performance indicators. - Support benchmarking and model comparison initiatives. - Feedback & Improvement - Provide structured feedback to AI training, product, and engineering teams. - Document quality issues and recommend corrective actions. - Assist in prompt optimization and response improvement strategies. - Contribute to AI model evaluation and continuous improvement programs. - Data Review & Validation - Review training data, evaluation datasets, and human feedback annotations. - Validate data quality and ensure compliance with established standards. - Identify inconsistencies in datasets that may impact AI performance. - Support AI quality calibration exercises and review sessions. - Documentation & Reporting - Create detailed evaluation reports and quality summaries. - Maintain documentation of testing procedures, findings, and recommendations. -