Source description
About the role
Key Responsibilities Monitor AI applications for response quality, accuracy, relevance, latency, errors and availability. Track API usage, token consumption, request volumes and operational performance. Identify possible model drift, data-quality issues and changes in AI response behaviour. Evaluate AI-generated responses for accuracy, groundedness, consistency, tone and compliance. Create test cases and conduct regression testing after prompt, model or knowledge-base changes. Clean, label, organise and validate datasets used for AI evaluation and improvement. Support human-feedback and RLHF-related activities through response ranking, annotation and quality review. Maintain and version-control system prompts, prompt templates and AI instructions. Document prompt changes, test results and release history. Troubleshoot failed or slow AI requests by reviewing logs, API responses, authentication, rate limits and integrations. Reproduce issues and escalate them to AI engineers or development teams with clear diagnostic details. Maintain incident records, operating procedures and troubleshooting documentation. Follow responsible AI, information-security and data-privacy guidelines. Required Skills 2-5 years of experience in application support, production support, AI/ML operations, QA, data operations, DevOps or a related role. Basic understanding of Generative AI, Large Language Models, prompts, tokens and AI APIs. Exposure to OpenAI, Azure OpenAI, Microsoft Foundry, Gemini, AWS Bedrock or another AI platform. Ability to understand logs, error messages, API responses and monitoring dashboards. Basic knowledge of REST APIs, JSON and Postman or similar API-testing tools. Familiarity with Git or another version-control platform. Strong analytical, troubleshooting and documentation skills. Excellent attention to detail, particularly when reviewing AI-generated responses. Good written and spoken English. Comfortable working independently from home and aligning with UK working hours. Preferred Skills Basic Python, SQL or scripting knowledge. Experience with monitoring tools such as Azure Monitor, Application Insights, Datadog, Grafana, Splunk or CloudWatch. Exposure to LangSmith, MLflow, prompt flow or similar AI evaluation and observability tools. Understanding of RAG, embeddings, vector databases or enterprise knowledge bases. Experience with Jira, ServiceNow or other ticketing tools. Knowledge of responsible AI, data privacy and AI security practices. Qualification Bachelors degree in Computer Science, IT, Engineering, Data Science or a related discipline. Candidates with strong relevant practical experience will also be considered. Why Should You Apply? Permanent work-from-home role. Work directly with a UK-based client. Build hands-on experience in GenAI operations and LLMOps. Work on model monitoring, AI evaluation, prompt management and production troubleshooting. Choose between regular full-time employment and a freelance engagement. Long-term career opportunity in one of the fastest-growing areas of technology. Apply now if you want to build your career in AI Operations while working remotely on enterprise Generative AI applications for an international client.