Principal Quality Assurance Engineer

Job Title: Principal Automation Engineer (AI) Deltek is seeking a Principal Automation Engineer with deep expertise in AI-native test automation to help shape the quality engineering foundation Deltek s next-generation, AI-first ERP platform for project-based businesses. This is not a role for someone who automates feature regression. It is a role for someone who can harness AI tools to build intelligent automation frameworks that reason, adapt, and self-heal. You will be the automation architect behind Deltek s in-house AI-native test automation platform combining Playwright with LLM-powered agents (Planner, Generator, Healer). You will extend, evolve, and industrialize this framework, integrating AI tools at every layer: test generation, self-healing selectors, LLM-as-a-Judge evaluation, and CI/CD-gated quality pipelines. If you are fluent in Playwright, agentic AI workflows, and modern test engineering and want to build something genuinely new rather than maintain legacy frameworks we invite you to join our team. ERP domain knowledge is a strong plus and will accelerate your impact. Responsibilities: Architect and evolve the AI-native automation framework extending Playwright-based agents with LLM-powered planning, test generation, and self-healing capabilities. Use AI tools extensively (Claude, GitHub Copilot, LLM APIs) to design, generate, and augment automation suites reducing human authoring effort while increasing scenario coverage. Build and maintain Playwright agent pipelines for end-to-end workflow automation across Deltek s Projects, Workforce Management, and Financials modules. Integrate LLM-as-a-Judge (LLMaaJ) evaluation into the test pipeline to automatically score AI-generated outputs, detect hallucinations, and validate response quality against golden datasets. Design and implement AI safety and correctness test cases : hallucination detection, bias testing, output guardrail validation, and behavioral consistency across edge cases. Own the CI/CD automation pipeline (GitHub Actions / Azure DevOps) for AI-enabled releases including regression gates, model-response validation, and automated quality dashboards in ReportPortal and Grafana. Validate AI/ML outputs including prediction accuracy, recommendation relevance, natural-language responses, and inference API payloads. Build and maintain golden datasets for AI drift detection, regression baselines, and LLM evaluation benchmarks. Collaborate with Product Managers, AI/ML Engineers, and QE leads to define AI feature release quality gates and automation coverage targets. Mentor QE team members on AI-assisted automation patterns, agentic testing concepts, and framework best practices. Contribute to test strategy for data migration validation of schema fidelity and record correctness. "," Disclaimer : This job posting has been aggregated from external source. Role details, content, and availability are subject to change. Applicants are advised to confirm the latest information directly on the company website before applying.