Your mission
A senior, hands-on engineering role responsible for creating measurable confidence in both the conventional software platform and the agentic AI workflows behind it. You'll test the complete journey from incoming emails and files through extraction, material matching, RFQ creation and quotation - verifying not only that the software executes correctly, but that the resulting business data and AI decisions are accurate, safe and commercially sensible. The objective is to turn quality from manual judgment into an automated, evidence-based release process. Your gates should answer: does the application work, are the integrations behaving, did the AI extract the right information, did the agent choose the right tool with valid arguments, is the business outcome correct, has this change regressed anything, and is the release safe to deploy.
WHAT YOU'LL OWN Test strategy. Define and own the testing strategy across the platform - what gets tested at which layer (unit, component, contract, integration, end-to-end, production verification), where deterministic assertions end and statistical evaluation begins, and how quality gates map to release risk. Keep it a living document engineers actually use, not a compliance artifact.
Software test automation. Automated coverage across the React web app, backend APIs and services, auth and authorization, database operations, third-party integrations, email and document ingestion, background jobs and async workflows, multi-tenant configuration and isolation, and end-to-end RFQ and quotation journeys. Playwright, pytest, API clients, SQL, CI/CD. Focus on critical business journeys and high-risk failure modes, not test-case count.
API and integration testing. Beyond happy-path responses: request and response schemas, permissions, invalid and incomplete inputs, error handling, idempotency and duplicate requests, timeouts and retries, async state transitions, webhooks and event processing, data consistency across services, tenant isolation, and integration failure and recovery - running in CI with enough diagnostic output to explain a failure.
AI and LLM evaluations. Versioned evaluation datasets built from real business scenarios, edge cases and observed failures. Targets include extraction accuracy, material and product matching, classification and routing, agent decision-making, generated RFQ and quotation content, grounding, hallucination frequency, instruction adherence and final business correctness. You define the scoring methods - exact assertions, semantic comparison, rule-based and model-based graders, human review where needed. Because behaviour is nondeterministic, expect quality thresholds and statistical comparison rather than identical output every run.
Agent workflow testing. Inspect the whole execution path, not the final response: was the correct tool selected, were its arguments valid, were calls made in the right order, was the right source data used, were permissions and tenant boundaries respected, did retries and recovery work, did the agent stop or escalate when uncertain, did it avoid unnecessary actions, did the outcome match the business objective, and were latency and cost acceptable.
Controlled use of AI testing agents. Use bounded agents to analyse requirements, suggest missing scenarios, generate candidate tests, explore unusual journeys, create adversarial inputs, reproduce defects and analyse traces. You stay responsible for validating what they generate and for defining authoritative pass/fail criteria - AI output never becomes an uncontrolled release decision.
Release readiness. Define and maintain the criteria: critical journeys pass, API contracts stable, no cross-tenant access violations, extraction accuracy above the agreed threshold, no material regression in agent task completion, known critical scenarios still succeeding, latency and cost within limits. You communicate what passed, what failed, what's uncertain and the risk of proceeding, and…