I ensure AI products are ready to release.

I took responsibility for quality and release assurance on the first customer-facing AI product at Reward Gateway, the employee-benefits platform Edenred acquired for £1.15bn. The application had already been built. The evaluation infrastructure had not. It was a retrieval-based assistant built on AWS Bedrock with OpenSearch, serving many enterprise customers from one system. It shipped natively on iOS and Android as well as the web, and I owned release assurance across all three surfaces. A failure of isolation was not a bug in one account but a disclosure across several. There was no defensible basis for deciding whether it was ready to launch.

Because the model’s wording varied between runs, conventional exact-output checks were insufficient. The same prompt can return three different answers and all of them be acceptable. I replaced pass/fail equality with semantic assertions, classifier-based validation and sampling across repeated runs, so the suite could judge whether a response completed the task and stayed inside the system’s rules. The adversarial half covered direct and indirect prompt injection, jailbreaks, tenant data isolation, personal data leakage, authentication flows and retrieval scope.

The suite covered unit, integration, API and functional layers, with dedicated adversarial security and tenant-isolation coverage, and a set of benchmark answers reviewed by subject-matter experts to judge whether the assistant was actually right.

The suite made the limits of the launch evidence visible. Without a labelled dataset, answer accuracy could not be measured at scale. Response times had no agreed service level, and the pipeline handling updated and deleted content remained untested. I turned each gap into an explicit residual risk for the go/no-go panel to assess. The product launched with a clear account of what was known, what remained uncertain and where further investment would produce better evidence.

The same approach applied to continuous evaluation. Every run queried the model and carried a usage cost, so test frequency was a commercial decision as well as an engineering one. I quantified the cost and made the trade-off visible to the people responsible for it. That is part of release assurance too: not pretending a control exists when the organisation has not yet chosen to fund it.

The suite became the release gate for later AI work and set the platform’s wider AI testing approach: AI-assisted test generation, and risk-based test selection driven by what a code change touched.