A launch decision with no evidence behind it.
I took responsibility for quality and release assurance on the first customer-facing AI product at Reward Gateway, the employee-benefits platform Edenred acquired for £1.15bn. The application had already been built. The evaluation infrastructure had not. It was a retrieval-based assistant serving many enterprise customers from one system, where a failure of isolation is not a bug in one account but a disclosure across several. There was no defensible basis for deciding whether it was ready to launch.
Because the model’s wording varied between runs, conventional exact-output checks were insufficient. The same prompt can return three different answers and all of them be acceptable. I replaced pass/fail equality with semantic assertions, classifier-based validation and sampling across repeated runs, so the suite could judge whether a response completed the task and stayed inside the system’s rules. The adversarial half covered direct and indirect prompt injection, jailbreaks, tenant data isolation, personal data leakage, authentication flows and retrieval scope.
The results exposed gaps that mattered before launch. I also told the go/no-go panel what the suite could not do: without a labelled dataset there was no way to measure answer accuracy at scale. It was recorded as residual risk, and the panel launched knowing exactly what the evidence did not cover.
The suite became the release gate for later AI work and set the platform’s wider AI testing approach: AI-assisted test generation, and risk-based test selection driven by what a code change touched.