I was asked to sign off an AI assistant.
I joined the project after the assistant had already been built. There were no tests or test infrastructure, and the only representative environment was the cloud stack it ran on.
It was a multi-tenant retrieval assistant, with each customer’s data separated behind a shared retrieval layer. My first question was what it would return when a user asked for data they were not authorised to see. I wrote around 180 tests covering direct and indirect prompt injection, jailbreaks, tenant isolation, PII, the authentication flow and the reach of retrieval.
Designing the attacks was not the hardest part. The harder problem was evaluation: a system that answers differently each time cannot be tested by asserting exact output, and conventional automation is a poor fit. The useful checks were bounded properties rather than exact strings, semantic and classifier-based assertions, and repeated runs to understand the distribution rather than trusting a single pass.
The suite exposed gaps serious enough to challenge the release decision. Programme management needed a final report for the launch panel, which in practice meant a recommendation: yes or no.
I did not give a binary verdict. I separated what I could attest to from what remained unknown, and put each unknown in front of the panel so that accepting it was explicit. They approved the launch with the failure modes in front of them.
I also used follow-up question rate as a product metric: if someone has to ask again, the first answer did not do its job. The suite remained the gate for every AI feature when I left.
The instinct came from defence: define the boundary before asking a system to operate inside it.
The point was not certainty. It was being informed rather than surprised.