Photo credit: Quentin Rey via Unsplash
There’s a famous psychological experiment, invented by Peter Wason in 1966, which demonstrates that humans are subject to confirmation bias.
Imagine that you are shown a set of cards with numbers on one side and letters on the other. You are asked to check the validity of this hypothesis: if a card has a vowel on one side, it must have an even number on the other. Four cards are dealt out, showing A, B, 2 and 5, and you are told that you may turn over two cards to check the hypothesis. Which cards do you turn over?
In tests, most people will turn over the A card and the 2 card (which did you pick?). Unfortunately, this wastes one of your two precious tests. Turning over the A card is valid: if you find an odd number on the other side, it disproves the hypothesis. But turning over the 2 card is useless. If you find a vowel it gives you no new information, and if you find a consonant it neither confirms nor disproves the rule: the hypothesis says nothing about whether a card with a consonant has an even number or odd number on the other side.
Nevertheless, turning over the 2 feels like the right thing to do: the desire to see the vowel that we are expecting is almost visceral.
This might seem like an entirely abstract experiment based on a situation that we would never encounter in real life. That’s generally untrue: we are creating, testing, proving and disproving theories about the world all of the time. But it’s especially untrue if you are a software tester: if you do such a job (a job which the rest of us should salute as an essential service) then you spend your time formulating and applying tests (or using tools which build tests for you).
And you learn early in your testing career that the happy paths, the paths that confirm that everything is as expected, the vowel on the other side of the 2, are only a tiny, and relatively uninteresting subset of the cases you must create. The much more interesting tests are those which push the boundaries of behaviour: the text in the number field, the date of birth in the future, the negative payment value, the SQL injection, the prompt injection and all of the weird and unexpected combinations of values. The tester must have an appetite for destruction: to find the weak points in the software edifice and bring the whole structure crashing down. And then do it again when it’s supposed to have been fixed.
The often unglamorous, unrecognised business of software testing has become front page news recently, even if it has not been reported in those terms. The ‘rogue AI agents’ that ‘broke containment’ were test cases with unintended consequences. Indeed, they tested things which the testers did not intend to test: as well as determining the performance of agents in cyber exercises, they also discovered flaws in their guardrails and testing environments. In the process, they reminded us of a few things.
First, confirmation bias applies to testing rigs as well as to software products. It turns out that it’s not enough to tell an agent that it does not have access to the Internet when, in fact, it has access to the Internet. Agents are good at finding loopholes, just as testers are (perhaps because they were trained on transcripts of exercises in which people attempt to find loopholes).
Second, as well as having an appetite for destruction, testers must have a horror of harm. The point of bringing the whole system crashing down in a test environment is not to see the crash (even though it might be awesome to see) but to prevent it from happening in a production environment. Testers provoke and simulate mayhem and disaster so that end users don’t have to experience them.
And, third, just as developers cannot blame the code, the environment or the data for failures in testing (no matter how hard they try), the people who build AI agents and the people who build the environments that test AI agents can’t blame the agents when they behave unexpectedly and are allowed to do so in a way that causes wider harm.
Good testing requires imagination, ingenuity and cunning: it demands that we overcome confirmation bias and seek the unhappiest of unhappy paths through our code. It also requires responsibility, caution and a sense of duty: a belief that testing is an essential protection for the end user.
Recent incidents have been caused by testing that failed on both those counts: belief that a software product and its environment would behave as expected, and harm when that belief was disproven. The more power we give AI, the stronger and more sceptical our testing must be: the more damage we must be prepared to cause in a safe place, and the harder we must work to prevent that damage leaking into the world.
