Good news! It’s all their fault. Bad news! It’s your fault too.
Photo credit: Gruescu Ovidiu via Unsplash
Several organisations have recently made statements about cyber attacks which they committed using AI systems which they failed to control. The language of these statements is interesting, not just because of its technical detail (or lack of it), but in how different parts are framed.
The parts which describe the causes and impacts of the incidents talk in the passive voice, and ascribe actions to non-human actors and external parties: ‘The incident occurred . . .’, ‘The models identified and chained vulnerabilities . . .’, ‘a model accessed the Internet . . .’, ‘our third party evaluation partner . . .’, ‘a misconfiguration by an independent company inadvertently allowed . . .’, ‘some of the agents being tested had engaged in sustained, potentially harmful activity . . .’, ‘an AI agent took autonomous, unsanctioned action . . .’.
When we read these phrases, we might wonder whether any human beings work at these organisations at all, and what they were doing when the incidents occurred.
By contrast, humans appear prominently in those parts of the statements which deal with responses to the incident. These parts are expressed in the first person plural, and full of action and intention: ‘we are implementing strict controls . . .’, ‘We’re improving and adding stronger protections . . .’, ‘We are building fine-grained network controls . . .’, ‘We are introducing monitoring . . .’, ‘we need to change how we model such threats . . .’, ‘We need to better co-design evaluation environments . . .’.
When we read these phrases, we might wonder why such clear and bold action was not taken earlier.
The varying tone of the different parts of these statements, from being a passive victim of misfortune to being an active righter of wrongs, is no doubt influenced by considerations of liability (we may wonder whether the legal expertise applied to the language came from human lawyers or from the organisations’ own AI models). However, I also think that they are examples of a cognitive bias which we find frequently in enterprise technology: the fundamental attribution error.
This term, coined by the psychologist Lee Ross in 1977, refers to the idea that when things go badly for us, we blame external factors, but when things go well for us, we take credit for our personal achievements. And we reverse this bias when judging other people: when things go badly for them, we assume it was their fault, and, when things go well for them, we assume it was due to external factors.
We can easily find examples from everyday life. My bonus was underwhelming because the company hasn’t been doing well this year. My promotion was well deserved and long overdue. Their bonus was cut because they missed all their targets. And their promotion came from being in the right place at the right time.
Fortunately, cognitive biases are simply biases and need not dominate our thought. We can avoid the self-centred and cynical consequences of the fundamental attribution error by being humble, generous and thoughtful. Unfortunately, humility, generosity and thought often vanish when we are under pressure.
Evidently, the organisations which have recently committed cyber attacks felt this pressure. We can point at them and ascribe their responses to the fundamental attribution error, but (if we are trying to be humble, generous and thoughtful) those of us who work in technology should recognise that we also make this error all the time.
The language we use to describe failure follows the pattern of the fundamental attribution error, blaming external factors, whether we are talking about an incident or a project that has gone off the rails. ‘The server crashed . . .’, ‘The patching routine stopped running . . .’, ‘The alert didn’t go off . . .’, ‘The supplier delivered late . . .’, ‘Our partner has been underperforming . . .’.
And the language we use to describe our responses also follows the pattern, ascribing agency to us: ‘We’ve implemented a regime of failover tests . . .’, ‘We’ve launched a rapid review of patch status across the estate . . .’, ‘We’ve revamped the alerting system . . .’, ‘We’re holding the supplier to account . . .’, ‘We’ve increased our supplier management review meetings . . .’.
This type of language is sometimes justified by appealing to the concept of the blameless post-mortem, which rightly asserts that assigning blame to individuals is counter-productive, and leads to incident responses which involve firing people rather than finding root causes. However, blamelessness is intended to protect people and teams, not to mask the organisation’s accountability: it may encourage someone to speak about the server that has never been patched, the code that nobody understands, and the script that always has to be run by hand, but it does not excuse the existence of these weaknesses. It merely highlights the responsibility of the whole organisation to do something about them.
Enterprise technology is, too often, a domain of learned helplessness: a maze of complexity and mayhem which it feels impossible to control. It is not surprising, therefore, that when something goes wrong, we are inclined to attribute the failure to factors beyond our grasp. But, when we choose to take responsibility for any part of enterprise technology, as a CIO, a manager, a team leader, an architect, an engineer or an operator, I think it is vital for us to take ownership of everything within our scope, to recognise that it is ours, for good or for bad. This ownership should be apparent in our language as well as our actions.
The organisations that have recently committed breaches as a result of their own control failures may never show full ownership and responsibility for the tools they are building. But the rest of us can make a start with the rather more established, less exalted and more mundane technology on which people, businesses, economies and nations depend.
