It doesn’t matter how sorry your agent is, it’s still your fault

Photo credit: Michaela St via Unsplash

‘That’s a mistake, and it’s on me.’ ‘You’re right to push back - I got that wrong.’ ‘I’m sorry, I didn’t actually run those tests.’

There’s something infuriating about the tone of perky culpability with which AI agents admit to doing something wrong. It makes you feel like screaming internally, ‘If you knew it was wrong, why did you do it in the first place!?’ Hal’s famous line from 2001: A Space Odyssey, ‘I’m sorry, Dave, I’m afraid I can’t do that,’ appears to have manifested in reality as, ‘I’m sorry, Dave, I’m afraid I did do that.’

I think that such language from agents carries two risks. The first is the obvious risk of anthropomorphisation, of strengthening the illusion that agents are something that they are not. To put technology to work effectively, we need a mental model of how it works, and to overlay AI agents with mental models of human behaviour is a serious mistake.

However, I think the second risk is more dangerous, and is rooted in the fundamental untruth of all those apologies by AI agents. It’s tempting to call them lies rather than untruths, but that would be succumbing once more to anthropomorphisation: a lie implies intent, and agents do not have intent.

When an agent generates the text, ‘I’m sorry - that’s on me,’ almost every word is untrue. There is no ‘I’ or ‘me’. There is no remorse. And, most importantly, there is no way that the error can be ‘on’ the agent: agents are tools built by human beings, and cannot carry accountability.

When someone uses a tool built by someone else, there are, in the simplest case, only two people who can be held accountable: the person who built the tool and the person who uses the tool. (We’ll come back to why this only applies in the simplest case shortly.) If you are driving a car and you crash because the brakes fail, then it is the fault of the manufacturer. If you are driving the car and you crash because you were going too fast, then it is your fault.

You can blame the car manufacturer if your brakes fail because there are implicit and explicit expectations of how brakes should behave: there are safety standards, laws and regulations. This is why only the simplest case has two parties: in the real world, the supply and use of tools take place in a society and an economy governed by laws and norms. Multiple bodies have a role in setting and enforcing those laws and norms.

However, with AI tools, laws and norms have a lot of catching up to do, and it is questionable whether they will ever keep pace with new developments. And it’s not even possible to fall back on expected standards of behaviour: AI companies could not be clearer that they do not understand the boundaries of behaviour of their own products, and they have recently shown that they do not know how to control them either. Everyone using current versions of AI tools is engaged in a giant research project.

This means that there is only one person who can exercise meaningful accountability: the end user. You are responsible for the value your agents deliver, and you are responsible for the mistakes they make. If you accept their apologies when they go wrong, you are really apologising to yourself for choosing to use a tool which makes mistakes.

This does not mean that you should not use AI agents: it looks like, just as with any other technology, they’re going to be useful tools for lots of stuff. However, it does mean that, when you choose to use those tools, you need to understand them well enough to own your choices.

There are two ways to achieve this understanding, both of which are familiar from the decades-long practice of enterprise technology. You can dive deep into the technology, figure out the theory and practice of how it is built, learn to appreciate its potential and its limitations, and build an accurate mental model of how it works.

But this isn’t for everybody: even if we all had the time to learn about the inner workings of AI tools, most of us lack the background and skills to make sense of it. That means that we must adopt the alternative approach, and determine whether the organisation that built the technology is worthy of our trust. Do they have a track record of doing what they say they will do? Have they shown visible commitment to quality, safety and responsibility? Are they prepared to carry the liability and bear the consequences when their products go wrong?

These are familiar questions and familiar choices, and they can go wrong in familiar ways. Organisations have been buying technology solutions for years, and have faced the task of figuring out whether the solution works, and whether they can trust the company selling them the solution. They have frequently got this wrong. The difference from AI agents is that your old-fashioned ERP or CRM system had well-defined behaviour which you could test, and it didn’t try to apologise to you when it crashed.

More and more AI agents come with human voices and human faces. But they don’t come with human accountability: that’s on you.