What your shiny new agent can learn from your old-fashioned IT department

Photo credit: Mushvig Niftaliyev via Unsplash

At the root of the current AI panic is the idea that, when we give AI agents goals, such as beating a cyber test, or retrieving government health data, they will somehow evade their human gaolers, ‘break containment’ and wreak mayhem across the Internet. The implication, reinforced by calls for a pause or a slower pace, is that these agents are so clever and devious that they outsmart even the people who built them. They’ve gone rogue! We couldn’t stop them! (So perhaps we’re not liable . . .)

Hype aside, we now have plenty of examples of goal-seeking systems seeking goals which have been set by others, taking unexpected (and insufficiently controlled) steps in the pursuit of those goals, and causing unplanned harm in the process. To some observers, this is a completely novel phenomenon arising from the autonomy of AI agents. To others, it is wearily familiar.

If you’ve spent your career working in IT departments on large projects, you have plenty of experience of people (business executives, project sponsors, your boss) firing high level goals (increase customer retention; reduce cost; achieve compliance with new regulations) into a complex mechanism (the IT department) filled with expertise and intelligence (analysts, architects, engineers) with access to enterprise assets (systems, data, budgets, tools) and ending up with results that they didn’t want or expect (cost over-runs, technical debt, migration failures).

Sometimes these bad results are due to the planned action failing to fulfil the goal (the project fails, the system crashes), sometimes they are because the act of goal-setting incites unexpected behaviour (when you set the goal of increasing the number of support tickets resolve, you find support staff creating unnecessary tickets), and sometimes they are because pressure to reach the goal exceeds the pressure exerted by guardrails (when the project sponsor has fallen in love with a particular product, the RFP is pre-configured to pick a winner).

I should say at this point that I write those paragraphs with a combination of affection and regret. I have been part of corporate IT departments for most of my professional life, and will continue to advocate for the value they provide and the people who work in them. But I cannot pretend that they are always efficient machines for turning strategic goals into good outcomes. They are made up of complicated humans, messy processes and complicated technology which, too often, add up to a machine for turning hope into disappointment.

Fortunately, we can learn some lessons from this disappointment. Let’s start with just two:

Nobody knows what they really want

If you observe the amount of effort that goes into building business cases, creating investment portfolios, running procurement processes, writing user stories and capturing requirements, you might imagine that every IT project starts with a clear, detailed, precise, logical, coherent expression of what it is trying to achieve.

That impression collapses the moment you work on an actual IT project, and realise that most of your time is spent trying to figure out what your users want, reconciling contradictory goals, and pointing out that they really do want features like security and stability, even if they don’t know it. Eventually, if you are lucky, skilled or experienced, you will discover an interactive product development loop which enables you to discover the true nature of goals at the same time as you build the solution that meets them.

This experience shows us that the dream of AI agents, in which an end user simply expresses a goal, and the AI agent (subject to guardrails) builds a plan and fulfils this goal, is an illusion, and it continues to be an illusion as long as we simply try to specify the goal better. Many people today lament that the experience of using AI can be frustrating as it keep getting the results wrong: sometimes that is a feature of AI, sometimes it is part of the process of discovering what you actually want.

Nobody knows how anything works

Once again, if you observe the amount of architectural diagrams, design work, governance meeting, specifications, debates and decisions that go into an IT project, you might expect that it starts with a clear view of what needs to be built, what needs to be changed and what the solution needs to connect to.

This expectation collapses the moment you start trying to build things, and realise that you are dealing with incompatible data structures, interface standards written decades apart, environments that don’t exist and code that nobody understands. Eventually you realise that your design needs to be as flexible and iterative as your discovery of goals: you will discover what works through the process of trying to make it work.

This experience shows us that the dream of AI agents, in which the agent is simply hooked up to a bunch of back end systems, possibly via a gateway or some MCP servers, and is able to execute on its goals, is an illusion, and it continues to be an illusion if we respond simply by trying to create a tighter spec. The AI agent that keeps on breaking things is sometimes due to the inherent inaccuracy of AI, and sometimes part of the process of discovering how things work. It’s also a reminder that you’d better have a way of testing your agents, and that the software development lifecycle exists for a reason.

AI agents are not the first non-deterministic, goal-seeking mechanisms that we have created. We have lots of experience with creaky old systems made of humans, processes, systems and data. Thinking carefully about why they never quite do what we want them to do might help us to get agents to do what they are supposed to.