Let’s avoid the equation ai_fragility * software_fragility = very_bad_things

Photo credit: FlyD via Unsplash

Anyone who has built classical, deterministic software systems can tell you how fragile they are. Such systems perfectly follow precise, explicit instructions, and are incapable of operating outside those instructions. That means that, if you have written instructions which do not specify the behaviour you want, which are inconsistent with one another, or do not cater for unexpected values, your system will fail. If you are lucky, that failure will be loud: a crash with an ugly error message. If you are unlucky, it will be quiet: steady, ongoing corruption of data values which you won’t discover until later.

Anyone who has built newer, AI driven, probabilistic software systems can also tell you how fragile they are. Such systems treat inputs (prompts, context, training data and so on) as factors which affect the probability of getting different outputs. That means that, if your inputs are insufficient to tilt the balance of probabilities in favour of your desired outputs, your system will fail. If you are lucky, that failure will be obvious: results which are factually incorrect or logically incoherent. If you are unlucky, it will be obscure: results with subtle biases which you won’t discover until later.

There is a clear attraction in combining the strengths of these two different types of system. Business processes usually start with a customer interaction, trigger some form of decision making and data manipulation, and finish with another customer interaction. Customers prefer language to forms, and probabilistic AI systems offer the potential of interpreting that language, discerning intent and extracting data. Deterministic systems are still best at handling that data and intent, retrieving and updating records and performing calculations.

However, there is also a clear risk of multiplying the fragility of these two different types of system. This became apparent in the recent cyber attacks carried out by AI companies. Security vulnerabilities are a consequence of the fragility of deterministic software systems: weaknesses derived from an inability to cope with unexpected input or deliberate subversion. And the unexpected exploitation of those vulnerabilities is a consequence of the fragility of probabilistic software systems: unexpected outcomes which derived from a failure to anticipate open ended, goal-seeking behaviour.

This multiplication of fragilities might seem inevitable when experimenting with cyber attacks: after all, the goal of such experimentation is to determine what can go wrong (although not in the ways that they actually went wrong). However, fragilities can multiply even in ordinary circumstances.

Imagine a situation which may be happening in an office near you. An operational processing team, under continuous pressure to improve throughput and productivity, with a growing backlog and a demanding boss, is given access to a new set of tools to build AI agents. Eager to use these new tools (and to respond to their boss’s exhortations to do more with AI), they give their credentials to the agents, point them at their user interfaces, and ask them to automate their processes. Everything seems to work at first. However, the combination of a probabilistic system (the AI agent) interacting directly with deterministic systems (existing legacy) through an interface designed for humans creates failures. When the team is lucky, these failures are loud and obvious: screens crash out with error messages. When they are unlucky, they are silent and obscure: incorrect decisions are made, data fields are misused and controls are negated.

This is not an entirely new problem: we have been putting probabilistic entities with unbounded behaviour, otherwise known as humans, in close contact with deterministic software systems since computers were first invented. Over time, we have learnt how to anticipate human behaviour when using computer systems, and to build interfaces which handle exceptions. However, we still get it wrong, and every UI remains a potential source of vulnerability and error.

While we can learn lessons from our experience of building interfaces for humans, this experience won’t tell us everything we need to know to enable classical systems to cope with AI agents. As we are learning, the behaviour of such agents can be even wilder and more unexpected than that of humans, and happens at computer speed.

Protocols such as MCP may offer a compromise approach: a way of mediating between resources controlled by deterministic systems and behaviour generated by probabilistic systems. However, this does not mean that the problem is solved: we cannot simply throw an MCP layer across existing UIs, APIs and databases and imagine that everything is going to work.

I suspect that we still have some basic architectural concepts to discover. I also suspect that these will eventually include two layers of insulation: one which protects probabilistic systems from the precision driven fragility of deterministic systems, and one which protects deterministic systems from the open ended fragility of probabilistic systems.

That might sound complex and difficult. It might even sound tedious and troublesome. But decades of computing have shown us that the most interesting engineering and architectural challenges lie in the boundaries and interfaces: we shouldn’t be surprised to find them in the borderland between determinism and probability.