Risk isn’t boring if you are a lion tamer (and we are all lion tamers now)

Photo credit: Ivan Dias on Unsplash

I was at an event recently where a presenter apologised for talking about technology risk. They anticipated that the audience would perceive it as a dry and boring subject.

I can see why they felt that they needed to make this apology. When people hear the words ‘technology risk’ they often think about processes, risk registers, controls, reviews, assessments, forms, and all sorts of reasons that they can’t go faster.

I don’t think that we need to think about risk this way, though. Risk clearly isn’t boring if you’re doing a dangerous job. If you’re a tightrope walker, an astronaut or an airline pilot, then the danger is immediate and obvious, and risk management is an essential part of your work. Checking that the safety net is in place, that your oxygen tank is full, and that the doors are closed properly are matters of life and death.

I also don’t think that risk is boring even if you’re not doing a job which has safety of life implications. If you’ve built a few technology systems, then you’ll know that a lot of the most interesting problems arise when you ask the questions ‘what could go wrong?’ and ‘what are we going to do about it?’ How will you make sure that this system keeps running when that system has failed? What will we do when two copies of the same data disagree with each other? How can we achieve resilience, consistency and geographical separation without breaking the laws of physics? Risk is interesting if you like hard problems.

Right now, the technology industry, and all industries that use technology, are thinking and talking about the risk posed by artificial intelligence, particularly Generative AI. I’m not particularly interested in the existential threats of AI: I don’t believe that these risks are present yet, and I am not convinced that the development of current varieties of AI will lead inevitably to their manifestation. I think that there are more pressing questions about bias and reliability, and that we need to answer those soon.

There is a particular characteristic of Generative AI which makes risk management in this field especially interesting: the precise behaviour of the system is unknown. That is, it is impossible to test a Generative AI system with the full range of prompts that it will encounter, so it is impossible to know exactly how it will respond. This is different from pretty much every other technology product I have bought or built in the past: they have been computer systems with precisely defined behaviour, where deviation from that behaviour was a bug. Generative AI systems have broadly defined behaviour, the precise nature of which is discovered through interaction with the developer, the trainer, the tester or the user.

Perhaps, though, from the perspective of risk management, Generative AI is not unique. Its unpredictable behaviour may be a feature rather than a bug, but it is far from the only technology which behaves unpredictably. Anybody who has ever used a technology device has encountered some form of unexpected behaviour - a glitch, a freeze or an error - often solved by turning it off and on again. When we put these devices and the software that runs on them together to form large, complex, interconnected systems, we get even more unexpected behaviour. And when we connect those systems into large, complex, interconnected, globe-spanning systems of systems, it becomes very difficult to predict what will happen when things go wrong.

We are using techniques such as fine tuning, embeddings and meta-prompting to get unpredictable AI systems to do what we want them to do. We are also continuously learning new ways to monitor, measure, manage and respond to the behaviour of technology systems collectively made up of billions of lines of code and millions of devices - and to protect services and the people that depend on them when things go wrong. Both are forms of lion taming.

Through enterprise technology we have built the hidden engines of the world and made our lives dependent on them. Figuring out what happens when they go wrong is one of the most important things we can do. Technology risk management does not have to be boring.