- AI
- ambiguity
- APIs
- architecture
- augmented reality
- books
- bureaucracy
- career
- change
- Christmas
- cloud
- collaboration
- communication
- complexity
- computer history
- corporate life
- data
- decisions
- delivery
- devops
- end user tools
- ethics
- failure
- fear
- fundamentals
- gaming
- government
- halloween
- history
- humans
- hype
- identity
- infrastructure
- innovation
- language
- leadership
- learning
- legacy
- management
- measurement
- mental health
- money
- networking
- New Year
- operations
- partnership
- philosophy
- physics
- platforms
- prediction
- process
- procurement
- programming
- quantum
- reliability
- resilience
- risk
- science
- science fiction
- security
- shadow IT
- space
- standards
- strategy
- teaching
- teams
- technical debt
- technology advocacy
- testing
- thinking
- transformation
- TV
- virtues
- vision
- writing
While you’re worrying about Hackerbot 3000, don’t forget about Human 1.0
Let’s use the term Hackerbot 3000 to refer to all of those AI models which supposedly present a new and unprecedented level of cyber threat. These threats have been framed in a few alarming ways: AI companies have declared that their models are too dangerous to be released; governments have imposed restrictions and regulations; and AI organisations have revealed that they have made successful cyber attacks on other organisations (although that’s not quite how they describe it).
I like the term Hackerbot 3000, partly because it is silly (and, despite the seriousness of these threats and their consequences, there is something silly about the rogue AI agent cyber attack narrative), but also because it captures the impression that we are confronted with faceless, implacable and dangerous machines which, just like the T-800 in the original Terminator film, can't be bargained with, can't be reasoned with, don’t feel pity, or remorse, or fear, and absolutely will not stop . . . ever!
Let’s avoid the equation ai_fragility * software_fragility = very_bad_things
Anyone who has built classical, deterministic software systems can tell you how fragile they are. Such systems perfectly follow precise, explicit instructions, and are incapable of operating outside those instructions. That means that, if you have written instructions which do not specify the behaviour you want, which are inconsistent with one another, or do not cater for unexpected values, your system will fail. If you are lucky, that failure will be loud: a crash with an ugly error message. If you are unlucky, it will be quiet: steady, ongoing corruption of data values which you won’t discover until later.
Anyone who has built newer, AI driven, probabilistic software systems can also tell you how fragile they are. Such systems treat inputs (prompts, context, training data and so on) as factors which affect the probability of getting different outputs. That means that, if your inputs are insufficient to tilt the balance of probabilities in favour of your desired outputs, your system will fail. If you are lucky, that failure will be obvious: results which are factually incorrect or logically incoherent. If you are unlucky, it will be obscure: results with subtle biases which you won’t discover until later.
Failing fast is fun until you fail at scale
Finding bugs in your code is like being betrayed by your own brain.
Sometimes the betrayal is blunt and obvious. The system generates errors or crashes outright. When you look at the code, you realise that you have mis-spelt a variable name (again), or left out punctuation (again), or called the wrong function (again).
Sometimes the betrayal is subtle and insidious. The system appears to be working properly, until you realise that the data in the database is drifting away from reality. When you look at the code, it looks fine, until you wrap your mind around the logic and discover that it is decidedly wonky. And this is logic that you created.
We’re still struggling to learn forty year old lessons about cyber security
I just finished re-reading The Cuckoo’s Egg, by Clifford Stoll. It’s a classic of cyber security, telling how Stoll, as an astronomer-turned-reluctant-sysadmin, attempted to resolve a 75 cent computer billing discrepancy, only to be drawn into a story of hacking, surveillance and theft, involving the FBI, CIA and NSA.
The action in the book takes place in 1986 and 1987, so it is full of references to technology which was modern at the time, but seems quaint and old-fashioned today: 1200-baud modems, pagers and daisy wheel printers. What is striking, though, is how much of the hacking activity uses techniques which are still in use today. I won’t spoil the book for anyone who hasn’t read it, but will say that it contains examples of identity theft, credential compromise, dictionary attacks, supply chain attacks, vulnerabilities in commonly used software and privilege escalation. The pattern of behaviour would suit a cyber criminal working today: probe for assets which aren’t properly secured; gain access; take over an unused account; escalate privileges; move laterally; exfiltrate data; erase evidence. And keep on coming back and doing the same thing over and over again.
A few phrases to help resist AI illusions
‘It can’t hurt you: it’s not real!’
You might hear those words from the hero of a horror, science fiction or fantasy film. They could be walking through a dream world, subject to a hallucinogenic drug, or under the spell of a sorcerer. They know that the things that they are seeing are not real, and that all they have to do is to try to ignore what they think they can see and hear. Telling themselves that what they are experiencing is not real is a guard against fear, against stepping off the path, or, worst of all, the temptation to talk back to the illusions.
Dealing with current forms of AI can feel like this. Not just because AI is surrounded by hype, marketing, inflated expectations and a big dose of FOMO. And not just because AI can be used to produce fake videos, fake images and fake words.
Please pay attention to the safety briefing
What do you do when the air crew ask you to put down your books or devices and pay attention to the safety briefing? Do you follow their advice, because this aircraft may be different to those you have flown on before? Do you study the safety card when the briefing is over? Do you check that you know the location of the life jacket, under your seat or in the compartment next to you? Or do you zone out, diving deeper into the mental limbo that air travel induces, waiting for the moment when you can start reading, scrolling or checking emails again?
I expect that most of us regard the safety briefing as a dull but worthy formality, and don’t pay as much attention as we should. However, I also think that perhaps we should regard it differently, and learn some lessons about how we achieve safety in enterprise technology.
Do your approvals processes make it easier to do nothing than to do something?
Have you ever seen a project plan which is a victim of the approvals process?
You can usually tell when a plan has suffered in this way. There may be long gaps when nothing is happening, followed by frantic activity around a monthly or quarterly date. Or there may be design and planning work which is crammed into the plan far too early, in order to hit an approvals board. There may even be a whole part of the plan – and the team – dedicated to gathering data and writing requests for approval.
Technology people seem to hate approvals and love them at the same time. Nobody enjoys navigating their way through complicated and arcane processes where every signpost says, ‘Not this way,’ or ‘Try again.’ And yet we don’t seem to be able to stop ourselves from creating more processes: approvals to purchase, approvals to hire, approvals to release, approvals to change, and approvals to change the approvals process. I've certainly been guilty of implementing processes which seemed like a good idea at the time, but less so in practice.
Technologists are always crying wolf (because of all the wolves)
The computer had failed. Unfortunately, it was the Apollo Guidance Computer (AGC), the machine that controlled the flight of a small, fragile spacecraft to the Moon and back. Fortunately, it wasn’t in space: it was on the ground, in a simulator.
Margaret Hamilton, the leader of the MIT team programming the AGC, often had to work weekends to meet the urgent schedule of the Apollo programme, and sometimes brought her daughter, Lauren, to work with her. Lauren liked to play in the simulator.
Learn to fail fast? Technologists fail all the time
From time to time, organisations attempt to learn new ways of working. They attempt to become digital or agile or data-driven or innovative. These attempts come with some familiar ideas: that we should execute through cross-functional teams who are empowered to experiment. One of these ideas is that we should not be scared of failure, and that we should learn to fail fast.
These attempts sometimes elicit eye rolls from the technology teams, especially the idea that we should embrace failure. This is not because these ideas are invalid: in fact, they are welcome to technology teams, and reflect their preferred ways of working. However, technologists have a different relationship with failure than non-technologists.
On the 2025 to-do list: figure out AI agents
Recent years have seen waves of AI innovation breaking faster than we can figure out good practice. Organisations around the world are working hard, not only to find ways to put AI to work, but to do so safely and responsibly. The AI to-do list often seems to be growing longer faster than we can strike items off it - but the only route to good practice is practice.
The advent of AI agents promises to add more items to the to-do list. The AI agent wave started cresting in 2024, and will break in 2025. Several major technology vendors and platforms already offer their customers the ability to build, configure and operate AI agents in an enterprise context, and the ability for consumers to build agents or to subscribe to existing agents, cannot be far behind (indeed, it is likely that, by the time this article is published, it will already be happening).
All I want for Christmas is speed and reliability
What were your Christmas lists like as a child? Were they modest requests for improving books and educational toys? Or were they, like most Christmas lists, an extravaganza of wishes, containing toys and sweets and games . . . and possibly a dragon and a unicorn?
It sometimes feels like our requirements for software development are like a Christmas list written by a very small child that wants everything at once. We’d like the ERP package to have world class embedded processes, but we’d also like it to customise it to meet our every need. We’d like the cloud platform to give us capacity on demand and pay as we go, but we’d also like it to be on-premise because we’re worried about security. We’d like an AI system that predicts our customer’s needs, but we’d like to do it without using our data.
And being a project manager or a product owner can feel like a harried parent who can’t possibly afford everything on the list, and is worried that Christmas morning is going to be a disappointment. If I give them all the customisations they want, then we can never take the upgrade. I can give them security on cloud, but they’ve got to understand the shared responsibility model. I can build them an AI model, but not unless someone’s prepared to share the data.
Playing the triangle: breaking down technology risk
What’s more risky? Building a hotel on the side of a volcano, or trying to deliver a software project?
Years ago, while working for a bank, I heard a talk from a colleague in the Structured Finance team: this team created complicated lending structures for projects that carried high degrees of risk. He told a story about a loan for a hospitality business that was building a new resort on the side of a volcano. Unsurprisingly, that project required some complicated risk models.
I put up my hand and asked, ‘How do you model risk for IT projects?’
The banker smiled and shook his head.
‘We don’t,’ he said. ‘Far too risky. They fail all the time and we don’t know why.’