- AI
- ambiguity
- APIs
- architecture
- augmented reality
- books
- bureaucracy
- career
- change
- Christmas
- cloud
- collaboration
- communication
- complexity
- computer history
- corporate life
- data
- decisions
- delivery
- devops
- end user tools
- ethics
- failure
- fear
- fundamentals
- gaming
- government
- halloween
- history
- humans
- hype
- identity
- infrastructure
- innovation
- language
- leadership
- learning
- legacy
- management
- measurement
- mental health
- money
- networking
- New Year
- operations
- partnership
- philosophy
- physics
- platforms
- prediction
- process
- procurement
- programming
- quantum
- reliability
- resilience
- risk
- science
- science fiction
- security
- shadow IT
- space
- standards
- strategy
- teaching
- teams
- technical debt
- technology advocacy
- testing
- thinking
- transformation
- TV
- virtues
- vision
- writing
Let’s avoid the equation ai_fragility * software_fragility = very_bad_things
Anyone who has built classical, deterministic software systems can tell you how fragile they are. Such systems perfectly follow precise, explicit instructions, and are incapable of operating outside those instructions. That means that, if you have written instructions which do not specify the behaviour you want, which are inconsistent with one another, or do not cater for unexpected values, your system will fail. If you are lucky, that failure will be loud: a crash with an ugly error message. If you are unlucky, it will be quiet: steady, ongoing corruption of data values which you won’t discover until later.
Anyone who has built newer, AI driven, probabilistic software systems can also tell you how fragile they are. Such systems treat inputs (prompts, context, training data and so on) as factors which affect the probability of getting different outputs. That means that, if your inputs are insufficient to tilt the balance of probabilities in favour of your desired outputs, your system will fail. If you are lucky, that failure will be obvious: results which are factually incorrect or logically incoherent. If you are unlucky, it will be obscure: results with subtle biases which you won’t discover until later.
Good news! It’s all their fault. Bad news! It’s your fault too.
Several organisations have recently made statements about cyber attacks which they committed using AI systems which they failed to control. The language of these statements is interesting, not just because of its technical detail (or lack of it), but in how different parts are framed.
The parts which describe the causes and impacts of the incidents talk in the passive voice, and ascribe actions to non-human actors and external parties: ‘The incident occurred . . .’, ‘The models identified and chained vulnerabilities . . .’, ‘a model accessed the Internet . . .’, ‘our third party evaluation partner . . .’, ‘a misconfiguration by an independent company inadvertently allowed . . .’, ‘some of the agents being tested had engaged in sustained, potentially harmful activity . . .’, ‘an AI agent took autonomous, unsanctioned action . . .’.
When we read these phrases, we might wonder whether any human beings work at these organisations at all, and what they were doing when the incidents occurred.
Do you need an umbrella or a lifeboat?
How do you prepare for things that just keep on going wrong? And how do you prepare for the day when everything goes wrong?
Last week I attempted to distinguish between reliability and resilience, claiming that reliability is the ability to keep services running despite routine failures, while resilience is the ability to restore essential services despite unexpected catastrophes. But that basic definition is not quite enough to disentangle these related but distinct topics. In this article, I’ll explore three more important differences between reliability and resilience.
Reliability protects normality; resilience strives for survival
Empowered must not mean abandoned
The most stressful experience in my entire career was when I felt truly abandoned. A very long time ago, I was trying to run a project which was dependent on the work of many other teams - but those teams weren’t interested. The system I was building needed new infrastructure - but the infrastructure wasn’t even ordered. The budget I had inherited was clearly insufficient for the task - but no-one was prepared to provide more money, or change scope, time or quality. And, although I have not always been fast enough to ask for help, this time I did not hold back: I took every opportunity to make sure that everyone around me knew the problems the team was facing and the help we needed, including my boss. But no help came. We were abandoned.
I like autonomy and don’t cope well with micro-management. But the feeling of being truly abandoned was far worse - I ended up leaving that job and have never forgotten the lesson.
Predicting doomsday: is your transformation initiative in trouble?
Can this initiative be rescued? Or is it doomed to failure?
If you’ve worked on any transformation initiative, I expect that you have asked these questions when things were difficult. If you were a member of the team, you may have wondered whether it was time to find a new project. If you were a leader of the initiative, you may have wondered whether you were up to the job. And if you were a sponsor of the initiative, you may have wondered whether you should apply your sponsorship elsewhere.
In my last couple of blog posts, I wrote about the importance of sustaining energy in large scale transformation, and offered some suggestions on how to keep going. James Cole, who leads architecture for the British Red Cross, asked in the comments for suggestions about when to persevere and when to think again - about how to detect that your initiative is doomed.
Anti-Shunya for careers
I knew that the concept of zero (at least as a positional notation) was invented in India, but I didn’t learn the Sanskrit term for zero (Shunya) until last month. My friend and colleague Balasubramanian Ganesh (HSBC’s CIO for Retail Banking and Wealth Management) introduced me to this term, and explained how we are using this concept (and a load of cool instrumentation) to drive zero defects, zero incidents and zero vulnerabilities for software engineering.
Ganesh also asked me on stage at our Engineering Summit in Pune, India, to share some reflections on my career, and to offer advice on what has helped me reach my current position of Chief Architect for HSBC.
My reflection was that, while Shunya is a great goal for software, it’s not something I have followed in career terms: my career has been full of defects, incidents and vulnerabilities. That’s because it has taken a while to figure out what I am good at, and it turns out that, to figure out what you’re good at, you have to do some things that you’re not good at.