The software paradox: it’s rational to make plans; it’s irrational to expect them to work
Photo credit: Petr Sidorov via Unsplash
Software projects suffer from a paradox. They are typically expensive endeavours which are of great importance to the organisation which is executing them. It is rational to want to be assured that they will be successful, that they will not cost more than the organisation can afford, and that they will not take longer than the organisation can wait. At the same time, they are fundamentally unpredictable. Any software project worth doing involves work which no-one has ever done before, creating new logic and new capabilities, and integrating existing components in complicated and interesting ways. This is difficult work which involves intellectual challenge and problem solving. Things which look hard often turn out to be easy, while (rather more frequently) things which look easy turn out to be hard.
The introduction of AI into the software development process is unlikely to change this inherent unpredictability. Whether you believe that AI offers a modest productivity increment to experienced developers, or whether you believe that complex systems can be vibe coded over a weekend, it seems that the tricky 20% of work, the 20% that takes 80% of the time, is not going away. This is the work that requires imagination, domain expertise, original thought, persistence, grit and improvisation. It is also the work that is hardest to estimate and predict.
This paradox leads to common problems, and common responses to those problems. Unfortunately, those responses are often unhelpful, to the extent that they make the problems worse.
The script goes something like this:
SPONSOR: Why is this project taking so long? It was supposed to be done by now.
PROJECT MANAGER: The build phase is taking longer than expected. [TECHNICAL LEAD] can explain.
TECHNICAL LEAD: <Lots of complexity and detail>
SPONSOR: I don’t understand any of that, and I don’t care! Get this thing back on track.
PROJECT MANAGER: Of course. We’ll increase the frequency of meetings and give you a daily dashboard. We’ll also increase the detail of our status reporting, and add two more colours to our red, amber, green system.
SPONSOR: Hmmm. Good. And get <my favourite consulting form> to take a look at the estimates as well. I really want those development teams held to account!
If you’ve heard this script before, you already know the sequel in which, six months later, the project is still off track, its overhead has increased due to additional reporting and analysis, and the tasks in the plan appear to have acquired quantum properties, in which their status is less certain the more closely you look at them.
There are, of course, plenty of methods which attempt to recognise and acknowledge this inherent unpredictability. The practices of organising work into sprints and backlogs within Agile, of smaller, more frequent releases in DevOps, or of continuous delivery within a product management lifecycle, all attempt to provide a degree of near term control, while dispelling the illusion of long-term predictability.
And yet, that is still not how large, complex, strategic projects are planned, organised, funded and executed within most enterprises. Furthermore, when practices such as Agile, DevOps and product management are pitched to business sponsors, they are often positioned as ways of creating more certainty, not as ways of managing inherent uncertainty. I am confident that whatever stable practices emerge with the widespread adoption of AI will be positioned in similar ways.
I think that it is past time for us (and this ‘us’ means those of us who design, build and run technology for a living) to be open about the nature of software. It is rational to set goals, budget and deadlines within an enterprise; it is not rational to expect that software delivery will always conform to them without compromise. And it is even less rational to expect that increasingly forensic inspection of plans makes them go faster.
