Embrace *all* the fundamentals, of software, of models - and of humans

Photo credit: Steve A Johnson via Unsplash

I sometimes try to persuade business leaders that they should get to grips with the fundamentals of the technology they depend on. This often manifests as an exhortation to learn to code, not because I think they will be great coders (although they might be) or because we need more coders (although we do), but because engaging with the practical reality of building and running systems is the best antidote to the confusion, bewilderment and mystification that surrounds technology. And because they might make better choices about technology partnerships, investment, organisation, sourcing and strategy if they knew more about how it all worked.

For most of my career, this advice has been focused on traditional software: procedural code, written line by line by humans. However, as enterprises are attempting to figure out how to make effective use of AI, I think it is necessary for leaders to get to grips with the fundamentals of three types of work that goes on in their organisations: that carried out by software, that carried out by models, and that carried out by humans. I regularly see people confusing the characteristics of these three types of work, for example, claiming that an AI model is ‘like an intern’, or that there is no more need for a code base of software because AI will either make all the necessary decisions or generate all of the necessary code on demand.

With a lot more work, we could probably come up with a comprehensive taxonomy of the differences between software, models and humans. But here are some rough and ready thoughts which might be useful (note that I’m using the term ‘models’ to mean Large Language Models and other transformer based architectures, not the full range of machine learning models or other mathematical models):

Software is deterministic, efficient at transaction processing, and contains explicit, encoded judgement.

One of the reasons that I encourage people to learn to code is that it changes their relationship with software. It creates a feeling of empowerment (‘The machine does what I tell it to!’) combined with a daunting realisation of the amount of work required to exercise that power (‘I have to tell the machine everything that I want it to do!’). It dispels the illusion of software as a weird, capricious thing that has to be coaxed into behaving (‘Try exiting the app, waiting ten seconds, going into Maps, coming out and trying again.’) and replaces it with an image of software as a weird, capricious thing that misbehaves because we programmed it badly (‘This app has a terrible memory leak, and has to be shut down regularly to clear it.’)

Understanding software enables us to appreciate that it is deterministic (follows instructions precisely, and can do nothing else), efficient at transaction processing (fast and reliable at arithmetic, database updates, storage and retrieval) and that, if it can be said to contain judgement, it is because we encoded that judgement explicitly (the registration form rejects dates of birth in the future because we wrote the validation code that way).

Models are probabilistic, good at language processing, and emulate implicit, embedded judgement.

One of the reasons that I think that anyone engaging with LLMs should try to understand how they work is that they present a powerful illusion. Humans are predisposed to ascribe personality, authority, meaning and consciousness to anything that talks to them in their own language. It is not unreasonable when first confronted with an LLM to imagine that it carries expertise, has opinions and is thinking, especially when its interface goes out of its way to describe itself in those terms. We have to work hard at understanding to dispel the illusion.

When we do that work, we can learn appreciate that, unlike normal software, models are probabilistic rather than deterministic, and realise that our prompts are not instructions, but merely attempts to tilt the probabilities in ways that are useful for us. (This is why we should not be surprised when LLMs appear to disobey their instructions: they are not instructions, and ‘obedience’ is meaningless in this context.) That probabilistic nature means that LLMs are bad at transactional processing, but much better than traditional software at handling language. It also means that they can be used to produce outputs which look like judgement, although we should always be aware that those outputs are simply the product of the training data and the training process.

Humans are probabilistic, capable of original thought, efficient at making sense of new concepts, make real judgments, and are actually people.

Ideally, we shouldn’t have to say why it’s important to understand human beings. But there have been several recent examples of leaders who, in their haste to make announcements about AI, appear to have lost some of their appreciation of humans, so maybe it is worth revisiting the fundamentals.

Even if we thought of think as no more than components in the machine that is a modern enterprise, we can see how they differ from software and models. They are probabilistic, in the sense that they can be asked to follow rules, policies and procedures, but ultimately make their own choices (almost as if they had minds of their own). They may not be able to retain all of the data points and facts embedded in an LLM, but they are remarkably good at making sense of new information (you do not need to show a human a million examples of a new word for them to remember it). And while they may take comfort in cliches and received wisdom, they are also capable of completely new thinking (otherwise the training set for LLMs would be empty).

But the other feature of humans, of course, is that we shouldn’t think about them as components at all. Unlike software and models, they are people, deserving of respect, dignity and autonomy. I know that some people have claimed actual or incipient personhood for LLMs: I’m prepared to go out on a limb and say that those people are mistaken, and that LLMs are not people, are not conscious, and show no prospect of becoming either.

We have spent millennia trying to understand human purpose, motivation and values, and the best ways to live our lives. We haven’t always reached agreement on the answers, but it would be a shame to forget the fundamentals, just because we have new fundamentals to learn.

(Views in this article are my own.)