Good complexity, bad complexity
Technology architects give complexity a bad name.
We often make the mistake of talking as if complexity is inherently bad. We then compound that error by saying that complex architectures are difficult to understand, slow to charge and hard to recover. Then we make things even worse by counting big countable objects (systems, machines, products, versions and so on) and believing that we have measured the complexity in our architectures.
Some things are just inherently complex.
Fortunately, there is a way to correct these mistakes, but it requires us to embrace four ideas:
complexity can be good
complexity can also be bad
counting objects is a bad way to measure bad complexity
there are some better measures of good complexity.
The topic of complexity is itself complex, so I’m going to focus on complexity in enterprise software for today: complexity in enterprise infrastructure is a whole other topic for another blog post.
Good Complexity
There are many things in the world which are better for being complex.
The Internet is complex, societies are complex, and the human brain is the most complex entity we know in the Universe. I don’t believe that we would want any of these things to be less complex.
What makes these complex things good?
That’s a difficult question to answer in depth, but I think that it has something to do with a few key features: they are made up of simpler elements; those simpler elements have a degree of autonomy; those elements interact in a finite number of ways; and sophisticated behaviour emerges from the interaction of those simpler elements.
In software, of course, these are also some of the features of the microservices architectural style. While it can be difficult to implement, this style of architecture has some well known advantages: it is more flexible, more resilient and easier to manage than other, older architectural styles. It is an example of good complexity.
Bad Complexity
We can easily think of examples of bad complexity. The road system which is impossible to understand or escape, and which always brings you back to the same place in the same lane. The bureucratic process which requires you to fill in form after form and seek approval after approval, with no end in sight. The organisational hierarchy where it is never clear who can say ‘yes’ although everyone seems to be able to say ‘no’.
Bad complexity in enterprise software in a lot of different ways, but I think that there are two dominant examples.
The first is familiar and easy to see. It is the set of systems with obscure names and functions which interact through rigid interfaces, and where every change to one part of the architecture risks breaking another part of the architecture.
The second is also familiar but less easy to see. It is the single system which appears simple at a very superficial level, in that it only occupies one box on the architecure diagram. However, the simplicity of that box conceals a labyrinth of logic which few people if any understand, where every change to every line of code is approached with fear of unintended consequences.
These and other examples of bad complexity have common characteristics: obscurity (it is difficult to understand what everything does); interdependence (changing one thing impacts other things); lack of purpose (things do not have a clear, simple reason for existence); and overlap (many things try to do the same job, and it is not clear which one to use).
Bad Ways to Manage Bad Complexity
Bad complexity is a problem, and as architects it is our job to do something about it.
Unfortunately, the approach we have taken for many years is not always effective: we have counted a set of things (typically applications), and we have tried to persuade people that fewer applications is better than more applications.
Sometimes this measure helps: bad complexity is sometimes created by needless duplication, and if counting applications helps us identify and remove this duplication, it can be useful.
However, this measure can also be unhelpful: if I am solely concerned with the number of applications I manage, then I may convince myself that the right thing to do is to consolidate logic contained in many applications into a single application. This would be a good way to create bad complexity: to concentrate risk and to complicate code.
This measure is also being rendered irrelevant by the rise of the micro-services architecture style. If I successfully convert bad complexity to good complexity by decomposing a monolithic application into a range of autonomous components I have done a good thing, even though my count of things wil surge.
Better Ways to Measure Good Complexity
If our trusty old tool of measuring the total number of applications doesn’t work any more, what is the alternative?
I do not believe that we have found the perfect tools yet, but there are some clues in the characteristics of bad complexity. If bad complexity is characterised by obscurity, interdependence, lack of purpose and overlap, then good complexity is characterised by clarity (everything is self describing in clear terms), autonomy (everything can be deployed, changed, managed and scaled independently of other things), purpose (everything has a clear, well-bounded and simple reason for existence) and uniqueness (nothing else shares this purpose).
Measuring those characteristics is hard, much harder than just counting things. But nobody said that this job was going to be easy.
Don't Fear Complexity
Complexity is not bad and we should not be scared of it. As architects, it is not our job to eliminate or even reduce complexity, it is our job to convert bad complexity to good complexity, and we need to invent the tools that will help us do this job.
