Photo credit: Andre Hunter via Unsplash
Anger and frustration were entirely reasonable responses to last Friday’s global outage. While some people were relieved that online meetings were cancelled and that their inbox was quiet, others will have missed deadlines, disappointed customers and struggled to run their businesses. I expect that many of them shouted at their screens, their phones and (with less justification) their IT department.
It’s also normal to feel let down, exposed and worried now that the outage is over. What if it had gone on for two days, three days, or an entire week? What if it had started on a Monday rather than a Friday, or at the end of the month rather than the middle of the month? Will you be compensated, and will that compensation reflect the disruption caused? And, most importantly, what if it happens again? Is it okay that your organisation is so exposed to a single company?
These emotions will lead to demands for action. Some of those actions will be reckless (‘Cancel the contract!’) while others will be more considered and cautious (‘Check the contract.’). But, as with other global outages, there is a low chance that either of these immediate responses will have any impact: the reckless will run into the realities of the market, while the more cautious will yield, at best, some service credits.
I think that the most useful response, once the emotion has subsided, is to approach the problem step by step: to figure out what goals we want to achieve, identify steps we can take to achieve them, and determine whether we can afford to take those steps. (Note that I am focusing in this article on individual enterprises, not governments, who have a wider range of possible responses, and need to think about the implications for public services, the economy and society).
We can start with a very rough version of the problem statement represented by a day of dead email services, unresponsive spreadsheets, silent chat and empty video calls which goes something like this: our enterprise depends on a set of collaboration and productivity tools we buy from a well-known global provider. Now that these tools are provided as subscription services hosted on a cloud platform, it is possible for them to stop working, everywhere, all at once. The global provider has designed their platform to reduce the risk of such disruption. But we have been shown, starkly, that a global outage can happen, and that it can take a day to fix. We don’t know whether a day is the upper limit of potential disruption. What do we do to control the risk to our enterprise?
Writing out the problem like that implies some possible solutions.
For example, we might spot that the move to subscription based services exposes us to the stability of the platforms on which they run. We could reduce that exposure by running the services ourselves. But then, of course, we are exposed to our own ability to run infrastructure. It’s worth thinking back to the times when we ran on-premise email servers: did those ever go down? How quickly did we restore them?
Or, we could consider that, because we have been let down by a particular supplier, we should switch to another supplier. But, if we do that, we would have to figure out whether that new supplier was any more reliable. And we would have to estimate the cost of switching. How many days of disruption would we need to avoid to justify that cost?
Or, we might conclude that being exposed to a single supplier is too risky, and that we should have diversity of supply. But, for that to make any difference in a failover scenario, we would have to be using multiple suppliers for the same service. Are we prepared to pay that cost? Or is there a way of establishing a contingency service that we only pay for when needed?
You may have noticed that none of these potential solutions offers a clean, obvious answer: indeed, they all raise more questions. That’s because the problem we are grappling with is a question of risk and probability: no risk can be reduced to zero, and all risk decisions are trade-offs between cost, risk, convenience, flexibility, capacity, attention and whatever other dimensions matter to you and your enterprise. The aim of stating the problem is not to find quick fixes, but to give us some mental space in which to think through all of these trade-offs.
The good news is that there are a few principles which can help with this work.The bad news is that they involve yet more work. Still, who said that figuring out the correct response to a worldwide outage affecting hundreds of millions of people was going to be easy?
The first principle is that we should always look under the covers. Cloud-based subscription services such as those which failed last week follow a shared-responsibility model: roughly, you manage your consumption of the service and the provider manages provision of the service. In theory, you don’t need to know all the mechanics of how the service works. In practice, you need to know enough to judge what risks you are running. The shared responsibility model does not imply deliberate ignorance: rather, it implies that providers have a responsibility to explain how their services work, and consumers have a responsibility to make informed judgements about risk.
The next principle is to always put a price on things. As we’ve said, managing risk is a trade-off, but it’s difficult to make that trade-off if you don’t know what anything is worth. What was the cost of a day’s disrupted business? How does that compare to switching or contingency costs? Talking in such terms may feel cold and emotionless, especially immediately in the aftermath of a global service failure, but, at base, risk calculations need to be quantified. It’s reasonable to add some qualitative considerations afterwards: for example, you may say that you are never prepared to put customers in a position where they can’t reach you, not to protect your revenue, but out of a sense of duty. But, even if you are always prepared to pay this price, you need to know what the price is.
And the final principle is to figure out which services matter most. In theory, enterprises do this all the time: they sort their services into tiers, assign SLAs to them and so on. But how many companies emerged from last Friday’s outage feeling that they had assigned the right level of criticality to email, video-conferencing, spreadsheets and word processing? And how many released that they were rather more dependent on these services than they had ever imagined? Making good risk decisions requires looking at what the people in your enterprise actually do all day, and which tools they rely on, even if they’re not on the critical list.
Outrage in the face of an outage is understandable, but it’s not a plan for action. If we state the actual problem clearly, think it through, balance cost and risk, get a grip on the underlying service, and know what matters most, we stand a chance of building a real plan.