Why were high profile UK retailers experiencing IT failures recently?

🔥There was a spate of high-profile IT failures last week with the likes of Sainsbury's, McDonald's, Tesco and Greggs all affected.

The Sainsbury's outage was caused by a "software update" and McDonalds said a “configuration change” was responsible for their failure. Sainsbury's also stated their failure affected £9m worth of orders.

I started to think about some possible underlying cultural and technological reasons for the failures including:  

1. The cartoon shown below highlighting a complex software architecture with a single point of failure. This single point of failure is usually shown as a legacy key component or line of code.
2. The culture inside the organisation specifically in the areas of Product and Technology.
3. The underlying software architecture and any potential technical debt. A sign of underinvestment affecting the health of the platform.
4. Very closely associated with the culture, how the software teams might be organised (thinking of Team Topologies again)

Some further ponderings made me think:

👨🏽👩🏻 Real people are involved in these incidents

There are people under pressure (I am assuming that they would be) to quickly resolve these issues who need support while fixing the issue.

They also need to be allowed to look at remedial steps to ensure the incident does not happen again once everything is back to normal.

I started to think about the cognitive load of these individuals and the teams they worked in. In a recently published article on risk, I mentioned the risk of continuing to plough more and more “stuff” into software teams whereby they no longer know what might be causing an issue.

These failures could be a direct side-effect of the sheer levels of cognitive load the teams are experiencing. I also explicitly mentioned one of the goals of Team Topologies was to help manage and reduce team cognitive overload.

🦸🏼♂️ If there are superheroes (lauded by the business) flying around fixing these issues then this also points to some cultural challenges. 

📅 "Why was such a change done on a Friday, we never do changes on a Friday!"

Those of you that have read the book Accelerate will know about the benefits of practices such as Continuous Integration and Continuous Delivery (CI/CD).

Prior to these practices being introduced deploying on a Friday was seen as a "no-no" Many companies today are deploying hundreds of times a day to their production platforms so doing this change on a Friday, is itself not an issue. 

You could ask yourself about the inherent risk of such a change on a Friday but again I would raise some further questions about the underlying culture such as:

🐘 Was this a "BIG" release e.g. were we deploying vast swathes of changes to the platform at once? Big releases by their very nature will increase the risk of something going wrong compared to smaller releases. Smaller releases are also generally easily to resolve.

💥 Are releases an EPIC event with many people across the company involved, in case, something goes wrong?

🔁 Has this change been run through a CI/CD pipeline?

🧪 What degree of testing automation exists in the CI/CD pipeline and why did this automation not pick up the issue prior to release?

⭐ If we were to assess the DORA software performance level of each of the affected organisations where would we find them? 

Now if these changes did not go through an automated CI / CD pipeline and the company’s software estate looks like the cartoon above, there is a high likelihood that culture / organisational design challenges have resulted in the resulting architectural issues. This would also explain the big and high-risk releases.

 🔍Failure leads to inquiry!

Companies should always be learning especially when things go wrong. The quote “Failure leads to inquiry” is from research on organisational culture by Ron Westrum. His work is also used in the Accelerate book, this hypothesised and proved that a specific Westrum organisational typology a “Generative – Performance Oriented” culture predicted software delivery performance and organisational performance.  

Culture aside for the moment, further questions need to be asked:

⚙️ If this was a code issue, can the unit tests / integration tests be bolstered?

🛑 Was this human error? The McDonalds failure was caused by a configuration error so what types of automation could be added to prevent this from happening again?

⚠️ What early warning signs can be built (or didn't fire) to pick an issue up before it gets to production?

🪙 Interestingly, there was some indications that some of these issues may have been caused by payment system failures. The UK Payments Systems Regulator (PSR) stated they "were aware of the recent payment issues" and were reviewing the situation. Was there a way to switch to an alternative when the primary one failed?

Closing thoughts

My thoughts will not necessarily reflect the exact cultural situation inside a company or the maturity of automation in their software ecosystem.

Even the most progressive companies in the world can experience failures as this is the nature of building any software. When the big cloud platforms such as Azure and AWS experience their own failures this severely affects us and our customers.

Learning is super important, and the likes of Netflix are always looking to build more resilient platforms by learning from their failures. After the AWS outage in 2011, Netflix implemented a raft of changes to protect their customers and themselves from anything similar happening in the future.

Hindsight is a wonderful thing but what was the cost of the investment to continually improve the software of the affected companies vs. the loss of revenue and reputational damage caused by these failures? 

Some actionable takeaways:

• We should use CI / CD approaches to reduce the risk of any customer impacting changes to our software.
• We should learn from failures to ensure we are constantly looking to improve the resilience of our software platforms.
• We should adopt approaches that tune the organisation's structure and culture to deliver value whilst reducing the cognitive load on individuals and teams.

📧 I would love to hear your thoughts on my ponderings, if you have any questions or would like to have a chat about any of the above let me know!

Ravi Nar | LinkedIn