Disaster Recovery: hope for the best, but plan for the worst

Most of us try not to spend much time thinking about huge events that could seriously damage our businesses. The good news is that you don’t need to if you’ve got a robust disaster recovery plan in place, as our Head of Technology, Kenny Watson, explains.

‘Disaster recovery’ is a term that most of us have heard, but relatively few people fully understand. It’s easy to think only of the huge events that make the news, such as the global outage that hit Facebook, Instagram and WhatsApp in late 2021. That lasted about six hours and is estimated to have cost the group anything up to $100 million just in lost advertising revenue.

A few weeks later, a piece of bad code in one of Amazon Web Services’ automated servers caused a snowball effect that disrupted everything from Disney amusement parks to Netflix to ticket sales for an Adele concert.

But it’s not just the tech giants that need to think about disaster recovery. Anything that could disrupt your business – human error, deliberate sabotage, extreme weather, and more – needs to be identified and planned for. That as true for SMEs as it is for the global giants.

Obviously backups are a fundamental part of that, but there’s much more to this issue than that. How will you recover those backups when you need to? If your office is inaccessible, or if it’s destroyed, where is everyone going to go? You need to have worked out answers to all of those kinds of questions, made sure that all employees know what procedures they should follow, and recorded everything clearly and in detail. In short: you need a plan.

The record of that plan is called a ‘runbook’. It should be stored somewhere that can be quickly and easily accessed in an emergency. It states how you will access your backups – there are several main ways of doing this, but they all need to be set up in advance. Another key aspect of the plan is where staff will go: perhaps working from home (do they have the equipment they’ll need? Can they communicate with each other?) or in a pre-prepared alternative location (how will they get there?) Again, all this needs to be thought through and prepared for.

Your HR and Operations departments have just a big a role to play as IT. Communication is key: the best plan in the world is no use if none of your employees know about it or understand how to implement it. Don’t assume that it will always be clear when the plan needs to be invoked. That should also be clearly defined in advance, so that managers know exactly when they should initiate disaster recovery – and when they shouldn’t.

Of course, frequent testing is also essential, as you would do with any business-critical function.

The disaster recovery plan also needs to reflect the business’s needs. For example, a large organisation with multiple servers that needs to be back up and running in minutes will probably use a product that creates a backup system in the cloud. These backups can be set up so that the system looks to users as if it’s based in the office.

About fifteen years ago I was responsible for the disaster recovery plan at a call centre. We had a back-up server at a location in Hamilton, along with the necessary facilities for about 10% of our staff to continue working. So if we had ever experienced a total shutdown – the office being destroyed in a fire, for example – within four hours we would have been back up and running at 10% capacity.

Bear in mind too that many different types of scenarios can require a disaster recovery response, and not all can be dealt with in the same way. So you’ll need separate plans for, say, lost data (whether accidental or malicious), complete system failure, and damaged or destroyed premises.

If this all seems like a lot of work, there are two things to consider. One is that being able to demonstrate that you have a robust disaster recovery plan has many benefits right now, even if you never have to use it. Shareholders and investors will be reassured to know that the business’s value won’t suddenly be wiped out. Potential clients see it as a strong point too – in fact, it’s now standard for big tender documents to ask for full details of your disaster recovery plan.

The other, of course, is the potentially huge problems you’re avoiding. Just look at the first lockdown in March 2020: the businesses that came through strongest were the ones that had a plan in place, or that were able to quickly come up with an effective and practical response. The ones who just set the main phoneline to divert to the boss’s mobile and hoped for the best… not so much.

In other words, a disaster recovery plan is equivalent to buildings insurance or critical illness cover. You hope the day when you need it will never come. But if it does, you’ll be very glad that you were properly prepared.