Danger! Testing Ahead

Danger! Testing Ahead: The first in a series of articles discussing testing technology systems.

There are many risks and challenges inherent in technology projects, and we can overlook risks, and even opportunities, some of which can arise from the testing process itself.

How might we improve our testing and quality processes, and what do boards and executives need to be aware of?

Note: For experts and specialists, this first article may appear to digress from the traditional discussion on software quality. Feel free to skip this and come back later in the series, or drop your comments and share your experiences in the comments below, with the aim of helping the community grow and improve.

Executive Summary

In this first article, I want to explore the technology implementation process, create a context for testing, and then without going into too much detail at this stage, highlight a benefit of changing our thinking about testing.

Testing is usually something that is focused in the middle of a process. Testing occurs just before somebody starts to use something.

In this article, I want to explore how testing can trigger continuous improvements, prevent deficiencies in critical plans, such as Business Continuity, when technology and data are potentially destroyed and the organization must find a way to continue, or Disaster Recovery, where technology and data are re-established.

The title is overly dramatic, isn't it?

If we were testing a fighter jet, supporting new clinical treatments, developing a medical device, protecting children, facilitating judicial process or protecting mine workers, it's not dramatic at all.

I have worked in all of the above contexts, so the title sits well with me and lays the foundation for the series, which explores when testing processes (or an absence of) have created risks, or even led to incidents.

But I do understand the thought. You're not launching rockets, or doing brain surgery (are you?). Perhaps you're just building software, or buying something and rolling it out.

Testing can be dangerous as the processes used can increase risk. Testing can also create a false sense of security. These points are all discussed as the series progresses.

Testing - It's not simple

Testing has a very broad context. So to be clear from the get go, I'm not limiting this discussion to "how to test the software that you're building". It's not simply about making sure software works as expected.

I'm exploring ideas, and inviting you to learn, or requesting that you share experiences of how we can connect many aspects of testing across differing contexts.

From the board throughout the entire organisation...how should we all think about testing?

It's always time for change

Digital Transformation. When will it ever end? It's a trendy term, so don't worry about that at all.

Instead, think about Evolution.

Change is constant. Organizations evolve (or die), and consequently the supporting systems and processes must also evolve.

Whenever there is change, there is change control, and testing falls squarely in this context. Whenever something new is required, a solution is proposed and delivered, and as part of delivery the solution is proven to have been tested, to ensure it meets expectations.

Testing can vary when you are building the solution, rather than just buying something off the shelf. But the risks in either case can still be significant.

Once solutions are delivered, they continue to add new features and capabilities over time. Again, this is why we say that change is constant.

To be provocative: It should end in attacks and disasters!

Any leader experienced in technology projects understands that the implementation of Business Continuity Plans, Disaster Recovery and Cyber Security processes, are critical for a sustainable organization.

At the board level, this all falls under Governance, Risk and Compliance (GRC).

For the less familiar, let's get concrete.

There are many summaries of technology delivery, such as plan, build and run, or build, test and deploy. Sounds simple doesn't it. Imagine what you want, build it, test it out, make it available for use. Job done.

But these very common summaries fail to acknowledge critical aspects of change - disasters and attacks.

As systems evolve, organizations become increasingly reliant on new features and capabilities. This leads us to the key points of this week's article.

When new technologies are deployed, or current technologies are enhanced, we need to ensure the business can continue to function in the case of technology absence or failure. Therefore, part of our testing activities can extend to ensure that key plans remain relevant and up to date:

Business Continuity Plans (BCP) anticipate disasters and prepare the organization to continue operation, whilst it concurrently recovers from a disaster.

Typically we mean something simple like, how will you keep operating when all of the technology disappears. Can you keep going whilst we replace everything?

Disaster Recovery Plan (DRP) details the process for re-establishing technology after systems and/or data have been lost. It is both a plan, and a level of redundancy, e.g. a second data centre, which provides the capacity to recover quickly from adverse events.

Of course, these are very simple explanations, but they help me get to the point...

Allow testing to drive continuous improvement

When an existing system is subject to change, how often do you go back and update BCP and DRP?

As technology evolves the action plans and infrastructure must evolve in step. If not, recovery will fail, be partial or may be delayed.

Therefore, I would suggest it's not necessarily enough to do annual reviews.

Testing of new or modified systems should encourage more frequent updates and testing of BCP and DR processes.

When the fan collides with the brown stuff, execution of recovery plans must deliver a complete and successful recovery. Not partial, without delay, and most critically, without creating a further setback. By setback, I mean, we can't do a recovery that puts the organization's technology where it was three months ago.

Technology programs and projects can encode the testing and more frequent execution of higher-level processes, so that nothing is overlooked, or deferred to end of year.

Testing - Article One - Key Points

• Why is testing dangerous in this context? The very presence of testing can lull us into feeling secure about our systems and processes. But a lack of testing, or lack of evidence showing tests have been updated, executed and results assessed, are all red flags for potential impact. If testing is too focused, it doesn't provide the required risk mitigation. If we test a new system, but the new systems capabilities are not re-established during Disaster Recovery, then again, adverse impact will occur.
• Testing of new and evolving systems provides multiple opportunities to ensure that higher-level plans and processes remain relevant and up to date.
• More frequent updates and tests of Business Continuity and Disaster Recovery means that any risk of regression can be identified and addressed. It also means everyone involved in the process of BCP/DR is more familiar with the process and ready to act quickly.
• For the board, understanding that BCP and DRP are well maintained, practiced and ready for execution is a critical part of Governance, Risk and Compliance.

What are your thoughts?

More formally, we tend to encode BCP and DR updates into annual review, or as part of significant programs of work.

Should we consider these key instruments of a sustainable business more as living documents?

Would more frequent updates be inefficient, or are they simply too detailed and unnecessary?

Cutting Edge

Where is the cutting edge of testing, BCP and DR? For example, automated cut-over for DR is becoming more of a norm.

What are your experiences in this area, and where is best-practice going?

Next Time

In the next article, we stare into the abyss of risks and incidents that have arisen from testing processes!