The role of QA in a DevOps world

If you believe what you hear, DevOps is the latest “big thing”. According to a recent survey, high performance teams practicing a DevOps culture deploy 30 times faster and have 60 times fewer failures than low performing teams. Companies having adopted DevOps have been reported reducing the average time it takes to get a feature into production by 83%. Attractive targets indeed!

However, like many new (and not-so-new) practices, DevOps also tends to be a bit misunderstood. Many organisations focus on the automated deployments and virtualised infrastructure. They have dedicated “DevOps teams” who specialise in such things, and create slick build pipelines with tools like Jenkins or Go. And many consulting firms make things worse by pushing “DevOps adoption strategies” that focus almost entirely on Continuous Delivery and automated deployments.

Deploying a broken application faster will not make it any less broken

In fact, there is a lot more to DevOps than simply automating a build pipeline. DevOps is not an activity, it is a culture, and a culture that goes much deeper than what appears to the naked eye. You could say that DevOps involves creating a culture where software development teams works seamlessly with IT operations so that they can work together to build, test, release and update applications more frequently and efficiently.

When you see a team that delivers 30 times faster, it is easy to be dazzled by the slick deployment, and not see everything that happens behind the scenes, that makes this rapid release pace both possible and worthwhile. It’s a bit like seeing an actor deliver a star performance, without thinking about the years of training and practice that got her there.

Without a quasi-obsessive attention to quality throughout the process, all the automation in the world will be to no avail. Without a clear and common understanding of how a feature is expected to deliver value, there is a good chance that efforts will (at least sometimes) be wasted on building the wrong feature.

In organisations that have successfully adopted DevOps, the software delivery process shows what I like to call the “3 Rs”:

• Rapid
• Reliable
• Relevant

Rapid is what typically comes to mind when we think of DevOps - getting the code into production fast. And you can’t have rapidity without automation, both in terms of automated deployment and in terms of automated testing.

Reliable means that what we are delivering into production will work, or, in the worst case, if it doesn’t we can get a fix into production. Automation is essential here too, in particular a good balance of automated testing at different levels. But it is a mistake to believe that automation by itself will ensure that your software is reliable.

Relevant means that you are deploying the features that will make the most business impact, and provide the highest relative return on investment, at this point in time.

All of this hinges on having a strong Culture of Continuous Quality embedded throughout the process.

A Culture of Continuous Quality helps ensure that you automate the right tests, in the most effective way. A Culture of Continuous Quality not only checks whether a feature is build to specs, but questions whether the specs represent the best way to build the feature, or even whether a feature is the best use of the IT budget at this point in time.

More importantly, a Culture of Continuous Quality relies on many things that come before effective Continuous Delivery or DevOps is even feasible. These include:

• Collaborative practices aimed at building a shared understanding and an aligned sense of purpose within the team (“building the right software”). This includes practices such as Behaviour Driven Development, Impact Mapping, Story Mapping, Example Mapping, XSCALE’s Agile Product Management, and others;
• Technical Practices (or “building the software right”), such as Test Driven Development, Pair Programming, Automated Acceptance Criteria, and Continuous Integration;
• A Culture of Trust, giving teams the autonomy and psychological safety they need to deliver creative, innovative, high value solutions, and avoiding the pitfalls of micro-management and local optimisations

Which brings us to the role of testers in all this. The culture of continuous quality is not just the responsibility of testers, but they can and should play a key role in making it a reality.

"The reports of my death have been greatly exaggerated.” - Mark Twain

Some organisations also seem to believe that they can get away with fewer testers when they adopt a DevOps. After all, if all the tests are automated, who needs manual testers?

As we have seen, Continuous Delivery, and DevOps more generally, is untenable with this Culture of Continuous Quality throughout the development process. This includes, but is in no way limited to, test automation. As a developer of open source test automation tools, and as a coach in effective test automation strategies, I can state quite categorically: If someone tells you that you can get rid of your manual QA activities through automation, they are trying to sell you something.

It is true that you don't have time for a separate, dedicated testing phase in a DevOps environment. But that doesn't mean there is no manual testing going on. On the contrary, manual testing, in the form of exploratory testing, performance testing, usability testing and so on, is done continuously throughout the project, not just at the end. Techniques such as Feature Toggles mean that a continuous stream of deployed artefacts doesn't mean putting features into the hands of the users before they are ready. The role of the tester in a DevOps project involves exploring, discovering, and providing feedback about the product quality and design, as early as it is feasible to do so, and not just at the end of the process.

Testers certainly should learn to leverage automated testing effectively. Automated tests at all levels are invaluable - there is simply no way to have enough confidence in your code to deploy several times a day otherwise. But testers should not have to take care of automation alone. The best automated tests are the result of tight collaboration between developers and testers, where developers can leverage their understanding of the code to make the automated tests more efficient, and testers can leverage their specific testing aptitudes to ensure that the tests are testing the right thing. The Screenplay pattern mean that testers can productively contribute to test automation efforts, without compromising the quality or maintainability of the test suite. even with relatively little experience with test automation.

So testers are not made redundant because you have the technical capability to deploy ten times a day. On the contrary, testers play a vital role in ensuring that the application that gets deployed ten times a day is worth deploying.

You can learn more on growing high performance Agile/DevOps teams in this whitepaper.

We help organisations get more value sooner out of their IT teams - get in touch to learn more!

JOHN FERGUSON SMART is an international speaker, consultant, author and trainer well known in the Agile community for his many books, articles and presentations, particularly in areas such as BDD, TDD, test automation, software craftsmanship and team collaboration. John helps organisations and teams around the world deliver better software sooner through more effective collaboration and communication techniques, and through better technical practices. John is the author of the best-selling "BDD in Action", as well as "Jenkins: The Definitive Guide and "Java Power Tools", and also leads development on the innovative Serenity BDD test automation library.

JAN MOLAK is a full-stack developer and coach who spent last 12 years building and shipping software ranging from award-winning AAA video games and MMO RPGs through websites and webapps to search engines, complex event processing and financial systems. Jan's main focus is on helping organisations deliver valuable, high-quality software frequently and reliably through implementing effective engineering practices. A prolific contributor to the open-source community, Jan is the author of the Jenkins Build Monitor helping thousands of companies worldwide keep their builds green and the delivery process smooth.