Quality Engineering for CEOs: Exploring QE Strategy (Post 3)

TL/DR

• Smart quality engineering (QE) strategies can drive competitive business advantages.
• QE is a broad discipline that seeks to continually improve software release processes.
• A QE strategy can contain ~15 elements that include software testing concepts, quality assurance approaches, systems, economics, resources, and indicators. 
• The CTO or next highest engineering leader generally owns your company’s QE strategy.
• As CEO, you will want to hold your engineering leaders accountable to evaluate, create, monitor, and improve your firm’s QE strategy.
• You, the CEO, will likely use generous questions with your team to ensure that their strategy is strong, energizing, differentiated, and activated.

Introduction

Welcome to the third article in the series Quality Engineering for CEOs. My goal is to help chief executive officers (CEOs), and the software teams that work with them, appreciate why quality engineering is worth deep understanding. 

In my first article, I teed-up my background as a CEO and how I came to my viewpoints. 

In my second post, I tackled the core spend questions that CEOs— and their CFO counterparts—often ask. There, I offered that software engineering personnel spending should generally drive quality resource budgets. I also asserted that companies can often target 10%-20% of engineering personnel costs for quality resources.

Today, I’ll dive into a key question for cross-industry CEOs dependent upon software to power their digital customer experience: 

What does a great quality engineering strategy entail? 

To guide our exploration of this question, I’ll use a perspective on strategy derived from Michael Porter’s work in the 1990s. “A strategy is a set of activities that an organization undertakes to gain a competitive advantage. Strategy should dictate fundamentally how we drive our investments and resources. With a sound budget and a smart strategy, we can drive the success of our companies.”

As you can imagine, there’s not a one size fits all approach to quality engineering strategies. What’s important is that companies understand strategic elements and options so that they can thoughtfully build and experiment with a strategy that is right for them. 

Before we dive into strategy, let’s make sure we’re clear on the definition of quality engineering and how it relates to quality assurance and software testing.

Definitions: Software Testing, Quality Assurance, and Quality Engineering 

Software testing, quality assurance, and quality engineering are terms that are sometimes used interchangeably. In today’s article, I’ll use an inverted pyramid for conceptual clarity, giving a tip of the hat to the original pyramid of software testing as inspiration (and also winking at the evolution and rethinking of the pyramid through various anti-pattern explorations, debates, and experiments). In this model I offer that quality engineering can be seen as a higher order concept that includes quality assurance and software testing.

From the bottom up, I will use the following definitions: 

Software Testing: a set of activities undertaken to discover bugs, issues, and/or improvement opportunities. There are many other definitions, including IBM’s and Wikipedia’s. My favorite book on software testing comes from 2001: Lessons Learned in Software Testing. I still find it relevant today and keep a copy near my desk for reference (noting I work for a quality management company).

Quality Assurance (QA): an approach that leverages software testing and other methods to ensure that products function and perform well. At times, QA is used as shorthand for manual testing. I’d argue that it’s broader. To go deeper into quality assurance, consider ISO, the source of much deep thinking on QA. For a shortcut on the history of QA and ISO, this TechTarget article is helpful. 

Quality Engineering (QE): a discipline that includes quality assurance, DevOps, CI/CD, shift left, issue reduction, and other techniques that improve software release processes and outcomes. Like QA is sometimes used as a shortcut for manual testing, some in the industry use QE interchangeably with test automation. While I’ve personally done this in the past, I now believe that this is a limited perspective. I hold that QE as a concept is much bigger. It breaks down traditional build --> test --> deploy silos and revisits how software is fundamentally produced. Compelling perspectives come from Abstracta, the CTO Club, and others. Props to Jeff Wilkinson and Accenture for their early work on QE as an element of digital transformation. If you’d like to go deeper, Antoine CRASKE 's blog post is excellent, inclusive of his clarity on “build better, build faster” and his focus on sustainable speed in service of the business.      

As you may have gathered, I'm attesting here that quality engineering is the most important concept in the inverted pyramid. It’s where having a sound strategy can make a world of difference. So, let’s next turn to the elements that make up a quality engineering strategy.

Elements of a Quality Engineering Strategy

Below, I offer that a QE strategy can consist of ~15 elements. Using the inverted pyramid above, my mental model is that a QE strategy is a superset of elements also included in software testing and quality assurance strategies. 

Let’s explore each of these further, beginning with the bottom row and working our way up.

 Software Testing Strategy: ~5 elements. The goal of a clear software testing strategy is to define when, how, and where testing will occur. Important elements include cadence, gates, coverage (hardware, location, language, payments, etc.), ownership, technologies, and techniques (smoke, exploratory, scripted, regression, localization, etc.). From your perspective as a CEO, chances are that your engineering team can pretty clearly articulate their software testing strategy. Whether or not it’s optimal is something you’ll seek to explore through a set of questions, covered below.

Quality Assurance Strategy: ~5 elements. A broader quality (assurance) strategy wraps the ideas above and includes other important elements: statement, objectives, metrics, processes, and improvements. Summer Weisberg , CCO of Testlio , offers a compelling and informative perspective here on quality assurance strategies. Most companies today are relatively clear on their quality assurance strategy, although few are as crisp on quality assurance as they are on software testing.   

Quality Engineering Strategy: ~5 elements. Building on the ~10 elements above, a quality engineering strategy adds and covers the items below. From my perspective, few companies have mastered the integration of simplicity and sophistication needed to unlock a powerful quality engineering strategy.

• Purpose: an emotionally connecting and inspiring single sentence that answers the core question of why. An example for a mobility provider might be something like, “We passionately pursue quality engineering to ensure that our software enables riders to joyfully visit the places that matter to them.” 
• Differences: going back to Porter’s definition of strategy, it’s critical that your Quality Engineering Strategy helps you win against your competitors. Your engineering leaders should be able to explain what’s different about your approach. Candidates for differentiation include: 

+ Speed: building, improving, testing, and releasing faster; 

+ Tolerance: using intentional testing gates to provide a superior product experience;

+ Cost: gaining better results while spending less than your core competitors.

• Systems: per our definition above, quality engineering is a large concept. Your strategy should address core software tools and human-in-the-loop, AI-powered processes that enable shift left (and shift right), intelligent pipelines (CI/CD, feature flags, etc.), and more. 
• Economics: in my last post, I offered that quality engineering spend relative to core software engineering should often be ~10% to 20%. Your team’s strategy should be grounded in thoughtful economics and financial realities. Of course, the higher spend percentage, the higher your expectations as a CEO should be (on speed, coverage, and confidence). 
• Resources: in today’s world, your strategy can leverage a myriad of resource combinations. In-house. Outsourced. Crowdsourced. Automated. AI-powered. In-market. For many companies, a combination of resource types, intelligently leveraging in-house and partner-provided people along with a multitude of modern, integrated software systems is a winning combination. But sound resource strategies can vary. For example, I know CTOs who only use scripted test automation. Others who are focused on AI-powered no-code solutions. Some who use only manual testing. Many who are 100% partner-focused. Others who do everything in-house. You get my point. Resourcing strategies can vary but what’s important is that they are soundly thought out and reconsidered regularly.  
• Indicators: software testing data includes things like test coverage, turnaround time, issue validity, and addressed issue rates. Quality assurance metrics broaden to consider escaped issues, resolution times, product ratings, customer satisfaction, and product-driven churn. Quality engineering indicators must consider the entire journey of software product development. Important QE indicators may include:

+ Issues/Time: looking at bug trend lines (e.g. issues over a one year period) along with relative factors (issues per run, issues per test, issues per hour, etc.) gives teams a sense of whether or not improvement efforts are yielding results. 

+ Engineering Productivity: there is an ongoing debate about the pros and cons of measuring productivity for individual engineers, development teams, and software organizations. McKinsey offers this perspective. However you choose, or don’t choose, to measure activity or productivity, you’ll want to think about impact.    

+ Engineering Impact: core engineering KPIs like CCRs (closed to committed rates) and cost/head help you evaluate your engineering investment. QA indicators like those above (e.g. product ratings) help you determine how your product experience impacts revenue retention (e.g. GRR) and growth (e.g. bookings). QE indicators help you understand whether or not your engineering team is flourishing. Time to hire, % regretted attrition, and % reported flow state (per former Stripe CTO David Singleton’s work) are concepts worth considering.  

Note 1: I offer that indicators are what you measure. Benchmarks help you understand how your indicators compare to others. Trendlines let you see things over time. Targets give your teams direction and energy. All are important pieces of a holistic approach to data-driven quality engineering work.

Note 2: Janna Loeffler offers a slightly different framework for a quality engineering strategy that I find strong as well. There are others out there too. I find the exact framework isn’t what’s most important. Rather, adopting and applying a thorough and thoughtful framework is what matters, perhaps using a MECE approach in the process.

A Cautionary Word About Tactics

Early in my software industry career (~1998), I was trained in the Objectives-Strategies-Tactics (OST) approach to business planning and goal setting (thank you again Mike Maples, Jr and Scott Harmon and the rest of the team at Motive). In this model, objectives provide tangible “whys”. Strategies offer “whats”. Tactics describe “hows”.

I reflect on this model regularly, especially as a quality engineering strategy can quickly bloat to include the tactical elements of specific projects, initiatives, and deliverables. Your engineering leaders will likely help their more junior team members separate strategy from tactics. But if your team struggles with creating and managing a QE strategy, you may need to help with guidelines that align with other aspects of your company’s strategic planning approach.

Developing a QE Strategy

While it’s valuable for you, as the CEO, to have a perspective on the elements and concepts above, your software engineering leadership team is responsible for actually evaluating, articulating, and activating a bespoke quality engineering strategy for your company. If you have a CTO, this should sit squarely on her shoulders. 

If you don’t use the CTO title, your highest software development leader (e.g. VP, Engineering) should drive your strategy in close partnership with their DevOps, engineering management, and quality leads.

A comprehensive quality engineering strategy is a significant exercise that can lend itself to 30+ page documents covering the 20+ elements above. For many organizations, the quality engineering strategy should be a core element of an annual planning exercise. From my perspective and experience, quality engineering strategies should be reconsidered at least every 2-3 years, given the pace of change in software, AI, skill sets, methodologies, and processes.

Partners, including consulting and managed services firms, can help your team consider strategic possibilities, document elements of your strategy, provide benchmarks, help with targeting, and challenge your team’s natural biases. 

You as the CEO don’t need to go into all of the details of your team’s strategy. But I do recommend that you hold at least a one hour meeting on an annual basis, ideally including your CPO and CFO. Here, your CTO (or equivalent) and their team will present core elements of their quality engineering strategy. Your job will be to ask questions, per below. 

Generously Questioning Your Team’s Strategy 

As a CEO, your work is often to ask thoughtful, probing, generous questions. HBR recently unpacked five question types that CEOs can use: investigative (what’s known?), speculative (what if?), productive (now what?), interpretative (so, what...?), and subjective (what’s unsaid?).

In my experience, you can use generous questions like these to help your engineering leaders spot holes, opportunities, better options, and more. You don’t have to find the answers for them. But you can listen closely and offer guidance on where to keep working if you sense that the team might not have fully explored important avenues. 

Examples of questions that you might ask your engineering leadership team about their quality engineering strategy include:

Investigative Questions (What’s Known?)

• What new quality engineering options have you explored?
• What role will AI play in QE going forward for us? 
• Do you know how our competitors are pursuing quality engineering?
• Why do you believe this QE strategy is the best approach?

Speculative Questions (What If?)

• Could we think about and tackle quality entirely differently?
• What possible QE approaches have we not even considered? 
• Might we dramatically transform, recombine, and/or re-imagine how we approach quality?

Productive Questions (Now What?)

• Are we ready to move forward with our QE strategy?
• Do we have the people, partners, and/or tools that we need to be successful?
• What’s our key next step to unlock this strategy?

Interpretative Questions (So, What...?)

• What have we learned through our QE strategy exercise?
• Are we clear about what we’re trying to achieve with this strategy?

Subjective Questions (What’s Unsaid?)

• How do you and your team really feel about this quality engineering strategy?
• How much debate went into creating this strategy?
• Are key people in our company genuinely aligned with our approach to quality engineering? 

Additionally, I would encourage you to gently ask your team to articulate their quality engineering strategy in one sentence. The way you, as the CEO, ask this question is important. Ideally, it will come from your “sage” place and help unlock your team, avoiding the “saboteurs” that we as leaders grapple with (per perspectives on Positive Intelligence from Shirzad Chamine ). 

I encourage you to be patient with your team while you simultaneously set a high bar for strategic clarity. What you might hear in a one sentence quality engineering strategy summary could sound like: 

• Leverage AI-powered automation to create a new chapter of engineering efficiency 
• Emphasize human-in-the-loop processes to balance spend with speed
• Activate a multi-gate QE approach to ensure near zero production escaped issues
• Use partners to help us go faster, increase confidence, and reduce costs
• Integrate quality engineering with our global software operations
• Provide engineers with new resources to help them build quality into everything 
• Rebalance our use of humans and machines, ensuring a new practicality to manual and automated approaches 
• Merge today’s DevOps and Quality Engineering functions to expedite our pipelines

Your team’s exact statement will certainly differ. But I hold that if they are unable to summarize their strategy succinctly, you’ll want to give them help (such as a consultant) to ensure that they land on something tangible, clear, memorable, and energizing.

In Closing: Additional Perspectives on Quality Engineering Strategies

If you are a CEO and are still reading this piece: thank you. Your software development teams, along with their partners and vendors, are likely appreciative of how deep you’ve gone into this subject already, especially given everything else on your plate.

If you are a software development leader, and/or a quality engineering manager, I hope that this article has given you new ideas, resources, and perspectives on how you might drive quality engineering with your company’s executive leadership team—including your CEO. And while CEOs can sometimes seem intimidating, and often ask probing questions, at the end of the day they are dependent upon you and your expertise. They want to see you shine.

So, I encourage you to think big, be bold, debate as many possible directions as possible, avoid attempts at consensus, and ultimately commit to a strategy with your full self. By bringing everything that you have into your strategy (all your creativity, expertise, excitement, passion, joy, and grit), you give your strategy the best chance of success. Might you even think of your QE strategy as a component of the infinite game that you play with your work? (Shout out to Anthony Lee for encouraging me to think in infinite game terms as a CEO). 

Finally, if you are deeply versed in quality engineering strategy, I’d love to hear from you. Please comment below and/or reach out to me. I consulted a number of people and sources for this piece while taking a leap on several concepts, definitions, and structures of my own. If you think that I’m off in areas, and/or you believe something different, dive in and let me, and the readers here, know. 

Going forward, I’m already thinking about post four. There, I may tackle quality engineering coverage (exploring how your company can use locations, languages, devices, and/or payment systems as they build, test, and release software). Or I may unpack the psychology of confidence and how a quality engineering approach can help software teams sleep well at night—or not. 

Let me know if you have a preference on these or other topics. Until next time, I wish you and your company a quality engineering journey filled with speed, innovation, confidence, impact, fun, and success.