Getting Clear About Agile

These days, “agile” is a term heard frequently in the worlds of technology and business. The word has many positive connotations, such as nimbleness, grace, and spryness—characteristics we might like to be associated with us and our organizations. And most of us have heard success stories about companies that have achieved better outcomes after adopting agile practices, so there’s plenty of interest in replicating that success.

According to data compiled by Ron Lichty and Greg Geracie, the number of software companies that have adopted agile (iterative) development practices has steadily risen over the past decade, and now greatly exceeds the number of companies that use the traditional ‘waterfall’ (pre-planned sequential) approach.

A 2019 Gartner survey finds that 85% of organizations have adopted, or plan to adopt, a product-centric application delivery model, which aligns with agile practices and DevOps. Agile practices have also been adopted in many areas unrelated to software, including medical devices, custom construction, finance, mergers & acquisitions…even fighter jet development.

Unfortunately, a large proportion of organizations that attempt to implement agile transformations find that their efforts fail to deliver the expected results. That doesn’t mean that agile practices are bad or ineffective; the failures can usually be traced to misunderstandings about the practices and how to apply them.

In this article, I’ll strive to correct some common misconceptions, and leave you with a clearer understanding of agile practices, and of what is needed to gain their benefits.

Agile is a Mindset

All too often, people think that “agile” represents a collection of artifacts and activities labeled with jargon like ‘backlogs’, ‘stories’, ‘daily stand-ups’, and ‘pair programming’. While those artifacts and activities are indeed used by agile practitioners, merely adopting them does not make an organization agile. You don’t become a great hockey player just by donning skates and pads and imitating Wayne Gretzky; you need extensive practice and learning, as well as an underlying zeal for the sport that drives you to ever-higher levels of performance.

To achieve agility, you and your organization need to adopt an agile mindset, which is centered around embracing change and responding effectively to it. A good place to start is by studying The Agile Manifesto, a concise document that was developed by early leaders of the agile movement. It defines 4 values and 12 principles that spell out important attributes of the mindset.

An important aspect of the agile mindset is expressed in the first 3 words of The Agile Manifesto: “We Are Uncovering”.  The authors didn’t write “We Have Uncovered”, because they knew there will always be more to discover and new ways to improve, and that continuous learning and continuous improvement are essential for agility. Be open-minded, embrace change, and keep growing and improving. Agility is something you continually develop, not achieve; it’s a journey, not a destination.

Agile Does Not Mean Out of Control

People who are accustomed to detailed long-range planning are often uncomfortable with the empirical, figure-it-out-as-you-go approach of agile development. They may imagine it to be a free-wheeling, chaotic situation, where front-line developers are steering the company in whatever direction their whims might dictate at any moment, instead of following a carefully planned strategy and tactical execution plan defined by corporate management or a Project Management Office.

There’s also a common perception that agile development is undisciplined, but developing in an agile environment actually takes more discipline than developing in a traditional waterfall environment. Teams must analyze problems, estimate effort, build collaboratively, and measure their output and productivity at a fast cadence, which cannot be done effectively without discipline. Practices like test-driven development (TDD) and behavior-driven development (BDD) require up-front work before code is written, and developers must discipline themselves to do that work. Though that up-front work may not be as exciting as writing the code, it greatly improves quality.

If you worry that adopting agile practices could mean losing control of your organization’s work, it’s time for a reality check: you probably already have less control than you think you do. You may be used to making detailed long-range plans, but how often do those end up dramatically changed or scrapped halfway to a target date? You may be in the habit of assigning projects to people, together with deliverable requirements, deadlines, and fixed budgets, but how often have those projects fulfilled expectations?

It can certainly be challenging to become comfortable with concepts like bottom-up planning, empowerment of front-line development teams, and evolving specifications. What’s important to understand is that agile development doesn’t eschew planning, documentation, and testing; it just does these important activities in small increments on a continuous basis as the development work progresses, rather than as large front-end and back-end tasks. Agile development still needs to align with broad strategies and roadmaps; it just manages the execution in a nimbler way.

 Doing the work in smaller increments has two important benefits:

• More product features get delivered to customers sooner, and can be monetized sooner
• Course corrections can be made on a continuous basis, in response to customer feedback, changes in markets, changes in technologies, and the like

The result is increased revenue, reduced waste, and reduced risk. What’s not to like about those outcomes?

Agile Isn’t Right for Every Situation

It’s important to understand that agile practices aren’t always the best way to work. They can actually reduce productivity in situations where the parameters of a problem are well-known, such as building an ordinary highway overpass, or establishing a new franchise location for a large restaurant chain. Such undertakings may be complicated —involving many tasks and details— but those tasks and details can be identified, planned, and documented before the actual work begins, so executing is a relatively straightforward process. You can plan the work, and then work the plan. With complicated (but well understood) problems, agile practices—which involve short development cycles and frequent inspection and adaptation—can add unnecessary overhead.

Where agile practices work well is in solving complex problems—those in which many factors and issues cannot be known before the work begins, and which require a process of discovery. An obvious example of this is technical research; there’s no way to know in advance what might work, so problems are approached by designing, conducting, and evaluating experiments, and using discoveries from each experiment to plan the next experiment until solutions to the core problems are found. Other examples include ventures into new territory (such as sending manned spacecraft to Mars), or launching a new type of consumer product or service for which there is no real precedent (consider Google Glass, Impossible Burger and Beyond Meat, FitBit, smart speakers, social media platforms).

These classifications of problems come from the Cynefin (pronounced kǝ-NEF’-in) framework created by Dave Snowden:

In the Cynefin framework, problems that fall into the Obvious (also called Simple or Clear) and Complicated domains are straightforward, and can be addressed via well-defined processes. Problems that fall into the Complex and Chaotic domains require iterative and reactive approaches, respectively; you simply cannot effectively plan in advance what your solutions will be.

Software development tends to fall into the complex domain, for several reasons:

• any particular software product is really ‘made’ only one time (there’s mass replication of the developed product, but this is different from constructing multiple units of an automobile model, bottles of shampoo, or other manufactured product)
• the best way to solve a particular problem often is not evident until several approaches have been investigated
• technologies, tools, customer needs, and market landscapes change frequently

Thus, an iterative approach that involves continuous discovery and adaptation works much better for software development than attempting to plan the work out in advance.

Agile Is a Team Sport, not an Individual Event

Agile practices work best when they are embraced by entire organizations, not just small groups. It’s difficult for an agile team to self-organize, manage its backlog, deliver valuable product steadily, and improve itself continuously when high-ranking people in a traditional corporate hierarchy impose their will and seek to dictate what will be built, when it will be built, and who will build it. To succeed, agile transformations require sustained support at the executive level, so that self-directed teams can work without interference and deliver great results. They also require executives to change their own behavior, focusing less on measuring inputs like hours spent and vanity metrics such as lines of code or team velocity, and instead focusing on value delivered.

Of course, transforming a large organization all at once is very difficult and risky, so most agile transformations begin with an individual group or department and expand from there. The early stages are especially challenging, because forces inherent in the existing organization will resist many of the changes of the agile initiative, and will actively try to reverse them. Leaders must understand these dynamics, and create a climate in which the agile pioneers can try out new practices, make a few mistakes without being penalized, learn what works for their own situation, and produce improved outcomes.

Understanding WHY is Important

Most organizations adopting agile practices will modify the practices after a while. That’s appropriate; the whole point of being agile is to optimize the system by experimenting and going with what works best for the specific circumstances. But all too often, organizations modify or abandon broadly endorsed good practices without understanding their purpose, and end up with bad outcomes as a result.

One example is daily stand-up meetings. Teams may get lax about member attendance, allow discussions to devolve into problem-solving sessions, or cut frequency to just a few times a week. Such teams do not clearly understand that the purpose of the meeting is to efficiently ensure that all team members start each workday knowing the current status of the team’s work, and that any issues are brought to light promptly so that risks to the delivery schedule can be minimized. When the purpose of the stand-up meetings is clear, teams get value from them, and outcomes tend to be better.

Another example is retrospectives. Great teams know that they are an essential tool for inspecting and improving the team.  Not-so-great teams may turn them into gripe sessions, or into blamestorming sessions, or fail to bring up important issues in a constructive way. Gaining nothing of value, the team may stop attending or even scheduling retrospectives, and devolve into dysfunction over time.

Yet another example is story points, a tool for estimating effort required to complete different development tasks. The popular work management software JIRA now allows story point values to be assigned to tasks with decimal values, rather than integers. Why? No doubt some customers asked for that capability, and Atlassian wants to keep its customers happy, even when they’re doing something that makes no sense. Agilists who understand story points know that they are not accurate estimates of hours or even scope; their purpose is to roughly estimate the relative effort of tasks, so that developers can decide which tasks to take on in a development cycle, and get a sense of how much work they’re getting done over time (velocity). Since story sizes can’t be precisely estimated, there’s nothing gained by pretending that adding decimal precision will improve outcomes.

Agile is a Journey, Not a Destination

All too often, I hear people say things like “Our company is agile” or “We’ve implemented agile”, as if it were a simple matter of switching to a new process. Worse, I often hear “We’re doing an agile hybrid”, which tells me that an organization wants the benefits agile practices can offer, but isn’t comfortable with giving up its old structures, processes, and behaviors. The outcome is predictable: outcomes fall below expectations, because superficial changes were made without understanding the underlying principles, and the mindset changes that must be made for an agile transformation to be successful.

Agile is more than a set of processes or techniques; it’s a mindset that’s centered around creating high value in realms of high uncertainty by:

• collaborating closely with customers
• collecting and responding to feedback frequently
• adapting to changes quickly, and
• improving practices continuously.

A truly agile organization understands that products and organizations are never finished; there’s always room for improvement that creates more value. Maintaining humility, staying open to new ideas, and embracing change are key behaviors that help practitioners develop and sustain an agile mindset, and continue the journey.

Acknowledgements

Most of the ideas and concepts I have shared in this article were imparted to me by agile coaches who share their knowledge generously. I have learned from many agilists over the years, but in particular I’d like to acknowledge Fred Fowler, who gave me my first training on Scrum and who hosts ongoing Scrum discussion groups, and Larry Apke, whose Job Hackers organization provides free training on agile practices to professionals in career transition. I was also strongly influenced by Barry O’Reilly, whose Unlearn book and podcast provide essential lessons for leaders who want their organizations to make successful agile transformations.

Jim Schibler leads product management teams that deliver software experiences customers love, and he coaches professionals on job search and career management. He writes on a broad range of topics; see more of his articles at his website.

Copyright © 2020 Jim Schibler — All rights reserved

Image credits: Saab S35 fighter courtesy Av Herrandersvensson; Hockey Kid courtesy Joshua Choate @ Pixabay.com; Scrum courtesy Pierre Selim @ Flickr.com; Stand-up Meeting courtesy Ben Terret @ Flickr.com; Lake View courtesy Matt Hill @ pexels.com.