The Future of Agile Isn’t Shit

This is the third of three articles in my “agile scatological” series.  I just wanted to be clear about the name, just in case I’m being too subtle with the overall theme. The first article, The Agile Community Shat the Bed, worked through what I felt were the five primary causes why the agile gold rush has finally ended. The second article, How Agilists Can Move Forward After Shatting the Bed, described four strategies that we could choose to follow as individuals to work our way out of the mess that the overall community has created. In this article I describe my vision for how I see the agile movement evolving over the next few years and share with you what I intend to do to help things along.

Based on what I’ve seen over the past year or so, and conversations that I’ve had with a wide range of people, I believe the following will happen:

1. We will move beyond manifestos.
2. We will focus on software development.
3. We will go beyond software development.
4. We must put agile techniques into context.
5. I will evolve my own work.
6. We will not start a new gold rush.

We Will Move Beyond Manifestos

The original Manifesto for Agile Software Development addressed the challenges faced by software developers in the late 1990s.  The ideas that it captures were adopted by the development community long ago, to the point where new developers struggle to understand stories about traditional strategies. It’s clear that the manifesto is no longer the radical artifact that it once was. Should we not move on?

Over the years various people have recognized the need for an updated manifesto.  Unfortunately, they all ran into problems:

1. The original manifesto won’t be updated.  The manifesto authors agreed that they would not update the manifesto unless they all agreed on any changes. My understanding is that several are adamant about not changing it, and whether you like it or not that is a decision that we need to respect.
2. Getting smart people together physically didn’t work out. For the tenth anniversary of the original meeting at Snowbird, Alistair Cockburn organized a new meeting to determine if the manifesto should be updated. Although it was a great workshop, with a lot of great voices involved, it was clear that there was divergence rather than convergence regarding how to move forward.  Not only did nothing came of it, but you also likely didn’t even know it happened.
3. Virtual collaboration didn’t work out. At the Agile20Reflect event in 2021, a month-long international virtual conference of over 200 sessions, many speakers discussed their visions for evolving the manifesto and a few efforts to work together were started. Nothing came of any of that either.
4. Doing your own thing doesn’t seem to work either.  There is nothing stopping people from writing their own versions of a new agile manifesto, and many such efforts occurred over the years. Myself and Mark Lines tried it with the Disciplined Agile® (DA) manifesto originally written in the early 2010s, with Al Shalloway joining in to evolve it into the DA Mindset in late 2019. It’s a great read that provides a foundation for enterprise agility, but it wasn’t widely adopted.

The original manifesto got the job done. It provided something to rally around, it set a foundation for the agile movement, and it has proven to be a pivotal artifact for software engineering.  But it was a one-shot deal. The manifesto worked at the time because there was a clear need for a better vision that the mainstream leaders weren’t providing. The manifesto reflected better ways of working and thinking that had emerged in practice that didn’t reflect the theory of the day. But this isn’t our situation today. There doesn’t seem to be any need for new manifestos, and even though some people are still trying their efforts fall flat in practice.  But have at it, maybe you’ll succeed where everyone else didn’t.

To summarize, The Manifesto for Agile Software Development is one of the most important, and concise, works in software engineering history. It is time to declare success and move on.

We Will Focus on Software Development

In the absence of a new manifesto, what are we to do?  How about focusing on software development practices?  Or if you don’t like that word, techniques, strategies, and ways of working (WoW) are also fine terms. The original focus of the agile movement was on how to be effective at small-team software development, most software development initiatives are ten people or less, and it makes sense to get back to that focus.

To do so, I believe we will see a renaissance of work in the following areas:

1. Requirements elicitation. There is far more to agile requirements elicitation than writing user stories and putting all the hard work onto the difficult-to-fill role of product owner.  I’m seeing the glimmerings of a revival in business analysis and, given the growing importance of AI within organizations, data analysis.
2. Software architecture. There is far more to agile architecture than the occasional architecture spike and hoping that the architecture will somehow emerge. At least that’s true if you care about building software-based solutions that stand the test of time and that fit into your organization’s overall strategy. A great resource for software architects is the International Association of Software Architects (IASA).
3. Clean code. There is room for improvement in our approach to writing code.  It’s not just about writing working software, it’s about producing well-crafted, usable software. A shoutout to Bob Martin who has done a great job with his Clean Code book (my understanding is that he’s working on a second edition) as well as his work in software craftsmanship in general.
4. Clean data. Just like we should develop clean code, we should also create clean data. Given the prevalence of data technical debt in most organizations it is clear that we have a lot of work to do in this space. The good news is that many data quality techniques exist, including agile ones.
5. Team leadership. The agile community has spent the past two decades bashing project managers and spinning naïve stories around Scrum Masters. Time to fuck right off with all of that. Teams need effective leaders, and different types of teams need different types of leaders. There is certainly room for significant improvement around what project managers are taught about leading a software development team (and yes, I’m looking at you, Project Management Institute). Similarly, it would behoove the agile community to develop a viable strategy for how to lead a software team that reflects all aspects of doing so, not just the fluffy bunny stuff that’s easy to certify “masters” in. Both groups could clearly learn from each other, and hopefully they choose to do so.

I foresee several challenges to making progress in these areas, albeit challenges that can be overcome with effort:

1. Traditional culture. Except for clean code there are still strong, traditional cultures around all these areas. This reflects the agile community never stepping up to engage those communities in a meaningful way, including investing the effort to understand their point of view and then doing outreach to them – it wasn’t enough to simply invite them to get involved with us, that was a pathetic cop out. My advice is for agile practitioners to join these communities, learn from them, and then do what you can to help them leverage agile strategies.
2. Agile culture. The existing agile culture that minimizes the importance of these topics is also problematic. This once again is due to a lack of investing the effort to understand them, a focus on fluff rather than stuff, as well as the marketing rhetoric of the agile industrial complex (AIC) that purposefully minimized their importance. My advice is to step back, choose to think outside of the mainstream agile box, and observe what it takes to be successful at agile software development rather than what the AIC wants to certify you in.
3. Lack of expertise. There is a general lack of expertise amongst agile practitioners because we downplayed these topics for the past two decades. Luckily many great resources on these topics exist that we can use as a foundation from which we can move forward.

To summarize, although the agile movement started with software development we didn’t do a thorough job of describing how to be agile in this space. Key topics, what I’ve referred to as “agile stuff” in the two previous articles in this series, were downplayed in favor of “agile fluff.” It is time to rectify that mistake.

We Will Go Beyond Software Development

Software development isn’t the only game in town. There has been a bit of a schism within the agile community over whether agile should be applied outside of software development. Agile came from the software world and certainly most of what has been written about agile focuses on software teams. However, being a reality on the ground kind of guy I can easily observe that agile has been, and continues to be, applied far beyond the software realm. And yes, “not software development” is different than software development.  This observation is yet another reason why the Manifesto for Agile Software Development is no longer a solid foundation.

Some of the areas where agile and lean strategies are being applied are:

1. Data management.  Data management encompasses the activities to create, manage, provide, and support data within an organization. The Agile Data site captures a wealth of material on this subject.
2. Enterprise architecture. Enterprise architecture develops, supports, and evolves a vision for how your organization is built. Agile enterprise architecture is a collaborative and evolutionary approach to doing so.
3. Finance. Finance – budgeting, financial estimation, financial tracking, and financial reporting – is an important aspect of all organizations. It is possible, and highly desirable, to do so in an agile/lean manner. The Beyond Budgeting community has been leading in this space for decades.
4. Governance. I’ve always said that you’re being governed, like it or not, and that you deserve to be governed well. Governance establishes chains of responsibility, authority and communication in support of the overall enterprise’s goals and strategy. It also establishes measurements, policies, standards and control mechanisms to enable people to carry out their roles and responsibilities effectively.  Agile team governance is the governance of agile or lean teams in a manner that reflects and supports agile and lean ways of working.
5. Legal. The way that lawyers perform their work within your organization can be enhanced with both agile strategies as well as artificial intelligence (AI).  There appears to be a fair bit of work going on in this space.
6. Marketing. I’ve always argued that marketing is inherently an agile activity, and a great resource is the Agile Sherpas blog about agile marketing.
7. People management. People management is my preferred term for “human resources (HR)" because I refuse to refer to people as resources. The Agile HR community site offers great material, and they share my philosophy that people aren’t resources, but choose to meet the existing HR community have way by using their terminology to be more welcoming.
8. Procurement. Having been on both the receiving and “inflicting” ends of procurement processes, I can safely say that there is significant room for improvement in this space. I’ve always been a big fan of the work being done by the Lean Agile Procurement (LAP) alliance.
9. Strategy. Strategy focuses on how to manage your organization’s resources, risk, and return to ensure your success and longevity.  The Business Agility Institute (BAI) is doing some good work around agile strategy.
10. Value/profit streams. Value streams focus on the entire round-trip process of sustainably providing value to your customers via products and services, and profit streams on how to sustainably bring software-products to customers profitably. Great work is being done by the value stream management (VSM), Amplio, TameFlow, and Profit Streams communities.
11. And many more. This list isn’t complete, my apologies. My goal was to make it clear that agile is being applied far beyond software, regardless of what some may claim, not to provide an exhaustive resource.

To summarize, it is possible and highly desirable to expand agile beyond the world of software development. Great work is occurring to do exactly that.

We Must Put Agile Techniques in Context

For many years now I have said that context counts, that your way of working (WoW) should reflect the situation that you face. This is an important observation because it makes it clear that there are no “best practices”, that any given practice works well in some situations and is a spectacularly bad idea in others. I’ve been looking for best practices for almost 40 years now, and although I have learned about thousands of potential practices I have yet to run into one that works well in all situations.

To properly understand a technique – be it a practice, a workflow, a pattern, or some other form of WoW – you want to know what that technique is, the advantages of applying that technique, the disadvantages of doing so, and the context in which it is known to be effective (and hopefully known not to be effective). The patterns community has been putting their work into context for decades, describing the initial context where a pattern is applicable and the resulting context after the pattern has been applied. We should do the same for all techniques, not just patterns. 

Putting techniques into context is important for several reasons:

1. We make better improvement decisions. Understanding context enables you to make better decisions about whether a technique is a potential fit for your situation. Different teams in different situations will adopt different techniques, it’s that simple.
2. We become more open minded. By seeing that agile techniques have drawbacks, we are motivated expand our horizon to include lean and traditional techniques. This gives us more options to choose from, increasing the chance that we define a WoW that is right for us. The “agile” teams that are successful in practice invariably follow a hybrid approach that isn’t purely agile.
3. We can have better discussions about WoW. By putting techniques into context we enable ourselves to have better conversations with our colleagues who may be familiar with a different set of techniques than we’re familiar with.
4. We’re less likely to fall for the rhetoric. Once we realize that all techniques are contextual in nature, we are far less likely to fall for the marketing BS of the people purveying new methods, frameworks, and tools. Yes, what they’re trying to sell you will invariably work well in some situations, but does that include yours?

I’ve been putting techniques into context now since the early days of agile. In fact, I came around to this line of thinking when I first read Kent Beck’s Extreme Programming (XP) Explained book. One of the many things that impressed me was how he was clear where he knew XP to work and where he suspected it wouldn’t. While at IBM in the late 00s I developed something I called the Agile Scaling Model (ASM) that described a collection of scaling context factors.  In parallel Philippe Kruchten developed his “octopus model” of 8 contextual factors that was very similar to the ASM. I have since evolved my approach through several iterations into what I call the WoW tailoring factors that describes seven factors you can typically adjust and five factors that you typically can’t. In the spirit of there being no best practices, even when it comes to assessing context, for the narrow domain of data quality (DQ) I focus on five factors that help practitioners to choose the right techniques for their situation.

To summarize, there are no best practices. By putting techniques into context, we become more open minded, we enable more effective conversations, we make better decisions about our WoW, and we’re less likely to fall for marketing BS.

I Will Evolve My Own Work

I have been evolving the material at Agile Modeling (AM) and Agile Data (AD) for over two decades now.  My effort on doing so has come and gone over the years, usually because I was focused on other methodology work such as Disciplined Agile or the now defunct Enterprise Unified Process (EUP). During the Autumn of 2022 I invested a fair bit of effort updating the material on my sites and modernizing the look and feel of them.

But that work slowed down while I focused on my education, I’m working on a Master degree in AI that I hope to finish in December 2024. As a result, I still have a list of things that I want to do, for the most part starting in 2025.  Not in any order, this includes:

• Put the techniques into context. A few of the techniques at the sites, particularly AD techniques, include contextual advice. Many do not because that information was captured in the DA toolkit, the purpose of which was to put hundreds of techniques (around 1700 when I was last involved) into context. This includes the dozens of modeling, documentation, and data techniques described at my two sites. It’s time that I include context information for all the techniques on my sites.
• Update examples. Many of my examples, although still valid, are dated. Plus, many of them are scans of diagrams drawn in my horrendous handwriting (as the old joke goes, I could have been a doctor). Time to do better.
• Cull some of the older material. I’ve pretty much done this already, although I expect I will still run into material that is no longer relevant. For example, I once had a detailed introduction to a new-fangled data standard called XML and a detailed set of Java coding conventions. While both were very popular during the ancient times when those technologies were new, they became little more than a maintenance burden for me so I dropped them.
• Flesh out the material. Many of the details behind AM and AD weren’t published on the sites, rather they appeared in books, blogs, articles, and courseware. It’s time to update and publish these ideas.

To summarize, I intend to do my part and lead by the example that I presented in this article. I’ve always followed a policy of making information as freely available as I can, and I intend to continue to do so.

We Will Not Start a New Gold Rush

A fundamental concept in defining the scope of an effort is that it’s always important to be clear about both what is in and what is out of scope. First, my advice is to not try to rename agile. In the second article in this series I indicated that I liked the word “adaptive” and I’ve also heard a good argument for “nimble”.  Although “agile” has clearly become a swear word for the moment we’re not going to fool anybody by renaming it. At best all we would accomplish is to do a lot of work rebranding agile to CoolNewName only to have people dismiss that effort with the observation “Yeah, CoolNewName is simply agile warmed over”.

Second, there simply isn’t an opportunity for a new gold rush on the horizon. The only thing that’s close is artificial intelligence (AI), and I’m certainly seeing a lot of former agile consulting firms shift to pitching themselves as AI consulting firms, but this opportunity is limited at best. The AI space is dominated by large tech vendors such as Microsoft, Google, and OpenAI and they will dominate any certification schemes moving forward. AI training is varied, complex, and requires significant effort.  Take it from me, I’ve just spent the last two years working on a Master degree in AI and I’ve just scratched the surface. The real AI leaders tend to have many years of experience and PHds, not a few days of fluffy bunny certification training. Building AI-based systems requires data expertise, something that tends to be in short supply in the agile community, so former agilists have a lot of catching up to do. In short, success in AI consulting requires significant stuff and very little fluff, making it a very different business model than agile coaching.

To summarize, the agile gold rush is dead and there isn’t an easy path back to another gold rush – it really is over folks.

What is the Timeline for This?

To be blunt, the next two years is going to suck for the agile movement, and deservedly so.

Things will start coming back for people who focus on valuable stuff. If we share this valuable stuff, and many agilists have a track record of doing exactly that, then the agile movement will get back on track. Time to do the hard work that we should have done two decades ago.