The Great Agile Chasm Between Business and Tech

During the 1990s silo busting was the rage. In 1994 there was even a large consulting company that sprang up called Silo Smashers.

So one would expect that the Agile movement, which was all about creating business value rapidly, would be even more potent at breaking silos to remove the chasm between business stakeholders and product engineers.

But that’s not what happened, paradoxically. In fact, in many cases the Agile movement actually made the silos worse.

To understand why, one has to look at the trends that emerged from the Agile movement. The first trend was the rise of so-called “Agile frameworks”. The eXtreme Programming (XP) framework was one of the first. In fact, many credit it with launching the Agile movement, at least in the US.

But then the Scrum framework rapidly eclipsed XP because the creators of Scrum launched a quick-and-easy 2-day “certification” program, claiming speciously that Scrum “was Agile”, and a tsunami of people rushed to get certified as “Scrum Masters” in order to easily and quickly change the a lucrative “Agile” career, since Agile was the hot new thing: after only a few years, there were thousands of newly minted Agile “experts” – their certification said so – and they evangelized Scrum everywhere they went, and so Scrum quickly took over and became synonymous with “Agile”, even though Scrum and “Agile” are two very different things.

So here’s the problem: Scrum defines a role for business stakeholders: the Product Owner. The idea is that each product owner has a product development team. But in most companies, products need many teams. So what happens is that each “Product Owner” is really just a business analyst called “Product Owner”, and the real Product Owner is someone else – some manager from IT.

That’s because IT usually owns the engineering resources. So the business stakeholders assign an IT manager to “build the product”. Thus, we have the chasm: there is a set of business stakeholders, and there is the poor IT manager tasked with building what is thrown at them. That IT manager rarely has any participation in conceiving the product, or any role in the marketing discussions about it.

Hello chasm.

It’s not that way everywhere, but what I described is an extremely common picture, and these “Agile frameworks” like Scrum kind of cemented it that way not so much because of what they say, but more because of what they don’t say: they don’t specify what happens when you have many teams. And they don’t address the whole business problem of how you organize marketing and product management and development.

So businesses have to figure that out, but they hear that they need “Agile teams” and those are really just dev teams told what to do by business analysts called “Product Owners” and managed day-to-day by people called “Scrum Masters”, and so it’s natural to lump all those tech teams under an IT manager.

Agile frameworks like Scrum kind of cemented it that way not so much because of what they say, but more because of what they don’t say.

Now to be fair there is something called “Scrum At Scale”, but not many companies use it, and it doesn’t really solve the problem, because it still is all about cranking out work. It says little about product design or marketing or business. It basically presumes that “the business figures that out” and then tosses it to IT to get it done – but at scale.

Hello again chasm.

So all that silo busting that happened during the 1990s? Agile brought the silos back. It was definitely an unintended consequence, but it happened.

There also is a kind of attitude within the Agile community that contributes to this. The community has from the start been very anti-manager. So much so that the book Accelerate, by DevOps giants Nicole Forsgren, Jez Humble and Gene Kim, says,

“within the DevOps community we have sometimes been guilty of maligning leadership…And yet, one of the most common questions we hear is, “How do we get leaders on board, so we can make the necessary changes?”

They refer to the DevOps community, but Jez Humble has long considered himself to be an “Agilist” and therefore views the DevOps community as an extension of the Agile community. (As evidence for this, his seminal DevOps book Continuous Delivery uses the word “Agile” 48 times.)

And Mark Schwartz, an Agile luminary in government circles and former Immigration Service CIO wrote in his book A Seat At the Table,

“Agile approaches seem to remove IT leaders from the value-creation process.”

So it’s not our imagination. And it’s kind of hinted at in the Manifesto itself when it says,

“Give [programmers] the environment and support they need, and trust them to get the job done.”

In other words, Leave Us Alone.

And it also says,

“The best architectures, requirements, and designs emerge from self-organizing teams.”

Again, Leave Us Alone.

And we’d like to point out that the above claim is not even true: self-organizing teams that operate well are anomalies. That’s one of the things explained (with evidence) in the book Accelerate: the most effective teams have leaders – “transformational” leaders. From the book:

“High-performing teams reported having leaders with the strongest behaviors across all dimensions [of transformational leadership]: vision, inspirational communication, intellectual stimulation, supportive leadership, and personal recognition.”

Indeed the whole anti-manager aspect of Agile culture has contributed to corporate America only grudgingly accepting Agile. And now that the Agile frameworks have not produced the agility that they claimed, and a new shiny object has arrived – AI – executives are only too quick to leave Agile behind – they didn’t like it anyway: it didn’t use their vocabulary and it derided managers – the very people who the Agile community sought to convince.

Self-organizing teams that operate well are anomalies.

That’s why Agile needs a hard pivot. Agility is too important to forget about. In fact, true agility is the most critical strategic capability of all today. The most agile companies will be the ones that pivot quickly as AI continues to evolve, and as the pace of change in all things speeds up. It is ability to quickly pivot that matters most.

True agility is the most critical strategic capability of all today.

But we need real agility – not these frameworks. Real agility comes from the way that people lead. It is about how decisions are made. It is about how managers stay informed, and what questions they ask. It’s about behavior: the Agile “frameworks” have nothing to do with real agility. They were a distraction – a costly one.

It comes down to being good leaders: leaders for today, who welcome technology instead of saying “I don’t need to know about that”. Leaders who accept change as natural. Leaders who are inclined to take cautious risks and sometimes some not-so-cautious ones. Leaders who can see the long trend instead of just the current tack.

Those are the leaders we need.