Why Agile? Why any process at all?

If you have worked as an Agile Coach, you must be familiar with resistance to Agile. It comes from various sources (mostly leaders at different levels) and their complaints could go something like this:

• "Agile is not relevant for our case. Our situation is different."
• "Our team is doing good already. Why do we need to change what is working for us?"
• "Agile has slowed things down. There are too many meetings, the time for actual work has reduced."
• "In the name of empowerment, team members are taking things easy (overestimation, complacency, etc.). And, as a team lead (PO/SM/Manager), I feel my hands are tied – I can’t push the team anymore."

Before we attempt to answer ‘Why Agile?’, let us try to understand a more basic question – Why do we need any process at all?

Imagine three close friends working on a new business idea – working hard, day and night, with full focus and commitment. They plan work and execute collaboratively, possibly with overlapping work responsibilities. Despite the uncertainty around their business case, things are moving well and forward. Do they need a well-defined process?

May be yes. May be not.

While they could use some process elements, they could very well do without a structured process. In such real-life situations, the flow of work is very dynamic and their decisions mostly ad hoc, on need basis, instead of being guided by a structured process.

Now imagine, somewhere down the line, their idea becomes successful, and they are now the leaders of a 30-people startup. Would they need a process now to guide the efforts of their big team?

What if, 2 years later, they are a 200-people profit-making organization? Or further down, they are a 2000-people organization; would they need a process then?

In my experience as a Coach, having worked with multiple startups and small to mid-size organizations, the need for a process is felt more strongly as the organization grows in size. And the expectations from the process are usually the same – ensure predictability of results/outcomes:

1. Good amount of work gets done every month/week/fortnight (predictable throughput).
2. Completed product/features work as expected, without failure (predictable quality).
3. The product releases happen on time (predictable turnaround time).
4. The market responds to our product releases in a favourable manner, just as we expected (predictable business outcomes).

While the expectations from a process are clear, we all have seen some great teams - of qualified and committed individuals - that can make results happen, no matter what the process. And, there are also some good managers who can make things happen, irrespective of the process, and despite their differing styles (through force, motivation, or charisma).

While great teams and great managers can make things happen, when you are a senior leader looking at hundreds or thousands of people, you want a more reliable solution that is independent of specific high performers.

You want a system that encourages, guides, and shapes people and their efforts to ensure positive results, time and again. And, that usually is the promise of a process.

In the twentieth century, we saw various processes evolve that aimed to create such a system that can ensure expected results, in both manufacturing and service industry. In the service industry, Waterfall was the most successful of them all, with RUP (Rational Unified Process) perhaps a distant second. Of course, this was before the Agile wave changed the landscape.

Before moving on, lets summarize the answer for ‘Why we need a process?’:

1. A process gives us proven practices that help us plan and execute our work in an optimal way (reduce effort and time).
2. It increases the chances of positive results by prevention or early identification of failure.
3. A good process decreases over-reliance on specific high performers.

Now, back to our main question - 'Why Agile?'

While all processes intend to provide us a proven recipe for success, all processes are not the same – they vary in two key aspects, their GOAL and their STYLE, or approach.

Agile evolved from the limitations of the waterfall process – both on Goal as well the Style.

The primary goal of Waterfall was 'successful project completion' – completion of all the work by a pre-determined date. And, in its attempt to achieve the scope/schedule targets, it enforced freezing the scope early in the process – something that does not work well with new product development in today’s market dynamics where the business requirements keep evolving.

Agile shifts the focus from fixed scope/schedule targets to ‘Business Value delivery’, and helps adapt to moving business targets through incremental value creation and faster feedback. In comparison, Waterfall was a defensive approach that focused more on preventing/minimizing failure on way to a distant end result.

In terms of Style, Waterfall focused on sequencing of practices, documentation, and compliance. And, a successful implementation of the process relied heavily on external agents.

On the contrary, Agile goes beyond just process and practices, and focuses on improving the team and organization culture also. It shifts the process control inwards, in the hands of team members by improving alignment, autonomy, collaboration, accountability and transparency – elements that promote self-organization among team members. It should be noted that team’s self-organization is essential to harness collective intelligence of the whole team, promote intrinsic motivation, improve individual skills and confidence, and reduce over-reliance on specific high performers. If done right, if shifts the performance level upwards for all team members, not just a select few.

While the above two points (focus on business value and self-managing teams) sound like common sense and somewhat cliché, most of the resistance to Agile comes from leaders’ poor understanding about them. Some would call it a legacy of waterfall – leaders chasing ‘scope/schedule targets’ more than ‘value creation’, and favouring ‘directive leadership’ style over ‘self-managing teams’.

Here are some common manifestations:

• Rohit (team lead) says “we don’t need Agile” as he has high-confidence in his own ability to ensure team’s success and wants the organization to stay dependent on him. He likes being in control and sees the idea of self-managing teams a threat to his own positional power.
• Deepa (Product Owner) maintains a transactional relationship with the team and avoids attending Scrum team meetings. She does not like team members asking too many questions on requirements as she already knows what is best for the customer.
• Amit (Program Manager) believes his job is to ensure ‘timely delivery’ and finds it natural to demand delivery commitment based on firm and frozen plans and drive the execution. Stepping back into an enabling leader who facilitates communication/coordination across teams during planning and execution sounds like a risky proposition that could make him lose his job.
• Shruti (Senior Manager or Director) knows she will be evaluated by her manager based on amount of work done by her teams. Over time, she has developed sharp focus on ‘productivity’ and sets velocity targets for the ten teams that roll up to her. She gets regular updates from her team leads and does not feel the need for direct interaction with teams. She is fine if her team leads want to try this new idea called ‘self-managing teams’ as long as the team performance does not slip down.

In general, an organization that sees Agile as a tool for timely delivery or higher developer productivity has little patience to appreciate the idea of self-manging teams. They are likely to achieve only temporary benefits from their Agile adoption (if at all). Their leaders would continue to see Agile as just another process and continue to resist the cultural aspects of Agile.

Some say the agile maturity of the organization is limited by the Agile understanding of its senior most leaders. What do you think?

Look forward to hearing your thoughts.

(The views expressed here are my own, based on my experience as an Agile Coach over the last ten years, and my interaction with other Agile coaches. They do not represent the views of my current employer (Siemens), nor are they related to my experience at any specific organization.)