Why are the success rates of big projects and transformations so low?

What could make your project fail?

We brought all stakeholders together to discuss a major software project.  The requirements were well documented, the work breakdown ensured granular planning, and the right resources were allocated.  The meeting concluded with a can-do speech by the project sponsor. We were off to a great start. Fast forward nine months: milestones were missed, and major overruns faced a project that couldn’t be delivered. Sounds familiar? I have been involved in hundreds of projects throughout my career, many of which never delivered what was envisioned.

0.5% of large projects achieve the trifecta of meeting their cost, time, and outcome targets

In his illuminating work, "How Big Things Get Done," Oxford University professor Prof. Bent Flyvbjerg delves into the intricacies of project success and failure. Drawing on the analysis of 16,000 diverse projects, ranging from kitchen renovations, IT projects to the construction of nuclear reactors, he makes a startling revelation: a mere 8.5% of these projects manage to stay within their initial estimates regarding cost and time, and a meager 0.5% of large projects achieve the trifecta of meeting their cost, time, and benefit targets—some projects instead of merely failing, crash and burn spectacularly. Familiar examples include Berlin's endlessly delayed airport and Boston's infamous "Big Dig," which ballooned fivefold over its original budget and California’s train to nowhere. The Dutch government spent EUR 1.3B more than planned on their large IT projects and two-thirds have major overruns. According to McKinsey, a staggering 70% of transformation initiatives fail.

Flyvbjerg aptly observes, "Purposes and goals are not carefully considered. Alternatives are not explored. Difficulties and risks are not investigated. Solutions are not found." This shallow analysis leads to hasty decisions, unrealistic forecasts, vague objectives, underestimating external factors, and a lack of contingency plans. There are many cases of politically motivated underestimation to push things through and get the project started. However, to lay the foundation for successful projects (including transformations and new ventures), one must meticulously define the goal and success criteria, deeply understand the context, and have a singular focus on the outcomes from inception to completion. Planning should be a deliberate, slow, and rigorously iterative process informed by both experience and experimentation.

Think slow. Act fast.

Are the "why" and the context crystal clear?

To execute a project successfully, posing the right questions is essential. Continuously ask "why?" and "what value are we creating and for whom?" to unearth the underlying motivations behind your decisions and actions. When you schedule a project, you can refer to similar projects to gauge duration, cost, and potential pitfalls and risks. These references provide a more reliable anchor assessment than bottom-up planning, which often needs to be revised to address both known unknowns and unknown unknowns. A third-party perspective on the project is invaluable for uncovering overlooked details and biases. Thoroughly evaluate potential downsides and have a strategy to mitigate risks. Nobel laureate Daniel Kahneman warns against the "planning fallacy," our tendency to overestimate benefits, underestimate costs, and recklessly take on risky endeavors. Flyvbjerg's advice is clear: "Think slow, act fast." You can maintain agility and rapid iteration within the confines of a well-thought-through architecture and plan. Most large-scale projects are better approached as a complex system: many dynamic forces can propel the project to success or failure. Understanding the full context, influences, and dynamics will be critical for success.

Experienced leaders resist the temptation to rush through a project and prioritize thoughtful design and decision-making.

Is the approach truly modular?

Whenever possible, rely on proven methodologies and technologies and use mature modules. Loosely coupled systems and modular designs are a way to deal with complexity and change. The successful completion in one year and 45 days of the Empire State Building in 1931, ahead of schedule and under budget, hinged on its modular construction. All issues were resolved after the first five floors were completed, and the construction process became increasingly efficient with each subsequent floor. This applies to both hardware and software products as well. Computers, cars, wind turbines, and solar panels have become increasingly modularized with the intelligence residing in the software and algorithms. Software functionality running in the Cloud can be easily consumed as building blocks through APIs (Application Programming Interfaces). Amazon has done an outstanding job in making computing and storage super easy to consume, by building highly modular services.

Be stubborn on the vision but agile on the execution

Are the deliverables time-bound?

The daunting odds against a successful program, like a digital transformation, are rooted in the complexity of multi-year initiatives encompassing various interdependent projects. Clarity of purpose and goals is imperative for projects and transformations. Well-managed stage gates that assess and adjust course and speed regularly and loosely coupled, iterative deliverables keep the program moving in the right direction. John P. Kotter, the guru of business transformation, underscores the need to instill a sense of urgency. He emphasizes the challenge of coaxing people out of their comfort zones, the impatience to rush into transformation without thorough planning and experimentation, and the trap of overconfidence in a favorable outcome. According to Kotter, a sufficient sense of urgency arises when approximately 75% of a company's management acknowledges business-as-usual unacceptability, and there is a genuine commitment to change. "What has got us here won't get us there." One of the companies I worked with recognized that the future lay in deep customer relationships and integrated solutions, transitioning toward an agile enterprise. However, they reverted to conventional methods rooted in traditional product pipelines when push came to shove. Adding features or tweaking supply chains couldn't fend off digital competition. Successful transformation necessitates changes across all dimensions of an organization’s operating model. As Donald Sull observed: “Active inertia is an organization’s tendency to follow established patterns of behavior—even in response to dramatic environmental shifts. Stuck in the modes of thinking and working that brought success in the past, leaders simply accelerate all their tried-and-true activities. In trying to dig themselves out of a hole, they just deepen it.”

Promote an open, diverse, and challenging environment.

Is the team up to the task?

Your delivery team's composition is very important. Seasoned, skilled professionals are indispensable to reaching your objectives. This is precisely why venture capital companies meticulously scrutinize a team's track record and capabilities before allocating capital. Experienced leaders promote an open, diverse, and challenging environment to avoid costly missteps. They resist the temptation to rush through a project and prioritize thoughtful design and decision-making. They know every project necessitates an initial phase of planning, architecting, modeling, and simulation. It's far more cost-effective to rectify mistakes during the architecture and design stage than during implementation. In software development, the cost of detecting and fixing bugs increases exponentially over time, with field fixes incurring the highest costs. The consequences of flawed design can be even more severe for hardware projects.

Effective communication is crucial, but engagement goes one level deeper. Stakeholders must actively participate in shaping and executing the program. 

How deep is the stakeholder engagement?

A compelling vision is crucial for stakeholders to grasp the direction and future. It inspires change. A dedicated coalition of believers must guide the major project or transformation, with all the leadership fully engaged. More than communication is needed: leaders must co-create, shape, and guide.  In a real-world example, a company I supported embarked on a year-long journey to formulate a compelling vision, during which all leadership team members became deeply committed. This commitment transcended their individual units as they collaborated on shaping the company's future. The broader organization, however, was only privy to some of the deliberations and iterations, leading to a sense that change was afoot. The plan was outlined to address this, and three major programs were identified for further elaboration with cross-organization teams moderated by a leadership team member. This process also revealed different categories of individuals: the players ("I am in the game"), tourists ("Let's see what is going to happen"), and prisoners ("This is never going to work"), reflecting varying levels of commitment to change.

Effective communication is crucial, but engagement goes beyond mere communication. Stakeholders must actively shape and execute the transformation. A stakeholder power and position map can help target engagement efforts. Focusing on the tourists —those individuals or groups on the fence—is critical. Rallying supporters create momentum while dealing with persistent resistors may require parting ways to eliminate negative energy.

Kotter advises tackling significant blockers, which may involve reevaluating organizational structures, ways of working, KPIs, incentives, and resistant managers. The organization must adapt across all dimensions, including bureaucracy reduction, formation of multidisciplinary teams, agile methodologies, and data-driven approaches. Experimentation with the new operating model on a business adjacent to the core can yield insights and lead to adjustments in strategy, value streams, organizational structure, processes, systems, performance, motivation, and culture.

System thinking helps to circumvent the common pitfalls of failure in large projects and organizational transformations.

What have we learned?

System thinking helps to circumvent the common pitfalls of failure in large projects and organizational transformations, as it highlights interconnections and the dynamic forces that drive outcomes. 

• Begin by creating clarity of purpose and fostering a culture of continuous learning and adaptability within the team.
• Seek external expertise and benchmark against successful peers to gain fresh insights and to find a proper reference.
• Emphasize thorough contextual understanding and risk assessment. Create scenarios to deal with unforeseen roadblocks.
• Encourage deep engagement and open communication with all stakeholders, where concerns and challenges can be explicitly discussed and addressed.
• Establish robust governance, clear roles and responsibilities, and actively manage (inter)dependencies.
• Invest in comprehensive recruitment, training, and development programs to match ambition with capabilities.
• Apply mature methods and technologies and a modular approach.
• Continuously monitor progress against milestones, seek continuous feedback, and be prepared to adjust the course when necessary.
• Finally, maintain an unwavering commitment to the original vision and goals while remaining flexible in the strategies used to achieve them. And...don’t compromise on quality.

By incorporating these principles into your approach, you can significantly increase the odds of success in even the most ambitious endeavors.