Transformed – Personal Reflection Part #4: Product Delivery

In part #1, I shared my personal reflections on the principles of product teams. In part #2, I discussed my thoughts on the principles of product strategy. In part #3, I offered my insights on the principles of product discovery. In part #4, let me share some more stories on the principles of product delivery.

Product Model Principles

I listed 15 principles where I have strong examples to demonstrate real-world experience. This is not an exhausting list of principles, if you wish to know more, please check out Marty Cagan's book.  I will post 5 articles to cover all 15 principles and my personal reflections.

Product teams

#1: Principle: Empowered with Problems to Solve

#2: Principle: Outcomes over Output

#3: Principle: Sense of Ownership

#4: Principle: Collaboration

Product strategy

#5: Principle: Focus

#6: Principle: Transparency

#7: Principle: Placing Bets

Product Discovery

#8: Principle: Assess Product Risks

#9: Principle: Test Ideas Responsibly

Product Delivery

#10: Principle: Small, Frequent, Uncoupled Releases

#11: Principle: Deployment Infrastructure

Product Culture

#12: Principle: Principles over Process

#13: Principle: Trust over Control

#14: Principle: Innovation over Predictability

#15: Principle: The company cares about what the leader cares about

#10: Product Delivery Principle: Small, Frequent, Uncoupled Releases

According to Cagan, if you are serious about taking care of your customers, you will need to invest in your ability to deliver very frequent, very small releases. The smaller the release increment, the faster you can ensure the quality of the new capability, and the faster you can be confident you're not introducing regressions. And with these small, frequent releases, if a problem is introduced, it is much easier to identify the cause (since you only changed a small number of things).

In reality, most organizations struggle with truly implementing Agile practices. They may claim to follow Agile, Scrum, or Scaled Agile methodologies, meaning they run two or three-week sprints, but they often fail to release anything to production until one or two quarters later. This delay undermines the essence of Agile, which is about delivering incremental value to customers quickly.

An interesting observation about production releases is that some organizations claim to deploy to production every month. However, these "production" environments are often internal, meaning no actual internal or external customers or users interact with the new functions for extended periods, sometimes over two years. This practice highlights a significant disconnect between Agile principles and actual execution, ultimately delaying customer feedback and value delivery.

Small, frequent and uncoupled releases require a major different way of working compared to the old model.  This is what most DevOps transformation is about.  Let me share some common themes, whether stemming from a lack of knowledge on how it can work, a lack of interest in learning what good looks like, or simply because change is too difficult.

Mix production deployment and production release

I often hear concerns like: “We can't deploy changes to production too frequently because our customers can't handle frequent updates, and our training department can't keep up.” Another common issue comes from customers who say: “Our production codebase must be frozen during tax season from December to May. Once tax season ends, we face significant challenges catching up on five months' worth of changes.”

Many people treat deployment and release as the same thing!  A production deployment is not a release! Lack of this distinction is one of the major reasons why many organizations wait for months to deploy and release anything.

RELEASES

We can define a release as a change that alters the observable behavior of the system as experienced by end users. This is distinct from most deployments to production, which often don't "release" anything new to users; for example, changes that are part of refactoring or re-architecting the system.

A release also doesn’t necessarily require a production deployment. For instance, new behavior can be hidden under a feature flag that can be enabled at runtime. A release is about changing the behavior of the system in ways that impact business metrics and matter to stakeholders. It's focused on improving the external quality of the system as experienced by end users.

DEPLOYMENTS

A production deployment can include new code hidden in a separate execution path, or changes such as refactoring, improvements to cross-functional requirements, or updates to test coverage. Ultimately, deployments enhance the internal quality of the system and may add new capabilities that can be released later.

Deployments can be fully managed by the product team and driven by technical needs, such as improving the system’s maintainability or keeping the code delta small between the trunk and production. Any unplanned change in observable behavior resulting from a deployment should be considered a regression.

DIFFERENCES

There are differences between production deployments and releases. For example:

1. Production deployment has no perceptible changes in the behavior of the system vs. Production release has visible change in the behavior of the system.
2. Production deployment is driven by technical needs vs. Production release is driven by product requirements.
3. Production deployment has no impact expected on business metrics but might have an impact on technical metrics vs. Production release implies an impact on business metrics such as user engagement.
4. Production deployment for rolling back changes means something went wrong technically with the change itself vs. Production release for rolling back changes means a product hypothesis didn’t yield good enough results.
5. Production deployment can be performed routinely vs. Production release can be performed when there is something substantial to release.

OVERLAP OF DEPLOYMENT AND RELEASE

A production deployment is often not a release, and a release is often not a production deployment. But, in some cases, the two will inevitably coincide: for example, when exposing new behavior through a static feature toggle (which require code updates), or with visible changes that are so small that they don’t require a feature flag, like bugfixes.

Separating them is particularly empowering under the You-Build-It-You-Run-It model. Deployments can occur daily or weekly, focusing on technical improvements and maintainability, while stakeholders retain full control over releases that impact the product's market performance. This distinction allows for more frequent, iterative updates without disrupting the user experience or market strategy.

However, deployments and releases have often been treated as synonymous in many companies and organizations, that means every potential deployment is a release even there is no user behavior change but only system improvement.  Thus, all those small potential deployments have to packaged together with a major release to release to production.  The bigger the release, the bigger the pain, and the higher the risk.   Additionally, in some organizations when security has unchecked power and their main job is to block any releases, on top of that, when there is no trust between product engineering and security, they will absolutely not allow separation of production deployment and production release.

Lack of well-planned dependency management

Dependency management spans various facets of a software product's lifecycle, including objectives, roadmaps, requirements, architecture, design, source code, tools, and operations. Each of these facets can harbor dependencies that need careful oversight. Without effective dependency management throughout the software product lifecycle, teams often face the need to release large codebases, which requires extensive and costly regression testing. This lack of management is a significant factor contributing to the necessity of large, infrequent releases rather than smaller, more manageable ones.

Software dependency management refers to the process of identifying, resolving, maintaining and visualizing the dependencies that a software product relies on. Dependencies in software development can be broadly categorized into two types:

1. External Dependencies: These are libraries, frameworks, modules, tools, or services that the software product uses, but which are developed and maintained by external parties.
2. Internal Dependencies: These are dependencies within the software product itself, such as modules or components that interact with each other. These dependencies are managed within the product’s codebase.

Dependency management can become chaotic for a PaaS or SaaS product in the cloud, especially when different parts of the system are owned by siloed organizations. For example, one organization might handle cloud services, including cloud account administration, VPC, and capacity planning. Another organization manages EKS in AWS or AKS in Azure, while yet another oversees EKS apps. Separate teams might be responsible for security tools and DevOps pipelines. All these components undergo regular changes, with minor updates happening monthly and breaking changes occurring quarterly. Without effective dependency management and automation, daily escalations are inevitable. Instead of focusing on deploying business logic and functionality, many organizations spend endless time coordinating and catching up with constant changes. Good dependency management with automation is critical to facilitate small and safe incremental change to production. 

Don’t practice trunk-based development

From 2010 to 2020 this decade, I have witnessed many debates on branching strategy in my own teams or other product teams, or major players in the industry.

•       Master only (trunk-based)

•       Development isolation

•       Feature isolation (topic branches)

•       Release isolation

•       Servicing isolation

•       Hotfix isolation

As the industry moves toward faster releases and software teams grow, most organizations have adopted trunk-based branching strategies. Trunk-Based Development is a key enabler of Continuous Integration (CI) and, by extension, Continuous Delivery (CD). When team members commit their changes to the trunk multiple times a day, it satisfies the core requirement of CI: that all team members commit to the trunk at least once every 24 hours. This practice ensures the codebase is always releasable on demand and helps make CD a reality.

However, a challenge arises: how can developers check in code daily, maintaining high test automation quality, when it's impossible to complete a new feature within a day? The solution is to hide incomplete features or functions until they are ready for release while maintaining code quality and avoiding storing code in a feature branch for extended periods to prevent merge conflicts. This is achieved through feature flagging management.  LaunchDarkly is one of the most popular tools to support feature flagging in the market.

Many organizations claim to support trunk-based development with committing code daily or without implementing feature flagging. Like Agile or Scaled Agile, engineers might check in code every day, but the trunk or main branch is never releasable. They still rely on manual checklists before major releases each quarter or longer.

Lack of DevOps architecture attributes: Testable and Deployable

Many aspects of DevOps are grounded in science. In the book "Accelerate: The Science of Lean Software and DevOps," Dr. Nicole Forsgren and her team’s research found no significant correlation between system type and delivery performance. What truly matters are two key attributes: deployability and testability.

Software product companies and organizations that grasp these fundamental architectural requirements excel by focusing on enhancing both attributes. This focus significantly boosts their software delivery performance, resulting in stronger business outcomes and larger market shares.

Low-performing companies often fail to invest enough time, effort, and resources into improving deployability and testability. This issue becomes even more critical as dependent infrastructure services, such as cloud services, third-party services, and open-source codebases, change rapidly. Without effective test automation and deployment environments, these companies struggle with lengthy update, upgrade, and release cycles. Consequently, they end up paying substantial amounts to extend support for those out-of-support services or libraries.

Security is often a blocker

Security in DevOps is paramount, and its complexity requires humility to acknowledge the ever-evolving nature of the field. With 28 years in the software industry, I have observed that the best DevSecOps companies are characterized by the mental maturity and humility of their security leaders. During my four years as DevOps CTO at HP Software, I partnered with the security product group to sell and land products like Fortify SAST/DAST. Our most successful customers were those who approached security with humility and discipline. For instance, a Senior Vice President from a government border-control high-tech department shared that their organization achieved a 95% pass rate in their DevOps pipeline, an unheard-of feat. She attributed their success to extensive training and education, which instilled a sense of pride in developers for maintaining high-security standards from the outset. She said, “Good habits are indispensable in both personal and professional life. Our engineers don’t even know how to write buggy code!”

Additionally, she noted that the organization was very disciplined; when there was a security issue, engineers and leaders willingly fixed it without exceptions. There was no engineering versus security mentality; the product team took full responsibility.

At Microsoft, often regarded as a security company, I collaborated with numerous security experts, including those who developed the security strategy with Bill Gates. These experts were notable for their humility, calmness, and thoughtfulness. Working with them made tackling challenging security topics enjoyable and productive.

Conversely, in organizations where security leadership exhibited arrogance, control, and dominance, DevSecOps issues were prevalent. These environments fostered a fight-or-flight or blame culture rather than collaborative problem-solving. Security's critical role in software release cannot be overstated, but when leaders abuse their position and the culture tolerates it, success becomes elusive.

A good security leader should be an enabler, educator, communicator, and listener. Only this type of leader can bring people together, inspire them to solve real problems, and confront the rapidly growing threats that all organizations face.

#11: Product Delivery Principle: Deployment Infrastructure

Deployment infrastructure has not been my strongest area of technical expertise, but it has become a crucial part of DevOps, product releases, and my job responsibilities in recent years. This diagram encapsulates the complexities involved. As we transition to a product-centric model for building PaaS and SaaS solutions, both the business logic and deployment infrastructure are now the product team's responsibility. This shift has caused significant upheaval in many organizations, fundamentally changing the way traditional Ops/Infra operates. Special thanks to a brilliant mind Paul Fijnvandraat from Microsoft for sharing this insightful diagram.

The deployment infrastructure has two main parts:

1. Deployment Infrastructure for Product Delivery Phase
2. Deployment Infrastructure for Product Operation Phase

Deployment Infrastructure for Product Delivery Phase

The first challenge is how we can have a robust deployment infrastructure to support the Product Delivery Phase.  The diagram highlights a fundamental issue faced by many companies: engineering teams lack isolated environments to thoroughly test their functionality. Consequently, engineers often resort to checking code into the trunk without essential testing against its dependencies. This practice not only contaminates the trunk but also destabilizes builds.

With no lower-level environment available, engineers are left to test in a high-level integration environment where multiple teams converge to test their functionalities simultaneously. This results in a chaotic scenario where many issues are surfaced only when everyone gathers, leaving limited slots for testing due to the overwhelming number of participants.

This model presents several challenges. Firstly, it frustrates engineers who take pride in their work but are hindered by the inability to validate their functionalities properly. Secondly, it leads to lower quality and increased release costs because numerous issues are identified only at later stages involving multiple stakeholders. Thirdly, maintaining high-cost, 24/7 operational testing environments, even when not fully utilized, becomes impractical and costly due to the need to coordinate downtime across global teams.

Deployment Infrastructure for Product Operation Phase

The second challenge is how the Deployment Infrastructure can support Product Operational Phase.

As Cagan pointed out, the most common way of doing this is to have your deployment infrastructure support A/B testing. This is considered the “gold standard” because you can easily isolate the contribution of just a single new capability. Many companies operating in the product model are running many (often hundreds or more) of these tests simultaneously. The deployment infrastructure runs these various tests, collects the analytics for each test, and continues until there are statistically significant results.

A/B testing is commonly facilitated through feature flagging mechanisms, with LaunchDarkly being a popular tool for this purpose. However, another significant advantage of feature flagging is its support for Trunk-based development, as highlighted in #10 of the Product Delivery Principles: Small, Frequent, Uncoupled Releases. This principle emphasizes the importance of integrating product code bases with feature flagging to achieve continuous integration and testing without long delays in feature branches.

By checking in code daily and protecting features with feature flags, teams can avoid prolonged code holding periods in separate branches. Instead, they merge directly into the trunk, enabling continuous integration and immediate testing. Once a feature is ready, activating its flag allows for functional testing across different environments.

As far as I know, all elite performers I worked for, or I supported had feature flagging management implemented for both product delivery and product operations. Unfortunately, many companies overlook or underinvest in developing this capability internally or acquiring tools for effective feature flag management. As a result, engineers often face challenges, leading to larger and lengthier releases that can impact productivity and delay deployments.

Product Delivery Principles

In the final installment of this series, Part #5, I will share my personal reflections on the principles of product culture, which is the most important aspect of product model.

#12: Principle: Principles over Process

#13: Principle: Trust over Control

#14: Principle: Innovation over Predictability

#15: Principle: The company cares about what the leader cares about