Transformed – Personal Reflection Part #3: Product Discovery

In part #1, I shared my personal reflections on the principles of product teams. In part #2, I discussed my thoughts on the principles of product strategy. In part #3, I will share my insights on the principles of product discovery.

I believe that product discovery is often the weakest area in many product teams because they lack the proper mechanisms to consistently present their ideas to real users and customers early and frequently. Based on decades of data, it clearly shows that the majority of solutions built in IT model (generally 70 to 90 percent) end up not delivering the necessary business results.

Product Model Principles

I listed 15 principles where I have strong examples to demonstrate real-world experience. This is not an exhausting list of principles, if you wish to know more, please check out Marty Cagan's book.  I will post 5 articles to cover all 15 principles and my personal reflections.

Product teams

#1: Principle: Empowered with Problems to Solve

#2: Principle: Outcomes over Output

#3: Principle: Sense of Ownership

#4: Principle: Collaboration

Product strategy

#5: Principle: Focus

#6: Principle: Transparency

#7: Principle: Placing Bets

Product Discovery

#8: Principle: Assess Product Risks

#9: Principle: Test Ideas Responsibly

Product Delivery

#10: Principle: Small, Frequent, Uncoupled Releases

#11: Principle: Deployment Infrastructure

Product Culture

#12: Principle: Principles over Process

#13: Principle: Trust over Control

#14: Principle: Innovation over Predictability

#15: Principle: The company cares about what the leader cares about

#8: Product Discovery Principle: Assess Product Risks

As an engineering executive, I am deeply involved in the selection, evaluation, and approval of tools. Despite the common understanding that DevOps challenges are driven by people (50%), process (40%), and technology/tool (10%), we often overlook that deciding on technology and tools primarily involves people and process. Here are a few critical considerations:

1. Who is the business owner of the tool, and in which organization do they reside?
2. How will this new tool integrate into the existing ecosystem of tools?
3. Which organization will manage, operate, and upgrade the tool?
4. Should we opt for SaaS, on-premises, or a combination of both?
5. How much effort will be required to customize the tool?
6. How will we forecast the tool's licensing requirements?
7. What is the plan for migrating code from the old system to the new tool?
8. Whose roles will be impacted by the introduction of this new tool?
9. What level of support and professional services will we need?
10. How will we enable tool adoption along with best practices?

These are organizational questions that need to be addressed. However, the most crucial questions often overlooked are whether the new tool is valuable, usable, feasible, viable, and ethical for us to use. This consideration isn't just about the product we are building; it's also about the tools we are buying. Whether the tool is part of the product ("cookie") or the product development process ("cookie machine"), its implementation has significant implications. Even if it is just part of the cookie machine, the way the tool is integrated greatly impacts people and processes, more so than technology.

Case Study in a Tool's Evaluation

Here's an example from my organization. I've removed some sensitive data, but the overall message remains unchanged. This email was sent when the tool was ready for legal approval.

What I realized, even with top-notch engineering principals I recruited from leading high-tech companies, is that brilliant technologists often lack a framework to assess risks effectively. This is where Cagan’s Product Discovery questions become so profound and simple to use.

Here was my email:

“Thank you for your dedication and hard work on this tool evaluation.  However, I had my observation as we discussed today. 

I always appreciate Jeff Bezos’s thinking in making decisions:  He’s always thinking three years out and only makes a few good decisions a day. Bezos only makes a few decisions a day. 'As a senior executive, what do you really get paid to do? You get paid to make a small number of high-quality decisions,' he said.

As far as the decision goes for XYZ, I asked the following questions that need to be asked, and evaluated so we know we have done our homework with confidence.

#1: Viability.  Will XYZ still be around in 2 years? What if they go out of business, and what is the impact on us? Given this vendor has only SaaS version.

For example: What impressed investors the most were two facts about Facebook’s early growth.

1. The first fact was the raw account of time Facebook’s active users spent on the site.  More than half of the users came back to the site every single day. This is an example of how a company can validate its value hypothesis, the customers find the product valuable.
2. The second impressive thing about Facebook’s early traction was the rate at which it had taken over its few college campuses. Facebook launched on February 4, 2004, and by the end of that month almost three-quarters of Harvard’s undergraduates were using it, without a dollar of marketing or advertising have been spent. Facebook had validated its growth hypothesis.
3. $12.7M in less than 1 year.

 Here is the data on XYZ:

1. XYZ is ranked 2,017,898 among websites globally based on its 8,713 monthly web visitors.
2. Competitor Rank: 26th out of 135 competitors with a Tracxn Score of 16/100. 
3. US monthly visit growth rate in US: -33.35%
4. India monthly visit growth rate in India: -41.07%

We have to answer this question with a backup plan or a roll-forward plan. It is too risky not to have an answer.

 #2: Meeting with Vendor is a must-have.

For all products we evaluate, we must meet with vendors and key stakeholders to ask questions and understand their future plans. These meetings expose a lot of information that shapes our confidence, trust, and perception. None of us have met with this vendor. For the most well-established vendors, we still scheduled 7-8 technical deep dive sessions with their technical architects and engineers.

 #3: SaaS vendor must-ask questions: data center location, DR details, SOC compliance, etc. XYZ is based in Europe, what about the GDPR requirements.

#4: QA team should lead this initiative:

Ideally, the QA team should lead this initiative as it is test related. The QA team is a more natural party to lead this, even though you can be the strategist behind the scenes. The QA team can engage with current users and development leaders to systematically gather their operational data, requirements, and training needs. When working with vendors, we also need to consider services and support needs, including training. Vendors are often better equipped to provide basic training at scale. I don't think this has been discussed yet.

 #5:  Do they have any community presence in GitHub for XYZ? 

Community is one of the most important aspects in today’s software work. A well-run community demonstrates strong leadership in vision, mission, and strategy. Here is what I typically do when evaluating open-source tools.

Evaluation criteria for DevOps Tools:

1. Community: An active open-source community, the number of commits, releases, contributors, etc.
2. Risk: Minimal security risk as vetted by the security and risk teams.
3. License: Compliant with the organization's OSS license policies.
4. Function: Support for both current and future functional needs.
5. Integration: The tool can integrate with the rest of toolchain, with minimal customization.
6. Redundancy: The tool does not create functional redundancy or duplication in the toolchain.
7. Skills: The team possesses the necessary skills to deploy and maintain the tool.
8. Openness: The tool is not chosen based on tribal knowledge or familiarity bias.

With those concerns, I don’t recommend moving forward at this point with legal and procurement till we have more confidence and supporting data.”

In the end, we collectively agreed that this tool wasn't suitable for us even though it was valuable, usable and feasible with validation from a PoC, but it is not viable, as a result, we decided to cancel the procurement process.

This became a valuable learning moment for everyone involved. I particularly valued Cagan's straightforward yet impactful Product Discovery questions that brought us all together. This is a cheat sheet I use every day.

#9: Product Discovery Principle: Test Ideas Responsibly

Two-year-long containerization project

As I mentioned previously, when the architecture group or CTO office has unchecked power, and transformation is driven purely by technology and tools within a project-centric model, it inevitably leads to negative outcomes. These include losing market opportunities, allowing competitors to gain market share, business setbacks, and the departure of top talent.

Years ago, I worked for a top-tier software product organization that led the market with its outstanding products. Our annual offsite celebrations were a testament to our success, with high morale, generous gifts, and fantastic dinners.

However, when containerization became a hot trend, the Architecture Group mandated that all product teams transition to containerized environments. This painful two-year process was fraught with technological challenges, and by the end, our containerized products were barely functional. In the fast-paced software industry, two years is a significant amount of time to not focus on customer outcomes and innovation. Our customers became dissatisfied, and our competitors capitalized on our prolonged period of technological transformation.

Meanwhile, as we were grappling with containerization, the shift to cloud computing began. We faced new decisions about which cloud provider to adopt. This added another layer of complexity, preventing product teams from delivering customer outcomes that aligned with our business goals.

The aftermath was dire. The organization underwent numerous reorgs, including changes at the BU CEO level, but these changes did not address the core issues. If the principles in this book had been available then, they could have saved our business. In today's consumer-centric world, technology and architecture-focused transformation alone is insufficient. Customers don't care about the tools you use; they care about how your solution solves their problems. Every day, we must focus on finding solutions that address customer problems while also benefiting our business.

Test ideas responsibly. While containerization is an industry trend, we must prioritize solving customer problems even as we adopt new technologies. Every new technology, despite its promising benefits, comes with its own set of challenges. Often, a technology's biggest strengths can also be its biggest weaknesses. If we focus only on the strengths and ignore the weaknesses, the consequences can be severe.

GitOps Model

Nearly three years ago, one of my major customers approached us for assistance in deploying a GitOps model. The sponsors were the company’s Platform Engineering VP and the Chief Architect of the Global Architecture Group. Having read some books and attended industry presentations, they were convinced that GitOps model was the silver bullet that could solve their problems and deliver business results.

What is GitOps?

GitOps is a way of implementing Continuous Deployment for cloud native applications. It focuses on a developer-centric experience when operating infrastructure, by using tools developers are already familiar with, such as Git, Infrastructure as Code, and Continuous Integration. The core idea of GitOps is having a Git repository that contains declarative descriptions of the infrastructure currently desired in the target environment and an automated process to make the environment match the described state in the repository.  If you want to deploy a new application or update an existing one, you only need to update the repository — an automated process handles everything else.  In general, this solution benefits organizations that want the advantages of deploying applications and infrastructure as code, with an audit trail of every change. The solution is especially suitable for highly regulated industries like insurance, banking, and finance.

At the time, our group showcased the GitOps community work in Azure Kubernetes Service (AKS) architecture using Flux as the GitOps operator and controller, Open Policy Agent (OPA) Gatekeeper to enforce policies with a validating admission webhook, and Syncier Security Tower as a GitOps control kit that provides an overview of all AKS clusters and helps manage policies.

During our initial conversation, it became clear that while they aspired to implement GitOps, they weren't ready for such an advanced approach. Their DevOps processes were in disarray: they still had manual build handovers, lacked true Continuous Integration, and had a dysfunctional organizational structure where different teams used disparate processes and tools, making integration manual and cumbersome. They relied on a monolithic system to run their primary business, making automation nearly impossible. Additionally, people were leaving the organization due to the old technology and lack of learning opportunities, so leadership hoped to introduce new technology to excite the team and curb attrition.

We proposed a value stream assessment (free of charge) to gain a comprehensive understanding of their current cultural, organizational, technological, and architectural practices. They invited key leaders from various functions, and what started as a one-hour planned session expanded into 5-6 two-hour sessions, including the final readout. We identified their major challenges, which included:

1)      Very waterfall process using SAFe

2)      Very limited product discovery effort as mostly decided by architecture team and stakeholders.

3)      Requirement management has no single tool and single process; some organizations still use spreadsheet.

4)      Monolith systems

5)      Huge technical debts

6)      Don’t want to release due to Tax session

7)      No Continuous Integration

8)      No Continuous Delivery and Self Services

9)      Weak on Functional and Non-functional testing

10)   Security test is at the end of Lifecyle

11)   Limited measurement on outcome

12)   No full-stack, engineers, security and operations are separate.

In the context of GitOps, we recommended that they at least implement true CI and CD practices within a full-stack model before attempting GitOps. Without these foundations, GitOps would merely be a tool exercise with no real benefits. The customers were very reasonable and logical. After this assessment, two of their engineering leaders acknowledged that their initial belief that GitOps could solve their product delivery problem was naive. Consequently, they agreed to engage in a comprehensive DevOps learning journey to understand best practices, purchasing our DevOps Dojo services.  As far as I know, three years later, they have not yet adopted the GitOps model. However, they have made significant progress in fundamental DevOps practices, setting a solid foundation before moving forward.

Like any medicine, while it might solve one problem, it could also have side effects. According to my doctor, anyone who is absolutely certain about something lacks humility; we don't truly understand anything until we use a scientific approach to validate or invalidate our hypothesis.

This chart outlines the basic concepts of GitOps. During our experiments, we encountered some interesting challenges:

1. Operational Resistance: GitOps is promising for operations due to its continuous reconciliation. However, this approach represents a significant shift for organizations with many operational staff who perform manual tasks. These individuals may fear for their job security since updates can only be made through Git version control, not directly in the production system. In the "Product Delivery Principle: Deployment Infrastructure" section, you'll see why in PaaS or SaaS systems, apps and infrastructure are part of the products and should be developed and managed by product engineering teams. While GitOps is promising technologically, it requires substantial organizational change management to address skillset, process, and organizational gaps.
2. Development and Continuous Delivery Challenges: Although GitOps is designed to improve operations, it should be practiced in development and continuous delivery environments as well. In development, we use self-service automation to auto-provision environments. However, when teams want to test by shutting down certain services or manipulating the environment, they face issues because the GitOps model overwrites any direct changes to the environment configuration. The configuration must be altered at the Git level. Changing the configuration at the trunk level affects everyone, so we must allow each team access to their own configuration. This requires control plane access, which may raise security concerns. Thus, GitOps' biggest strength in operations can be its challenge in development. Solving this paradoxical problem is essential before scaling to both development and deployment.
3. Performance Issues: We discovered that constant reconciliations between Git changes and target systems generated significant traffic, slowing down the Git version control system. This slowdown affected all engineers, even those working on business logic development, as Git repositories became painfully slow.
4. Concurrency Issues: Another surprise was the concurrency problems. With GitOps, everything is code, including configuration and infrastructure, making self-service for environments possible. However, this demand for more auto-provisioned instances led to concurrency issues, causing many failures when multiple provisions occurred simultaneously.

Addressing these challenges is crucial for successful GitOps implementation, balancing the benefits with the practical difficulties in both operational and development contexts.

In summary, it is crucial to test ideas responsibly, especially when dealing with new technologies and unproven architectures. Blindly promoting concepts that look good on paper can be dangerous. Architects cannot accurately judge the long-term viability of any architecture until it has been successfully designed, implemented, upgraded, and adapted to inevitable changes. Even then, the architecture must prove resilient under unusual circumstances to be considered truly robust.

Product Delivery Principles

In part #4, I will share my personal reflections on the principles of product delivery.

#10: Principle: Small, Frequent, Uncoupled Releases

#11: Principle: Deployment Infrastructure