What are some common pitfalls and biases to avoid when validating your assumptions?

In my experience this manifests in startup founders when they get really attached to a narrative that makes them feel good. It can be something that looks like a productive way to spend time, but is actually a waste. For example, when we were building the earliest MVP of Sophonaut, I heard that burnout was a problem for wealth advisors from a Sr. VP at a large bank. It wasn't enough data, but because I had thought burnout could be a vertical we could solve, we made business decisions off of that one data point. It ended up leading to a dead end we could have avoided.

An example of this bias can be seen in market research where a startup founder, eager to validate their assumptions, selectively focuses on positive feedback while dismissing or downplaying negative responses. This bias can lead to overconfidence in the assumptions being validated and overlooks potential risks or gaps in the value proposition. To mitigate confirmation bias, founders should design validation experiments with clear and falsifiable hypotheses, utilize objective metrics, and actively seek feedback from diverse and unbiased sources to ensure a balanced perspective on their assumptions.

How early stage startups can conduct customer discovery while avoiding confirmation bias: 1) Start with Questions, Not Assumptions Frame open-ended questions that encourage your audience to share their experiences and challenges without leading them to your solution. For example, instead of asking, “Would you use a product that does X?” ask, “Can you tell me about a time when you faced challenges with [problem]?” 2) Listen The goal is to understand, not validate. Let your customers do most of the talking, and resist the urge to jump in with your ideas. 3) Test Hypotheses with Data Turn qualitative insights into actionable hypotheses, then test them with experiments (e.g., landing pages, prototypes, surveys).

Selection bias occurs when the sample you choose to validate your assumptions is not representative of your target market. For instance, relying solely on feedback from friends, family, or existing customers can give a distorted view of your product’s potential. To avoid this, ensure you gather data from a diverse and representative sample that includes different demographics, user behaviors, and market segments. Use random sampling methods and reach out to a broader audience through surveys, interviews, and user testing. By avoiding selection bias, you can obtain more accurate insights and make better-informed decisions, ultimately strengthening your startup's foundation.

The easiest way to overcome selection bias is to constantly measure your current data set against a really strong set of "priors". There are two important take homes from that: 1) the important steps in market segmentation are the earliest steps and 2) you need to have a solid framework for measuring things that doesn't change too much with time. If you have one without the other, you won't have a way of knowing if your dataset is biased to begin with.

A startup testing a new product might only gather feedback from friends and family, inadvertently excluding diverse perspectives from potential customers. This bias can result in missing crucial insights or feedback from various market segments. To counter selection bias, founders should define their target audience and sample size based on the problem and solution, employ random or representative sampling methods, and balance qualitative and quantitative data sources to ensure a comprehensive understanding of their assumptions.

The most famous example of Survivorship bias is the bullet hole patterns from planes that returned from battle in WW2. Planes returning home had common patterns in where they were shot, and engineers were deciding what panels they should reinforce with armor. One engineer concluded: "if we reinforce the places with the most bullet holes, we will reduce the damage to the aircrafts!" Curiously, the planes that returned home had no damage in their engines. You see the problem with that line of thinking? The planes that were shot in the engines never made it back. The take home from survivorship bias is not simply that some things are not reproducible, it's that this data can lead to conclusions that are the OPPOSITE of the productive solution.

As the EU Sustainable Action Plan gains significant traction, companies and especially startups face increased scrutiny for the sustainability of their products. Through strategic investments, companies with a positive environmental impact can thrive, while those with a detrimental impact are compelled to adapt or face the potential consequence of investors reallocating their capital. For example, in Rahim Invest we assist our clients - Gulf region investors by conducting research on investee companies in making informed investment decisions, fostering a more sustainable future.

When founders start a business, they sometimes love their first idea so much that they don't want to change it. But it's important to listen to what your potential customers say, so you can understand their problems and find out their pain point. This can help avoid building products and solutions that nobody wants.

Validating bias and assumptions is an ongoing process, not a one-time event. Remain vigilant, question your biases, and continuously seek new data and perspectives. Imagine being told a product costs £1000 before seeing its features. This "anchor" price influences your perception, making a ££500 price seem more reasonable. Anchoring bias can distort your analysis. Avoid setting expectations beforehand, evaluate information objectively, and base your conclusions solely on the evidence at hand. We often overestimate our knowledge and the accuracy of our assumptions. When choosing respondents for validation, ensure they represent the true target population, not just a convenient subgroup that reinforces your/our assumptions.

In my experience, the most important thing to remember about cognitive bias is that simply KNOWING about all these biases above won't protect you from falling prey to them. There are cases where intuition and gut feelings serve the founder better than anything, and cases where they actually create more problems for everyone. Our job as founders is to manage our own psychology well enough to recognize how and when to use what tools.