Why relying on just a/b testing could be holding back your CRO strategy
As marketers and CRO professionals, we’re all familiar with the power of A/B testing. It’s essential to understanding user behavior and providing valuable insights into what works on a website.
But here’s the thing: A/B testing can only tell you what is happening, not why it’s happening. That’s where Mixed Methods Experimentation (MME) can unlock the true potential of your experimentation program.
𝗧𝗵𝗲 𝗹𝗶𝗺𝗶𝘁𝗮𝘁𝗶𝗼𝗻𝘀 𝗼𝗳 𝗾𝘂𝗮𝗻𝘁𝗶𝘁𝗮𝘁𝗶𝘃𝗲 𝗮𝗻𝗱 𝗾𝘂𝗮𝗹𝗶𝘁𝗮𝘁𝗶𝘃𝗲 𝗿𝗲𝘀𝗲𝗮𝗿𝗰𝗵:
𝗤𝘂𝗮𝗻𝘁𝗶𝘁𝗮𝘁𝗶𝘃𝗲 methods like A/B testing and analytics provide robust data about user actions, but they leave us wondering why users behave the way they do.
On the flip side, 𝗾𝘂𝗮𝗹𝗶𝘁𝗮𝘁𝗶𝘃𝗲 methods, like UX research and contextual interviews, help us understand the motivations and contexts behind user actions. However, these insights are often limited by sample size and generalisability.
𝗘𝗻𝘁𝗲𝗿 𝗠𝗶𝘅𝗲𝗱 𝗠𝗲𝘁𝗵𝗼𝗱𝘀 𝗘𝘅𝗽𝗲𝗿𝗶𝗺𝗲𝗻𝘁𝗮𝘁𝗶𝗼𝗻:
MME is the strategic combination of both quantitative and qualitative methods, enabling us to bridge the gap between what users are doing and why they’re doing it.
𝗨𝗻𝗰𝗼𝘃𝗲𝗿𝗶𝗻𝗴 𝘁𝗵𝗲 𝘄𝗵𝘆 𝗯𝗲𝗵𝗶𝗻𝗱 𝘁𝗲𝘀𝘁 𝗿𝗲𝘀𝘂𝗹𝘁𝘀:
Take, for example, a recent project with a global technology client. The team hypothesized that replacing product-focused imagery with lifestyle images would resonate with their remote-working audience.
𝘛𝘩𝘦 𝘈/𝘉 𝘵𝘦𝘴𝘵𝘴 𝘴𝘩𝘰𝘸𝘦𝘥 𝘢 𝘤𝘭𝘦𝘢𝘳 𝘥𝘳𝘰𝘱 𝘪𝘯 𝘤𝘰𝘯𝘷𝘦𝘳𝘴𝘪𝘰𝘯𝘴.
If they’d stopped there, they might have concluded that lifestyle images didn’t work. But by integrating UX research, they discovered that:
• Users, now shopping from home, preferred detailed product images that replicated an in-store shopping experience.
• The lifestyle images featuring glossy stock photos didn’t resonate with their authentic work-from-home environments.
Combining the insights from A/B testing with qualitative research could refine the approach and test better-informed hypotheses, ultimately leading to more impactful results.
𝗪𝗵𝘆 𝘆𝗼𝘂 𝘀𝗵𝗼𝘂𝗹𝗱 𝗶𝗻𝘁𝗲𝗴𝗿𝗮𝘁𝗲 𝗺𝗶𝘅𝗲𝗱 𝗺𝗲𝘁𝗵𝗼𝗱𝘀 𝗲𝘅𝗽𝗲𝗿𝗶𝗺𝗲𝗻𝘁𝗮𝘁𝗶𝗼𝗻 𝗶𝗻𝘁𝗼 𝘆𝗼𝘂𝗿 𝘀𝘁𝗿𝗮𝘁𝗲𝗴𝘆:
• Understand the what (quantitative) and the why (qualitative) behind user actions.
• Use qualitative research to form more user-centered and strategic test hypotheses.
• Combining multiple data sources allows for smarter, data-backed decisions.
• MME helps ensure your tests are driven by real user insights, leading to higher success rates.
Incorporating Mixed Methods Experimentation into your strategy ensures that you’re not just optimizing for what works but also understanding the deeper reasons behind it, allowing for more targeted and effective experimentation.
#cro #experimentation #uxresearch #conversionoptimisation
