How do you design a clear and compelling A/B test report?

Before you even launch, you should have a clearly laid out template for documenting what you’re testing. This includes details on the background and insight (what's the opportunity and why you’re testing what you’re testing), the hypothesis, primary and guardrail metrics, and test variant and implementation details (e.g. audience targeting). What you then don’t want to overlook is including a section to report on your findings.The questions you want answered here are: did the test end conclusive, winner/loser? Was the hypothesis proven or disproven? Performance of variants and the effect (e.g. +4% CVR%), any insights for why you saw the result, and next steps. From here you can repurpose details for presenting with visuals and story.

The big challenge with SMART tests for me is the narrow view. I have found a lot of value in discarding the restrictions of the SMART goals and instead running broad tests. While one person might test a red CTA button vs a green one and measure success on CTR, I would rather we test 5 of our brand colors for the CTA and also moving the CTA higher up the page. My 10 variations would give a much better view of whether color in general has an impact, or if the location matters more, in a much shorter total time vs running each variation as its own SMART test. At the end of the day, data points drive better hypotheses, so artificial constrains in generating data will only slow progress.

It all starts with understanding the background of what we already have in terms of designs & actual implementation, understanding the overall success stats around it, & basing that defining the A/B test goal & hypothesis. For example, in my current project my client's vision is go carbon neutral & we wanted to make EVs more prominent & something which is easy to navigate on the website (And what's better way than to have it placed somewhere on the Global Navigation of the website), so that people can navigate easily towards those products. Using this information & existing data around what the user flow is towards such products on the website, what is the overall conversion, and so on, we defined the goal & hypothesis of our test.

Formulate a hypothesis that aligns with the goal, outlining the expected impact of the changes being tested. This sets the stage for a focused and actionable test that will yield valuable insights for decision-making Next, determine the key metrics that will be used to evaluate the test. Identify relevant KPIs that directly tie back to the goal and hypothesis, ensuring that they provide meaningful and actionable insights. These metrics will serve as the foundation for measuring success and informing future strategies based on the outcomes of the A/B test When creating the report articulate these steps concisely to ensure clarity and make it compelling by highlighting how each element supports decision-making and drives actionable insights

One of the most important aspects I often see neglected in A/B testing is ensuring that the goals and hypotheses are aligned with the company's strategy. Diverging from the company's direction can lead to a lack of global impact and reduce the overall effectiveness of the program. This misalignment can also demotivate stakeholders, as they may feel disconnected from the company's objectives. Therefore, ensuring that your hypotheses are in line with the company's goals is crucial for the success of the test.

Know your audience. Leadership may not need to know intricacies of the test execution (testing tool is irrelevant) compared to the results, learnings, and next steps. However, if you are discussing test results among product owners, you may need more technical details for that audience.

The methodology and setup of an A/B test should be clearly outlined in the report to provide transparency and ensure replicability. This involves detailing the specific changes tested (e.g., variants A and B), the duration of the test, and the criteria for inclusion in the experiment. Additionally, it's important to describe how users were randomly assigned to each variant and any potential factors that could impact the results. Include information about any control variables or external factors that were controlled for, as well as any limitations or assumptions made during the test setup. To create a clear and compelling report, present this information in a structured manner.

The mistake often made is making reports either too complicated or adding unnecessary details. If the report is too complex, people won't understand it, and if it includes too many irrelevant details, it becomes overwhelming. For a small company, create accessible reports with a clear hypothesis framework, like the one on the Henkan & Partners website. In large companies, tailor reports to different audiences. I recommend a one-pager with key test information, a detailed report for those involved in the test needing in-depth insights, and a two- or three-slide deck for presenting the experiment to top management.

For multivariate testing, A/B/C... Testing, I randomize combinations. If we test 5 attributes with 4 options each, we will have 625 unique combinations. Statistical significance for that many would be a challenge. Instead, we use (ex. 50) random combinations. Each of the 4 options of each attribute shows up in ~12 versions, and we reach statistical significance with 92% fewer impressions. Each combination is labeled based on its attributes and values for simple parsing & analysis later. (ex: greenCTA_topCTA_informal_noHeader_OceanPic)

For A/B testing methodology and setup: Objective Definition: Clearly state what you aim to improve (e.g., conversion rate, engagement). Hypothesis Creation: Formulate a hypothesis based on insights or assumptions (e.g., changing the call-to-action button color will increase clicks). Variable Selection: Identify the element(s) to test (e.g., button color: red vs. blue). Control and Variation: Establish a control group (original version) and one or more variations with the change applied. Audience Segmentation: Divide your audience randomly to ensure unbiased results. Sample Size Determination: Calculate the sample size needed for statistical significance. Test Duration Success Metrics Data Collection & Analysis Ensure Technical Setup

Designing a clear and compelling A/B test report involves succinctly presenting your findings. In my experience as a CTO, the effectiveness of a report hinges on how well it communicates the core results and their implications. Utilize visual aids like charts and graphs to outline differences between control and variant groups. It's crucial to annotate these visuals with clear, concise captions that explain statistical significance and potential anomalies. Additionally, explaining the statistical methods in plain language ensures that stakeholders without a technical background can grasp the conclusions and their impact on decision-making. This approach fosters a deeper understanding and confidence in the test outcomes.

Presenting A/B test results is like storytelling. It's not just about data, but the narrative behind it. Compare control and variation with clarity, highlighting key differences and trends. Acknowledge errors or anomalies as lessons, much like in leadership. Confidence in results reflects your understanding, tempered with an awareness of limitations. Use visual aids for clarity, making complex data accessible. Your statistical methods are tools, but your interpretation in plain language is what truly informs and inspires action. Remember, data tells a story, make it compelling.

Pour partager les résultats, je montre d'abord l'original vs le variant en expliquant l'hypothèse testée et le changement unique effectué. Ensuite, je montre le CTA ou le KPI choisi et le taux d'amélioration du variant. Je précise également le nombre de visiteurs uniques, ainsi que le taux de fiabilité.

Present A/B test results with clarity. Summarize key metrics, visualize differences, and discuss statistical analysis and outliers. Contextualize results within the test's goals and hypothesis. Provide actionable insights for decision-making. Avoid technical jargon for broader stakeholder understanding.

When presenting A/B test results, it is important to communicate the entire chain of process, from the insight gathered, to a hypothesis framed based off of it, to the actual visuals of control and variation and then clear presentation of the performance against the main defined goal that determines the recommendation. Based on my experience have both the longer thought process explanation in a couple of slides, as well as a concise version, needed i.e. for the annual learning summary. Both have value and different purposes.

Analyze A/B test results to derive insights and recommendations. Identify trends, patterns, and significant findings. Utilize data to suggest improvements or actions for optimizing performance. Communicate these insights clearly and concisely to stakeholders for informed decision-making.

Proving incrementality is technically impossible, which is why we use statistical analysis. Be specific. "We found the XYZ audience segment clicks our CTA 5.1% more often (95% confidence range of -1% to +15%) when we put the CTA directly below the header image AND shrink the header image to 50% of standard height (75px vs 150px). No other audience segments responded positively to this combination of changes. We recommend a 20% mix of this new layout for the XYZ audience for the summer ABC campaign, which should drive an incremental $27.3mm (-$5.5mm to +$81.9mm)."

This is a crucial step for the A/B test that you are running because if the wider audience is not clear of what the test was intended to & what are the results, then it is a failed test. For the A/B test that we ran on the Global Navigation of the client's website, we learned that there was a drastic uplift in number of users navigating to the EV segment & checking the product, thereby making our test a success. The benefit we got from a 50-50 split in audience was that we got a fair understanding of user interest since equal number of people were exposed to both the experiences. As part of next steps: Implement the success variation, Meeting accessibility standards, Page Performance testing, Retesting the results to understand impact.

Interpret the results to derive actionable insights, explaining how the outcomes support or refute your original hypothesis. Based on the analysis, offer detailed recommendations on whether to implement the changes, conduct further testing, or explore alternative strategies. Highlight the implications for business objectives and future marketing activities. Comprehensive insights empower decision-makers to take informed actions that align with organizational goals.

Summarize the insights gained from your analysis and offer actionable recommendations based on the results. Discuss what the data reveals about the effectiveness of the tested changes and how these findings can inform future decisions. For example, if the new design improved user engagement, recommend implementing the change across the platform. Provide practical next steps, such as further testing or adjustments, to guide readers in applying the insights to their own projects.

Encourage further exploration by highlighting any limitations encountered during the test which may affect the overall findings. Discuss potential confounding variables or external factors that could influence user behavior during the testing period. Suggest areas for additional research or subsequent A/B tests that could provide deeper insights. This forward-looking perspective fosters a culture of continuous improvement and learning.

-Document the test in knowledge base with visual copy of both variations. This documentation will help maintain database of learnings and double down on what worked and prevent retesting same ideas repeatedly. In large organizations with multiple teams running various tests, sharing of knowledge accelerates the learning curve. -Delete the underperforming variation (not just turn off traffic to it) to maintain clean coding practice.

Make it matter to your stakeholders. Put it in their language. Speak about your results in terms of their goals. Start with a high-level summary following other advice on this article, and then ask your stakeholders what they would like to hear more about. Don't forget to talk about next steps. Periodically review your rubric for selecting A/B tests from a backlog with your stakeholders to make sure you're prioritizing the experiments your stakeholders want.

Don't forget to share your reports with the relevant stakeholders so that they can learn from them too! Share an email newsletter with a round-up of recently completed A/B tests or publish an article on your company's intranet every now and then. This is a great way to ensure that your organization can actively participate in the experimentation & optimization process.

In addition to the main report sections, consider including a discussion on the limitations of your A/B test and any external factors that might have influenced the results. Address potential areas for improvement in future tests and provide suggestions for additional experimentation. For example, if the test was limited to a specific demographic, recommend testing the changes with a broader audience to validate results. This comprehensive approach ensures your report is thorough and provides valuable context for interpreting the findings.