How can you run A/B tests with Optimizely and Google Optimize?

Unfortunately, Google Optimize and Optimize 360 are no longer available as of September 30, 2023. Depending your budget, there are other tools available. For example VWO (Visual Website Optimizer) has a free plan to get started and charges you based on "Monthly Tracked Users", which is great for smaller businesses that are just getting started. Finally, it's important to also consider GDPR compliancy of the tool(s) available.

Optimizely indeed offers a more robust set of features, especially when it comes to advanced testing types and personalization. However, its higher cost might be a limiting factor for smaller projects or teams with budget constraints. On the other hand, Google Optimize's simplicity and integration with Google Analytics make it a cost-effective solution for straightforward A/B testing. One disadvantage not explicitly mentioned in the article is the learning curve associated with Optimizely's advanced features. While they provide great flexibility, they might require more time and resources for the team to fully grasp and utilize effectively. Shortly, carefully assess your project requirements, team expertise, and budget constraints.

You budgets plays a big role here. If you are just starting, dont throw a lot of money into a tool that you will only use the basic features of it.

Google Optimize and Optimize 360 are no longer available as of September 30, 2023. Google is now working with a few A/B testing vendors for integration with the Google Platform and tools: - AB Tasty - Optimizely - VWO I also would also consider looking at Adobe Target and the Adobe ecosystem of tools. Adobe Target is a powerful A/B testing tool with robust personalization capabilities as well. Of course, there is cost involved with these tools, so review your budget and needs when evaluating each platform.

The first thing you should do is see if they both exist ... One of them does not, so this should help make the decision-making-process a lot easier.

One disadvantage not explicitly mentioned in the article is the potential for skewed results if the sample size is not representative or if the test duration is too short. In my design practice, I've encountered situations where A/B tests produced misleading insights due to insufficient data or inadequate testing periods. Always remember that metrics are indicators, not the ultimate goal. Focus on enhancing the user experience, and metrics will naturally follow. Continuously gather qualitative feedback, iterate on design variations, and involve users in the testing process.

Your Pi value is the core formula that you need to define before setting anything in the A/B testing tool. After that goes your primary and seconday goals.

Your CRO goals should align with the department or business goals. If a department is looking to increase engagement on a specific tool, increase add-to-carts, to increase conversions, the simplest method would be to align with either of those. It's also a good idea to set quarterly goals within the department / business so that CRO has a quarterly goal to align with.

There are many approaches to creating variations - AB testing, ABn testing, Multivariate testing, or user journey (multiple page) testing. Your approach will depend, primarily, on how much traffic you'll need to reach statistical significance. Leverage experts in UX, design, data analytics, and marketing to create the variation experience. UX will often have Voice of Customer data which will inform where opportunities lie.

There are lots of approaches to this, and it often depends on what the source of the traffic is. You might use a query parameter on the URL to ensure that only specific source traffic is captured into the experiment. You might segment by login status. You might segment by the page that the user has come from (referral page). Make sure that your experiment is isolated to the devices that you're wanting insights on - and ensure that if it's on all devices, that the experience is consistent across those devices.

Some tools are great for analysing data within the platform. Optimizely has the capability to ensure that P values are accurately calculated whereas Google Optimize (when it existed) didn't always. Generally, it's ideal for your data to feed into your analytics tool so that you're able to account for the total traffic as part of your overall reporting, and also keep consistency with respect to dimensions, goals, conversions, etc. Utilising your current (well-established) analytics tool will allow for a more consistent measurement and more robust reporting.

Make sure that the result of your AB test has statistical significance. And make sure to properly explain to stakeholders when it doesn't. With a statistically significant result, the difference between tested variations is not due to error or random chance. There are tools to calculate statistical significance. Just search Google for "Calculate your statistical significance".

Before you start testing your first A/B test make sure to run A/A test to make sure that your testing tools is being setup correctly.

Learning and iteration are pivotal in A/B testing. After an A/B test, analyze the results to glean actionable insights, focusing on why specific changes worked or didn't. This process often uncovers unexpected user behaviors or preferences that can inform future product development or marketing strategies. Use these insights to refine your hypotheses and develop new tests. Iteration is key; it's rare to find a 'perfect' solution in the first test. Each iteration should aim to build on the previous one, using the accumulated data to make incremental improvements. This fosters a culture of continuous learning, allowing your product or strategy to evolve.

Consider whether Google will be able to accommodate this particular challenge. My understanding is that it no longer does so do the rework and be prepared.