The Half-Baked Reality of Testing Google's Privacy Sandbox

Since the start of Google’s 1% deprecation of third-party cookies, the ad tech industry has released a slough of strongly-worded opinions on what the Privacy Sandbox solutions could hold for the future of digital advertising. 

For folks building the Privacy Sandbox, the future is bright; they have no issue touting the robustness of the proposed solutions, especially Protected Audience API (PAAPI, formerly known as FLEDGE). Industry experts, however, aren’t nearly as confident, making claims that the Privacy Sandbox isn’t ready based on evaluations that it doesn’t provide the kinds of solutions needed to facilitate digital advertising. These diametrically opposed views leave the average advertiser confused and simply wanting clarity on what is actually going to be supported, what is going to change, and what the impact will be. 

Spoiler: it’s still too early to make concrete claims one way or the other. Let’s talk about why…

TLDR; The common question we hear from advertisers is “when can we test?” meaning when can they–the advertiser–see how differently campaigns will perform after third-party cookie deprecation. Our assessment is this: since market adoption is still ramping up, performance and solution testing is possible at a limited macro level, but does not support performance impact testing for individual advertisers. 

Test Design

A good design to test a new system requires a representative test and control groups. There are several ways to achieve this. One of the most commonly employed is to use a randomized controlled test framework. Test participants are randomly chosen to participate in both the control and test groups. This results in comparable test groups that allow accurate measurement of impacts from treatment applied to the test group. 

We see good randomization of test cohorts with the 1% of browsers that had third-party cookies removed and in the 10% of browsers with labeled testing. However, in the mix we also have market players on the supply and demand sides who have self-selected, resulting in non-representative test groups. 

Next, let’s take a look at how much of the market has adopted these Privacy Sandbox solutions.

Market Adoption 

With all the press about third-party cookie deprecation and the Privacy Sandbox, it is easy to assume extensive performance testing is underway. While testing has indeed been ongoing since 2020, it's crucial to define what constitutes "meaningful testing" and how success is being measured. When discussing “meaningful testing” with advertisers, the questions asked are along the lines of “how will third-party cookie deprecation impact my campaign performance and therefore my bottom line?” However, it is too early to ask this question due to the lack of market adoption. Instead, let’s take a look at what NextRoll is seeing in terms of supply and demand side adoption.

Supply Side (Publisher)

During NextRoll’s currently test of the Privacy Sandbox, it is clear that the supply side still requires more time to scale up. Part of this is due to the low volume of browsers–instances of Chrome installed on individuals’ computers–opted into the Privacy Sandbox solutions. Traffic volume was also impacted by a lack of publishers who implemented the necessary changes to participate in the Privacy Sandbox at the time we ran the test. On some exchanges, we only saw a handful of publishers testing the Privacy Sandbox when we started testing; this barely counts as a drop in the bucket compared to the use of third-party cookies today. Fortunately, as testing continues, we see more and more adoption. Today, outside of Google, we see about 5% of publishers testing. While this is a good sign, supply side adoption is still far from an ideal state. 

It is likely that supply side publisher testing is also self-selected, which means those participating are not random and thus cannot be assumed to be representative of the entire supply side population. This makes it harder to draw meaningful conclusions from current testing.

Takeaway: Supply side adoption of Privacy Sandbox solutions does not match or even approximate the scale or mix currently run on third-party cookies. Without representative supply adoption, any tests run will be limited in their ability to measure future impact of third-party cookie deprecation. 

Demand Side (Publisher)

Similar to supply side testing, advertiser opt-in to testing has been largely self-selective. Some DSPs chose to test the Privacy Sandbox, while others have taken a wait-and-see approach. This self-selection leads to bias in the test as there are incentives and reasons to opt in or out of testing, resulting in a non-representative test group. At NextRoll, we designed our test to include all of our advertisers, which allows us a wide testing group. However, we have reason to believe this is not the case with other testing DSPs where advertisers are instead asked to opt in based on their expressed interest.  Additionally, not all DSPs are participating in Privacy Sandbox testing, which further limits the demand side test group. This generally low adoption leads to lower bid density, meaning the auction economics do not represent what we expect to see in the future at full scale. This means the testing data cannot be trusted to fully represent what we expect to see with representative adoption. 

This problem in demand side adoption is illustrated by the work Sincera has done to help track and validate Privacy Sandbox adoption:

Takeaway: Demand side adoption of Privacy Sandbox solutions does not match or even approximate the scale or mix of DSPs currently running on third-party cookies. Without representative demand adoption, any tests will be limited in their ability to measure future impact of third-party cookie deprecation. 

Consumer Side (Audiences)

Ending on a positive note, in January of this year, 1% of browsers were randomly moved off of third-party cookies. This was done to create browser testing cohorts that operated in a truly cookieless environment, which includes four tranches of 0.25% each, three of which operate on the Privacy Sandbox, and one that serves as a control group. As a result, this design does provide representative test groups for the industry to work with. 

Takeaway: On the consumer side, we have a near perfect test scenario and groups. 

The Implications of Low and Non-representative Adoption 

The Privacy Sandbox introduces a series of solutions meant to help replace third-party cookies and shape the future digital advertising marketplace. To assess these solutions, we need a representative market to what we have today. Given the nature of the selective sampling seen for both the supply and demand sides, we’ve ended up with a test market running auctions that we cannot trust to resemble the future economics of the Privacy Sandbox. Current programmatic advertising involves participation from all players. The current state of Privacy Sandbox testing is missing too many market participants for any of the data seen to be reliably indicative of the market as a whole. When third-party cookies are deprecated, programmatic advertising will again involve all market players. But, does the current state of adoption represent that today?

Looking at both the supply and demand sides of digital advertising, we can come to the conclusion that neither side has adopted Privacy Sandbox solutions at representative scale yet. The current testing market place is inherently non-representative of the current state and thus we must conclude it is non-representative of the future. This presents a limit to what conclusions can be made from tests running today.

What We Can Test Today

As previously stated, NextRoll is actively testing the Privacy Sandbox APIs; we have been delivering impressions and measuring results since mid 2023. While we are eager to understand how ad performance will be impacted, our first tests focused on the core aspects of these solutions. To support the Privacy Sandbox, we had to refactor our entire core bidding, segmentation, and ad serving technology. 

Our current testing, by necessity, is an end-to-end volume pipes test. We consider this test successful in that we were able to serve ads at volume using these new systems. Following this systems test, there are steps we need to take to continue to ramp up for a wider performance-driven test. We are actively working to test, implement, and improve the ideas our team is dreaming up to future-proof our current capabilities. Our test launched mid-March and we will be complete by the end of May. We will be sharing our results with the UK CMA and publicly in June. 

Implications for Advertiser-Specific Testing

Given NextRoll’s plans to test the Privacy Sandbox solutions, a natural question from an advertiser is: How will this affect my campaign performance?  Fortunately, we do not expect testing to negatively impact any of our advertisers. Given the low traffic volume available for testing, we expect the impact to be little more than a rounding error and likely not observable to individual advertisers. Earlier this year, we announced that we are running an industry-wide test across our entire advertiser base. Budget dedicated to this test is capped at 1% of ad spend across advertisers and fully funded by NextRoll. While 1% of ad spend doesn’t really impact campaign performance, taken on the whole, we hope to gain some high-level directional learnings that will enable us to improve our approaches. As we run these tests, we will be proactive in sharing our findings. 

We also plan to share our results with the UK Competition and Markets Authority (CMA). The UK CMA is pushing for testing to occur using the Privacy Sandbox prior to full third-party cookie deprecation. They want to ensure these solutions provide a path forward for cookieless advertising to protect advertisers, publishers, and the technology vendors who support both sides. We are excited to share our test results and findings with the CMA to help them make the best decision. 

Conclusion 

To summarize what we can expect from testing, it is useful to break down what we expect to see into micro (advertiser-specific) and macro (industry as a whole) results.

Micro 

Full advertising-specific testing (testing that helps illustrate how deprecating third-party cookies will impact campaign effectiveness for any given brand) is limited by the current level and representation of market adoption. 

So, what does performance look like when we see adoption? Unfortunately, we don’t have a crystal ball that can tell us the future. Rather, we can continue to grow our adoption of the Privacy Sandbox by integrating with more of our supply partners, as well as monitor market adoption. As we see opportunities to test, we will continue to publicly share our plans and results. 

Macro

At the industry level, we are confident that our planned testing will provide a view of what the Privacy Sandbox will be able to solve. As a preview, we already see very few use cases today that won’t have some level of support. However, the current level of adoption does not accurately reflect the larger industry now or what it will look like in the future post third-party cookie deprecation. Therefore any performance results seen today by any Privacy Sandbox testers will be extremely limited in providing a view of what future performance might look like.

With this being the case, we see the most recent delay in third-party cookie depreciation as a strong opportunity for the industry to design a better test. While the current random 1% of cookieless browsers is a fine start, we also need randomized test groups on supply and demand sides along with additional features and functions to help solve for the current gaps seen due to low adoption.

While we are skeptical about the current state of testing, we are actually quite bullish on the future of the Privacy Sandbox. While adoption in testing is low, everyone we work with in testing is committed to improving the sandbox. We not only hear this in our conversations with our ad tech partners, we also observe continuous improvement across the ecosystem and our confidence in a privacy forward digital advertising future continues to grow. 

This article was written by Rob Myers, Senior Product Manager of Advertising, at NextRoll.