22.5 A/B Testing Caution

Braun and Schwartz (2025)

• Marketers use online advertising platforms to test user responses to different ad content.

• Platforms’ experimentation tools deliver ads to different, dynamically optimized mixes of users, leading to nonrandom exposure.

• This “divergent delivery” confounds ad content effects with algorithmic targeting, skewing A/B test results.

• Algorithmic targeting, user heterogeneity, and data aggregation distort the magnitude and direction of A/B test results.

• Platforms have little incentive to help experimenters isolate ad content effects from proprietary targeting mechanisms.

References

Braun, Michael, and Eric M. Schwartz. 2025. “Where a/b Testing Goes Wrong: How Divergent Delivery Affects What Online Experiments Cannot (and Can) Tell You about How Customers Respond to Advertising.” Journal of Marketing 0 (0). https://doi.org/10.1177/00222429241275886.