Why companies avoid A/B tests even when data suggests better ad spending. Decision-makers fear experiments will expose past mistakes and damage their reputation. Running randomized media tests requires complex coordination with ad platforms, making it operationally hard for individual advertisers to execute alone.
Chapters
You have a lot of great papers on the optimal ways to run A/B tests. Let's say you have multiple times you can run a test or just one time and you want to figure out what the sample size should be. Why do you think companies are so averse to running experiments? One of the things we struggle with is we'll say, hey, according to the model, you should be spending a lot more on YouTube or a lot more on podcasts and a lot less on this channel. The payoffs could be very large, yet sometimes we hit this organizational inertia. There's this risk aversion. Is this something that you've experienced? How do you think about that?
I spent 2 or 3 years as a cheerleader for experiments saying, if you want to know the answers to these questions, you really should be running experiments. There's a couple things. One is what economists call an agency problem. The people deciding how much to spend on ads have made a career being the person who decides how much to spend on ads. It would be very bad for their career if they were to find out that they'd been spending the wrong amount. That makes the decision makers reluctant to find out the real answer because they've convinced everyone around them that they know the answer. The execution is actually incredibly complicated, especially in the media space, because you need your media provider to randomize the advertising for you. So there's a lot of communication overhead in getting that to happen. I actually think it's probably more likely that the platforms will either decide to do this or not because they can build one system and use it for multiple clients. But from a client sitting on the outside, it is tough to execute that randomized experiment properly.
