Michael Kaminsky breaks down four critical mistakes analysts make with spreadsheets when measuring campaign lift: skipping control groups, picking test markets by gut feeling instead of matching them algorithmically, running underpowered experiments, and ignoring confidence intervals. A single point estimate will mislead you. Use a checklist: control group, power analysis, incremental ROI, uncertainty ranges.
Chapters
A reader recently asked me, "Why should I use a geotesting tool when I can just do the analysis in Google Sheets?" And look, I get it. Eyeballing the data in a spreadsheet, it's a good first step. But things go sideways when you start using them as the analytical engine for answering questions like, "Did this campaign actually cause more revenue?" Here are the 4 most common mistakes that happen in spreadsheet-based Geolift tests and could be costing you real money. This is the most common error and probably the most dangerous one, because looking at before versus after, it just feels right. Here's the scenario. You're running a ride-hailing company. You launch a new ad campaign in Chicago, and 3 weeks later you open your spreadsheet, you look at rider volume, you calculate the percent change, and you present the result. The campaign drove a 12% lift. Sounds great, right? But what if rider volume in Boston, a city where you didn't run any ads, also went up 10% over the same period? Suddenly that 12% lift doesn't look as impressive. Maybe it was the weather, or maybe a competitor raised prices across the country and pushed riders your way. This is Panel A, the world as your spreadsheet presents it. It looks like the campaign really worked. Now look at Panel B. Same data, but we've added a control group—markets where the campaign didn't run. The control group grew at the same rate. The effect disappears. What looked like a campaign-driven lift was just just an underlying trend. Now here's the flip side. This second figure shows what happens when the campaign does work, but the spreadsheet still misleads you. The treated group grew, sure, but so did the control group, just less. The naive before and after view overstates the effect by attributing the entire increase to the campaign. Panel B separates the signal from the noise. The true treatment effect is the additional growth beyond what the control experienced. Without a control group, you're crediting your campaign for growth that would've happened anyway, and making budget decisions on that false credit. And when you divide the inflated revenue by your ad spend, the ROI number your spreadsheet spits out is just as inflated. The 4x ROI can easily shrink to 1.2x once you subtract the sales that would've happened without the campaign. "Let's run the test in California and just compare it to the rest of the country," says the agency managing your marketing spend. This sounds reasonable and is a good start, California's a big market, it's easy to measure, your team already has good data there. The problem is that California differs from the rest of the US in consumer behavior, media costs, demographics, seasonal patterns in ways that are going to dwarf most advertising campaign effects. Your test result ends up measuring the difference between California and everywhere else, not necessarily the difference between running ads and not running them. Proffer geotesting tools select markets algorithmically, optimizing for pre-test similarity, choosing tests and control groups that tracked each other closely before the experiment. So any divergence during or after the experiment is more likely to reflect the actual campaign effect. A human picking markets on instinct can't necessarily do this very well. And with that much noise baked in, only a massive lift will stand out. You might scale a campaign nationally based on something that worked in California, but just because it is California. So before running any test, you need to ask yourself, is this test actually sensitive enough to detect the effect size that I would realistically expect or you would realistically expect? The idea is that given your expected lift, how truly effective you think the channel is, the variance across your markets and the duration of the test, you can calculate whether your experiment has a realistic chance of detecting an effect if one really truly exists. This analysis is called power analysis. Think of it this way. You are trying to read a street sign from a mile away with weak binoculars. The sign is there, but you'll never be able to see it with your weak binoculars. Your conclusion shouldn't be, there's no sign. The conclusion should be, well, I just don't have powerful enough binoculars to be able to see the sign from so far away. The same thing happens with underpowered experiments. If your expected lift is 3%, but your test can only detect effects of 15% or larger, you're basically guaranteed to get an inconclusive result. You'll spend weeks of effort and budget and then conclude that the campaign didn't work when the real conclusion is that the test was blind. Here's the danger. A proper geotesting tool will tell you not to run the test if the power is insufficient. It'll say, given your markets and your expected effect size and your test duration, this experiment can't give you a reliable answer. So don't waste your time. A spreadsheet will never tell you that. It'll happily compute a result no matter how meaningless the setup. So you ran your test, you got a number. Great. But how confident are you in that number? Is the true effect somewhere between 0.5% and 9.5% or tightly bounded between 4.8% and 5.2%. Both look identical in a single cell in a spreadsheet. They both show 5%, but they're radically different situations for making a budget decision. And here's where teams really get it wrong. When the uncertainty range is wide, they often just ignore it. The intervals are too wide, so I just look at the midpoint. That's exactly backwards. The The width of the interval is the information. If your incremental CPA ranges from $40 to $900, well, you can't really make a budget decision based on $150. The test is telling you, "I don't have a clear answer yet." A proper geotesting tool produces a full distribution of plausible outcomes. A spreadsheet just gives you one number and hides all of the uncertainty behind it. I want to be clear, the spreadsheet isn't the enemy. It's great for tracking, organizing, reporting, and often a good first pass. But the problem is using it as the analytical engine for causal inference, for answering, did this campaign actually cause additional revenue enough for us to make a different budget decision? That question requires a counterfactual, proper market selection, power analysis, uncertainty quantification, and a clean separation between revenue you caused and revenue that was coming anyway. No spreadsheet formula gives you all of that. So here's what I'd suggest. Bring these 4 questions to your next meeting. Does our analysis include a control group? Did we run a power analysis before launching the test? Are we reporting incremental ROI or just standard ROI? And do we have uncertainty ranges for our estimates? If the answer to any of these is no or I don't know, it might be time to lead the spreadsheet behind. Hopefully that was helpful, y'all. If you've run into any of these in your own work, drop it in the comments. I'd love to hear about it, and I'd love to hear which of these was the hardest for you.
