Michael Kaminsky argues that Google Sheets can't tell you if your marketing test is actually powerful enough to detect results. Underpowered tests produce inconclusive data, and spreadsheets won't warn you. A proper tool does power analysis and flags when you can't reliably measure the lift you're after.
Chapters
Why should I use a geotesting tool when I can just do the analysis in Google Sheets? And look, I get it. It's a good first step, but things go sideways when you start using them as the analytical engine for answering questions like, did this campaign actually cause more revenue? A human picking markets on instinct can't necessarily do this very well. You might scale a campaign nationally based on something that worked in California, but just because it is California. You can calculate whether your experiment has a realistic chance of detecting an effect. This analysis is called power analysis. Think of it this way. This way. You are trying to read a street sign from a mile away with weak binoculars. The sign is there, but you'll never be able to see it with your weak binoculars. Your conclusion shouldn't be, "There's no sign." The conclusion should be, "Well, I just don't have powerful enough binoculars to be able to see the sign from so far away." The same thing happens with underpowered experiments. If your expected lift is 3%, but your test can only detect effects of 15% or larger, you're basically guaranteed to get an inconclusive result. You'll spend weeks of effort and budget and then conclude that the campaign didn't work when the real conclusion is that the test was blind. Here's the danger. A proper geotesting tool will tell you not to run the test if the power is insufficient. It'll say, "Given your markets and your expected effect size and your test duration, this experiment can't give you a reliable answer, so don't waste your time." A spreadsheet will never tell you that. It'll happily compute a result no matter how meaningless the setup.
