Incrementality Testing: How to Actually Measure Marketing Effectiveness (2026)

Years ago, eBay did something most brands would never consider. They turned off their brand search ads, the ones that show up when you search eBay shoes or eBay electronics. By conventional measurement, these ads looked wildly profitable. So what happened when eBay turned them off? Well, almost nothing. Traffic and sales stayed flat. The people searching for eBay cameras already wanted to go to eBay. These ads weren't convincing anyone to buy. They were just intercepting the customers eBay already had, and Google was getting paid for it. So eBay cut the budget on the channel, and that one change saved them millions of dollars per year. Now, here's the uncomfortable question: what would happen if you ran the same test on your best-performing channel? Would revenue fall off a cliff because it's truly performing, or would it stay flat? Most marketers don't actually know the answer. And that's a problem because if you can't answer that question, you might be lighting money on fire and calling it a win. Here's something every marketer has experienced. You're reviewing last quarter's performance and you pull up your dashboards. Meta says it drove $200,000 in revenue. Google says $400,000. TikTok says $100,000. You add those up and you get $700,000. That's great. The only problem is that your business only made $450,000 total. The math doesn't work. This isn't a bug. It's just how in-platform attribution works. Ad platforms use what's called deterministic attribution. When someone converts, buys your product, subscribes, downloads your app, whatever, the platform looks backward and asks, did this person see or click on one of our ads prior to purchase? If yes, they count it as a win. But converted after seeing an ad is not the same thing as converted because of an ad. Even more sophisticated models like multi-touch attribution, which spread credit across multiple ads instead of giving it all to the last click, still have this same blind spot. They're tracking who saw what, not what actually caused the sale. Some of those people were going to buy anyway. Maybe they heard about you from a friend. Maybe they saw you on a podcast 2 weeks ago. Maybe they typed your URL directly into the browser and your retargeting ad happened to hit them an hour before they checked out. Some of those people were likely driven by multiple different ads, and each ad platform wants to take credit for the whole conversion. In-platform attribution counts all of that as a win for paid ads. But is it? Let's put numbers to this. You spend $10,000 on Meta retargeting ads. The platform reports that people who saw those ads bought $25,000 worth of your product. At a 60% margin, that's $15,000 in gross profit on $10,000 in spend. $5,000 of pure profit. Looks like a no-brainer. We'll do that all day. But what if half of those people would have bought anyway? Without ever seeing the meta retargeting ad. Now your incremental revenue is $12,500 at a 60% margin. That's $7,500 in gross profit on a $10,000 investment. The platform showed you a winner, but in reality, that campaign just lost you $2,500. This is obviously hypothetical, but it happens constantly. Ad platforms are incentivized to take credit for as many conversions as possible, and I don't blame them, but that doesn't mean that those ad platforms actually caused those conversions. This often gets worse over time. When your brand is small and nobody knows who you are, attribution and incrementality are roughly the same thing. You turn on ads, sales spike. It's pretty clear that the ads really are working. But as the brand scales, everything gets noisier. People hear about your product from friends. You get press coverage. You're running ads on 5 different channels. Word of mouth really starts to kick in. Now it's no longer obvious what's actually driving purchases. And your attribution dashboards, which looked clean and confident when you were small, start to mislead you. You pour money into channels that look efficient but aren't really working while starving the channels that are actually driving growth but don't get the attribution credit. So how do you figure out what's real? The answer is incrementality, and it answers a completely different question. Attribution asks, did this person convert after seeing my ad? Incrementality asks, would this person have converted anyway even if they never saw my ad? That second question is what actually matters because the answer tells you whether your marketing caused the sale or just showed up on the path to conversion. Imagine two parallel universes. In Universe A, you ran an ad and the customer bought. In Universe B, you didn't run the ad at all. In Universe B, did the customer still buy? If they made a purchase in both universes, it means that your ad didn't cause anything. It just happened to be there. If they only bought in Universe A, that means that your ad actually drove that incremental impact. The obvious problem is that you can't actually observe both universes at the same time. This is called the counterfactual problem, and it's the fundamental challenge of marketing measurement. Well, how do you solve it? The same way the FDA does. When the FDA, the Food and Drug Administration in the US, tests whether a new drug works or not, they don't just give it to sick people and see who gets better. Some people get better on their own, so that wouldn't prove the drug actually did anything. Instead, they run a controlled experiment. One group gets the drug, another group gets a placebo, same conditions, same time period. The difference in outcomes between those two groups tells us the drug's true causal effect. So it's a measure of causation, not just correlation. Marketing measurement should work exactly the same way. You need a test group that sees your ads and a control group that doesn't. The difference in outcomes between those groups is your incremental lift, the thing your marketing actually caused. There are a few ways to run these experiments in practice, and they're not all equal. Some ad platforms offer built-in tests. Meta calls theirs Conversion Lift, and Google has similar tools. These are true randomized controlled trials. The platform randomly holds back ads from a subset of users and measures the difference. When done correctly, these experiments are scientifically extremely rigorous, but there's obviously a catch. You're trusting the platform to grade its own homework. On the other end of the spectrum, you have go dark tests. These are very simple. You just turn a channel completely off and see what happens. There doesn't need to be a lot of fancy statistics, just a before and after comparison, but now you're comparing different time periods. Not different audiences at the same time. So seasonality, promotions, and 100 other variables can contaminate your results. The option in the middle, and the one we think is practical for most brands, is geographic experimentation. Instead of randomizing individual users, you randomize by geography. You increase spend in some regions, you hold other regions constant as your control, and then you compare the outcomes. Your control regions tell you what would have happened in your test regions if you hadn't changed anything. That's your counterfactual. That's your Universe B. Geographic tests work across most channels. They don't require platform cooperation, and you can run them yourself. The challenge is that your regions won't be perfectly identical. Austin, Texas, and Denver, Colorado don't have the same population, the same baseline sales, or the same seasonality patterns. You can't just compare raw numbers between them. The traditional approach to doing this type of measurement in marketing is called matched market testing. The idea is straightforward: you find two cities or two regions that look similar—similar population, similar income levels, similar baseline sales—and you use one as your test market and one as your control. If Austin and Portland have similar demographics and buying patterns, you increase spend in Austin, hold Portland steady, and then compare. It's intuitive and it's been used in marketing for decades, but in practice it's really hard to find a clean match. Even cities that look similar on paper have different local competitors, different seasonality patterns, and different cultural dynamics that can affect buying behavior. And any of these differences can show up in your results and make it hard to tell what was caused by your marketing and what was just Austin being different from Portland. You end up needing a better way of constructing that control group. And this is where statistical methods, specifically synthetic controls, can come in and be helpful. In synthetic controls, an algorithm looks at your historical data and creates a weighted blend of your control regions that closely matches the behavior of your test regions before the experiment started. That synthetic blend becomes your counterfactual—what would have happened in your test markets if you hadn't changed anything. So the idea is that there might not be a perfect matched market for New Orleans in the United States. There just really aren't any cities like New Orleans. But we can create a blend of cities that allow us to create a synthetic New Orleans that is a better match for New Orleans than any other natural city might be. We could imagine that blend could be a combination of Houston, Texas, and Charleston, South Carolina, and down in Savannah, Georgia, where the blend is a better match than any of those cities individually could be. It sounds complicated, but tools exist to handle all of this for you. You don't need a PhD to run one of these tests anymore. Let me show you what it actually looks like. Let's say you're an e-commerce brand. You're spending $50,000 a month on Meta prospecting campaigns. The platform says you're generating a 4.7x return on ad spend and your marketing team is patting themselves on the back. But your CFO keeps asking the same question: would we have gotten these sales anyway? Your organic traffic is strong, your brand awareness is growing, and you've got a nagging feeling that Meta is taking credit for customers who were already coming to you. So you decide to test it. Here's exactly what that looks like. First, you need historical data, revenue by geography and by day, at least 90 days, ideally a full year. Most companies have this sitting in their marketing data warehouse or analytics platform. If you can pull a report from Shopify or Google Analytics 4 that shows daily revenue by state or DMA, you have what you need. Then you upload that data into an experimentation tool, in this case, Geolift by Recast. The tool charts your data by geography so you can do a quick sanity check. Are there any obvious anomalies, missing data? If it looks clean, you move on. Now you design the experiment. You tell the tool a few things, that you're planning to increase spend, that you're measuring revenue, roughly what you think the channel ROI might be, that's incremental ROI, very important, how long you want to run the test, and how much budget you're willing to put behind it. The tool then runs simulations across your geographies. It's looking for regions where historically when one goes up, the other tends to go up too, meaning that they're good candidates for comparison. It spits out the candidate test configurations ranked by statistical quality. You can click between rows to see how well the synthetic control matches each set of test markets. The tighter the historical fit, the closer those two lines track before the experiment starts, the more confidence you'll have in the results. The tool might tell you something like, If you increase spend by $15,000 in Austin, Boston, and Denver for 4 weeks, you have an 80% chance of detecting a statistically significant effect, assuming the channel has at least a 3x true incremental ROI. That last part matters. The tool is telling you upfront what this test can and cannot detect. If the true effect is smaller than expected, you might not be able to see it, or the uncertainty ranges just might be too wide to be useful. Next, you pull the trigger. Your media team increases meta-prospecting spend in Austin, Boston, and Denver by $15,000 over 4 weeks, while everything else stays the same. No new promotions in those markets, no other spend changes, a clean test. You let 4 weeks pass. Now we can analyze our results. You're going to upload the post-test data, tell the tool which cities were in the test and what dates the experiment actually ran for, and then hit the analyze button. Here's what comes back. The tool compares actual revenue in your test cities against what the synthetic control predicted would have happened without the spend increase. It shows you this visually. The gray line is your counterfactual, what would have happened. The green is what actually did happen. The gap between them is your incremental lift. And here's the result that matters. The table tells you the estimated incremental revenue in dollars, the ROI on that incremental spend, a confidence interval, and whether the result is statistically significant. Most people make a mistake here. They look at the point estimate and stop. Oh, we got a 3x ROI. That's great. But that number alone doesn't tell you enough. The confidence interval tells you the range where the true lift likely falls. Maybe it's 2.5x to 3.5x. If the bottom of that range still shows a positive ROI, a positive return for your business, you can feel confident the channel is working. But if the bottom of the range is negative or breakeven, even if the point estimate looks fine, you should be cautious. That channel might not be pulling its weight. Always plan around the conservative end of the range. That's what separates rigorous measurement from wishful thinking. So if you've never run an incrementality experiment before, where should you start? The best first test is the channel sitting at the intersection of high spend and high uncertainty. Something that looks great in your attribution dashboards, but you're not totally sure is actually incremental. A few channels tend to live in that quadrant. Retargeting is a classic one. It often shows incredible ROAS with attribution methods because it's great at finding people who are already deep in the purchase funnel. But being good at identifying people who are already going to buy is exactly the situation where over-crediting and attribution happen. Branded search is another. You're paying for clicks from people who already searched for your brand by name. eBay demonstrated this 15 years ago, and most brands still haven't tested it for themselves. These are all high-spend, high-doubt candidates, but there's something important to flag. Incrementality testing doesn't always reveal that a channel is wasting money. Sometimes it reveals the opposite. Upper funnel channels, things like connected TV, YouTube, podcast ads, out-of-home advertising, they tend to get undercredited in attribution because they influence purchases that happen days or weeks later through a different channel. Attribution gives the credit to the last click channel, or one of the channels that actually has clicks, but when you run a geo test on an upper funnel channel and see that revenue drops in the markets where you pulled the spend, now you have evidence that the channel is actually more valuable than your dashboards might be suggesting. This finding is equally or more important. Incrementality testing isn't just about identifying waste, it's about getting closer to causal truth. Sometimes the truth is that you should spend more, not less. So let's bring it back to where we started. Attribution tells you who converted after seeing an ad. Incrementality tells you who converted because of the ad. They are measuring fundamentally different things, and confusing them can cost your business real money. One of the best ways to know whether a channel is actually driving growth is to run an experiment. Geographic-based incrementality tests are the practical option for most brands. They work across almost every channel. They don't require you to trust an ad platform to grade its own homework, and you can run them yourself with tools like Geolift by Recast. Pick one channel, the one you're most uncertain about or the one that's most critical for your plan, get your geo-level data together and run the test. You might find that your marketing is working exactly as well as you thought, or you might have an eBay moment, and that moment could save you more money than anything else you do with your marketing program this year.