Michael Kaminsky explains how to build a real incrementality testing program: pick channels by spend and impact, run tests over multiple quarters with pre-set decision rules, and measure true incremental value and diminishing returns. Tests must integrate smoothly into operations, not disrupt them.
Chapters
You talked a little bit about this idea of geographic experimentation, incrementality tests, how you think about what does good look like for this type of an incrementality testing program? Any sort of like highlights of these are the things that make a best-in-class program, or these are the things that you really need to avoid or be careful about if you're trying to roll this out in your organization?
The word program is probably the important one. It's a process. It's not just one-off tests. I think identifying where you need to test and when. So there are certain channels where you will learn benefit far more by testing than others. And I think you can identify those fairly easily. And you could say, well, if we're spending this much here, we really need to know if this signal that we have right now is effective for us. I think it is about laying out a program and saying, this is what we'll do. We'll have, let's say, 4 quarters. We will test these channels here. These are our priorities. We will retest them if needed at these points. And just laying it out in that way and then identifying the questions that you need to answer. So one of the questions is, what's the true incremental value of this channel? Another question is, what are my diminishing return curves and do they change over time? Another question is, there's more than one way to use a channel. What are the optimized ways of using this channel? If I try this program versus this program, how does it look? Laying out those different questions, first of all, and then saying, this is how we can execute against them. With every one of those tests, lay out the decision that would be associated to it before you get the results. So you say, look, it's gonna say one of three things. It's going to say better, the same, worse. What are we going to do in each one of those situations? And it may be more of a spectrum than that, in fairness. We can agree in advance what decisions we'll make based off those results, rather than get to the results and then kind of equivocate and say, well, it could have been this, it could have been that. I think that's how a great program would work. The negative aspects, I don't want it to be too disruptive. I don't want to hamstring the marketing operations just so I get a signal of measurement, because that's not great partnership either. If you can do it in a way that doesn't really negatively affect the day-to-day operations, then that's where you'll get the most agreement in terms of running these tests.
