Michael Kaminsky explains identifiability in marketing measurement: whether data can uniquely pin down model parameters or prove causal claims. Noisy marketing data make it hard to distinguish between parameter combinations that fit equally well, and that matters because marketers need to know how a dollar of spend actually moves revenue, not just fit a curve.
Chapters
You talk a little bit about identifiability. Can you give a good explanation of this topic? What does identifiability mean in the statistical context? What are we trying to do when we're talking about identifiability of different parameters or different modeling approaches?
Part of the confusion is that people will use this word to mean different things, right? I think fundamentally the core of the word identifiability is talking about, given this data, can I get the correct answer? In our context, what that literally means is, given the marketing mix data that we have, can we figure out which set of assumptions is the right assumption without just assuming it? So can the data tell us something? Can the data say, yes, this is the right set of assumptions? In other contexts, people might use this word to mean, for instance, I have one class of model. I'm thinking of the models that I care about, these nonlinear models where we're modeling X and Y with some kind of Hill function. There would be a few parameters that determine how does spending relate to revenue in terms of those same ideas that I was just talking about a second ago, like how much money do I have to spend before I see some returns, how quickly do those returns level off. In many cases, those kinds of parameters might also be what we call weakly or not identified. There are many different values of those parameters that would give the same relationship between how spending and revenue work. Moderately complex function, and different values of the parameters can lead to similar shapes of that function. When you combine this with the idea that marketing data, and especially in data, is often pretty noisy, we're then going to be saying, okay, figure out these nuanced differences given noisy data, and you're just not going to be able to do it. You're going to find that many different values will give you roughly the same thing. There's a third type, which is about causality and causation, right? So often when we're talking about marketing mix models, what we really care about is if I'm going to increase spend on this channel by $1, how much is revenue going to change? And that is fundamentally a causal question, right? I'm saying I'm going to do something, I'm going to intervene and spend more money, and then I want to know how is revenue going to respond to that. Is my increase of $100,000 going to cause revenue to change? If it's not causing, then we don't care, We only care about what our money's causing to happen in the world. And so in many cases, people use this word identification also mean, can we figure out a causal story from the data that we have?
