Michael Kaminsky explains how to test whether your marketing mix model actually works: simulate data with known true parameters, fit candidate models to it, then check if you recover the right answer. When models are too complex for your data, different parameters can produce identical results, making it impossible to know which estimates are real.
Chapters
'cause I found that in industry, this sort of technique of using simulation to think about identifiability isn't actually that well known or used as much as I think it could be. Could you talk about the process of simulating data, recovering parameters, and then using that to think about identifiability? Like what does that look like in practice?
Sort of, in reality, we never know what's the true model, right? There's this famous box quote, you know, all models are wrong, some are useful, right? We never know what's the true model. That's not going to be the case if we are simulating data. So you can imagine sort of simulation as kind of like this best case scenario that's just never gonna happen where we know this is the true model. If we're the marketing mix gods, we know this is the correct response function and the correct data generating process. Under this best case world, we can generate a dataset or datasets changing sort of the parameters. And again, the benefit here is going to be that we know exactly the process that generated that data. Then we can kind of pretend like we didn't, right? We can pretend that we didn't know what was the truth, which is going to be reality when we go to fit some data, some models. We don't know what's the truth. We can pretend we don't know the truth, and then we can fit the models that we're thinking about, whether that be the correct model or another model we're thinking about. We can then see what comes out. We hope what comes out is that if we test other models, they don't fit as well as the true model. We hope that when we look at the parameters that we sort of recovered, that those parameters are similar to the parameters that generated the data. Because again, in this case, we do know what, what generated data. We're just pretending we don't. In that case, we have a well, at least statistically, a well-identified model, right? When we take that data, we are able to recover a unique model from that kind of data. Issues come up when that doesn't happen. For instance, in our paper, what we do is change to a completely different type of model. We fit it and then we say, wow, actually by all the standard metrics, you could never tell these two models apart. Or in the sort of parametric identification issue, you might say, oh, the data was generated by a hill function that has parameters 1, 2, and 3. And now we see that this says the parameters are 100, 25, and 6. Then you're like, okay, well, clearly something's up here. That will often happen, for instance, if the model is too complex to support the types of data that you have. So you're generating simple data, which I should stress, Markov mix models are often built on simple data, and then you're fitting something fairly complex to it. Different parameters can often generate very similar-looking datasets when the data is really simple. In some cases, you don't care, but when you're thinking about doing optimization or intervention, figuring out what's the right model does matter.
