Michael Kaminsky asks how to run experiments without disrupting marketing. The key is choosing non-intrusive measurement, deciding if an experiment needs to repeat, and pre-aligning with marketing and finance on objectives, budgets, and timelines. Experiments pull users away from marketing's reach, so teams need to agree upfront on what trade-offs matter most.
Chapters
Can you give some like tactical tips? One of the ways that I conceptualize of this problem is like, in order to make change, you have to work with the marketing team, but in order to make that universal across the business, the finance team still has to trust you. And so you have to achieve both of those goals. I'm curious if you have like tactical tips on like, oh, this works really well for achieving that, or, ooh, I've seen this go wrong if this happens, anything along those lines that you can share.
One thing that helps with the relationship with marketing is to not get in the way. Testing almost by definition has an opportunity cost associated with it. It does not help people achieve their goals in the short term because you're taking people out of the pool of people who can be impacted. And so, there's a negative impact that you will see through certain types of measurement. And so then the question for me is, how do I balance that need to provide measurement because we know there's long-term impact value that we get from understanding things, but also balance that against the needs of a company who have to hit goals in a certain in order to kind of achieve certain things. So I think one of the questions is how can you work with the marketing team to make sure that it's as non-disruptive as possible? There are certain types of measurement techniques that are less disruptive than others, and they may answer your question in a way that's sufficient. The other thing is, are you going to do this again? Do you need to measure this in a way that's repeatable in the future, or is this just a one-off? If you're doing a product launch that will never happen again, is there a value in keeping a holdout when actually you want a more splashier impact? So those kind of questions, I think, really important to get aligned with marketing team on. And then finance obviously have their own questions as well, and they come at things from a different viewpoint. But it's ultimately the same question, which is, where can I spend my money most effectively? And I think pre-aligning on all of those things with all of those different teams is the kind of the key is to say, we all have one common goal here. There's a bucket of money, we want it to be as impactful as possible. Here's all the options that we have to do that. What are the timelines? What are the restrictions? Do we need it to do X, Y, and Z? Those things, if we lay them out beforehand and all agree to them, I think help with this kind of broader narrative.
