Lift Testing at Harry’s | Michael Kaminsky (Recast, Harry's)

We started doing some experiments like lift test, like match market geo holdout lift test style experiments in order to get a read on. Okay, you know, when we increase spend in, in these marketing channels in these certain GEOs, can we get a read on how that impacts our offline sales? And so that was sort of dipping our toes in the water of trying to understand this phenomenon. And what we, what we learned there was that like, yes, we can get a read on it. Yes, these marketing dollars do drive in store sales, but it, we're going to need in general, a more sophisticated method for thinking about how to bring all of those pieces together. Geographic holdup tests. They're generally what a statistician would call underpowered, which is to say that they're not very sensitive. It's very difficult to tell, like is the ROI 1x or 1.5x or 1.75x? Right. It's very difficult to differentiate between those because the underlying data is so noisy. And so you can see that, hey, there's some lift, but there's wide confidence bounds around it. And so we're not really sure what the true number is. Plus other problems of not being able to measure them consistently. Right. And for a business like Harry is, you know, the holiday season is very different than the summer season, is very different than the spring season. And so, you know, if we ran a test in the spring, can we apply those results to the holiday? We had all of those problems. And so we knew that we wanted to think about more sophisticated ways of being able to bring that information together and then make smart bets about where to allocate our marketing dollars in order to drive the business forward. And so that was where we started exploring ideas of, you know, doing lift tests, combining them with marketing mix modeling, and then trying to have some more holistic way of driving the business forward across offline marketing channels as well as offline sales channels.

Did that analysis kind of uncover any gaps in the media mix that, for example, you were underspending in different offline channels or overspending in different areas?

Oh yeah, absolutely. And I mean, this is a thing that everybody sort of knew was the case, right? That if we optimize our marketing mix to the channels that are most optimized for e commerce sales. So in general, we're talking about like bottom of funnel digital channels that are very easy to track.

So your Facebook, Google.

Facebook, Google, Right. Like those two channels were great for e commerce businesses and they're optimized for that, but they're probably not actually optimized for driving sales at Target and Walmart. And so it became very clear very quickly that actually, like, we were sort of upside down in terms of how we were optimizing our marketing spending because we were so optimized towards the things that were best at driving sales in the e commerce site, but not necessarily things that were best at driving sales across the whole business. And again, I mean, I don't remember the actual details from Harry's, but across all of the different businesses that I've worked with over the last couple of years, you know, the things that are obvious start to jump out, right? More offline channels like radio are often not as good at driving e commerce sales, but a lot better at driving sales in Target and Walmart because people listen to the radio on the way to the store and their broader reach channels, where you have broader distribution, it makes more sense. And so there are a lot more. Once companies have more distribution outside of just e commerce and they have more ways to sell to customers, opening up some of those broader reach channels can actually be really, really impactful.

Sure. Even things like billboard ads, right?

Billboard ads, other out of home, all of the stuff that again, is, I think, really hard to make work if you're a pure e commerce business. Not impossible. Like I've seen e commerce and digital businesses make out of home work very well. But those sorts of things become, I think, easier to make work. And you know, you can tell a story in your head about why again, you know, billboard, it's a, people are in the car, they're going to Walmart, that's a great time to hit them, why that might actually work. And so again, these companies, as they make the transition, they want to start playing with these other marketing channels. But the measurement is really hard. It's really hard to make a connection between the billboard that someone sees, but there's no impression registered in a pixel. And then when they go to Walmart and grab something off the shelf and then pay in cash, Right. There's just like no way to really make that connection outside of some experiment or some statistical methodology for looking at, on average, what's the impact of this type of marketing investment.