Ramesh Johari advises early-stage teams to use off-the-shelf A/B testing tools like Eppo or Statsig rather than building custom experiments. Marketplace-specific measurement gets tricky when inventory constraints matter: a price-lowering experiment can overestimate demand lift because the treatment group consumes inventory the control group would have used, potentially pushing founders away from the revenue-optimal price.
Chapters
I actually wanted to ask a question a founder added to the Google Doc in advance, and it was around experimentation. And do you have like an experimentation toolkit maybe for, you know, for tools that you suggest, you know, founders to use for getting started with experimentation and testing?
Yeah, I mean, again, when we're talking about getting started, I think the key thing is not to let— not— don't get obsessed. So for example, if you go run an archive search for experimentation in marketplaces, that's not a good idea when you're starting. I just want to say upfront, I say this despite being an author of many of the papers that'll come up when you do that search. Like, don't do that. And it goes back to my earlier point. You're not a marketplace when you start, you're not a marketplace when you're early. That's not the experimentation problem you should be trying to solve. I think honestly these days when you're starting out, the right thing to do is probably to buy a third-party service and just use it to optimize the heck out of the basic vanilla things that experimentation, A/B testing can really help you with, like app design, you know, sort of like the workflow for the user and That kind of thing. So for that, I mean, there's a lot of great tools right out there now. You know, one, for example, Sven Schmidt, my co-author on this HBR article, is a former student of mine who's the head stats engineer at Eppo. There's Statsig, there's, you know, Optimizely, obviously, and quite a few others. So I think that's an easy way to get started. And depending on what you're looking for, one or more of these is more or less cost effective. Some people just use the tools built into Google Analytics. Some people use, you know, Adobe's Test and Target and, you know, any number of these kinds of things, right? Where I think it gets interesting is when you say, okay, when does it— again, in line with my earlier comment about metrics, this is, I'd say, a similar question. What makes measurement uniquely marketplace-focused compared to measurement that's like sort of a thing that any company should be doing, right? And I think that comes down to when do you get to a place where running experiments might mislead you. So let me give you an example of this, right? So let's say that I'm a marketplace that has the opportunity to set prices of some kind like Uber, and I consider an intervention that slightly lowers prices. So I'm going to A/B test that on riders. So half my riders see slightly lower prices, the other half of the riders don't. So what happens? Well, the riders that see slightly lower prices, they're going to be more likely to book rides. Right? Okay, great. Well, that's what should happen. So that, that should— what we should see is that there's a treatment effect on demand that's positive, right? But one thing that's going to be problematic here is that when they book more rides, they're actually reducing the inventory available to the control group that saw the higher prices. And so we're going to overestimate relative to what would have been happening if everybody saw lower prices. The folks that are seeing lower prices now are seeing less competition. because everybody else sees higher prices. And relative to that sort of world in which all the price— everybody sees higher prices, those folks are seeing very competitive riders that are seeing lower prices. So we're going to basically overestimate our treatment effect. The treatment folks are going to look better than they would if everybody was in treatment. The control folks will look worse than they would if everybody was in control. Okay, you can say that's not that bad. I've overestimated, you know, what's the effect on demand of lowering prices? What's the big deal? At least I got the sign right. Now here's the problem. The problem is you lowered prices, right? You lowered prices, demand goes up. And what do you expect is going to happen to revenue? That we don't know. Price times demand could go up or down. And one of the interesting papers we wrote recently is something that demonstrates that, hey, you know, like one problem with price experimentation in a marketplace is you might get the sign wrong on revenue. You might conclude I lower prices, of course, demand goes up, I might misread the sign of the treatment effect on revenue and conclude that actually my revenues got worse when they would've gotten better. And because that's the case, I can end up in a situation where I'm sort of moving my prices thinking I'm making my marketplace better, but in reality, you know, I'm actually moving away from a revenue optimal point. That kind of example only happens when inventory constraints start to matter on the opposite side of the market. Going back to the earlier theme, there's a very tight interconnection between metrics and measuring when are you a liquid marketplace, when do those constraints matter, and when should you worry about these kinds of effects in marketplace experimentation. So just at a macro level, I want people to understand that it's going to be— you want to know that you're really a marketplace where these effects are there. Before you go invest in the advanced sort of marketplace experimentation methodology, for the vast majority of early stage startups that envision themselves to be marketplaces, that's not where you're at. The right thing to worry about is just getting the brass tacks of vanilla A/B testing, right? And for that, I think a lot of third-party tools are great. And again, I'll flag the HBR article because I think it does a great job of bridging the gap between where we're at today with third-party platforms and kind of what they provide. And up to sort of what is the frontier of scaled A/B testing look like in sort of modern technology businesses?
