Ramesh Johari explains when to randomize both sides of a marketplace: target the constrained side (users at Netflix, listings at Airbnb) to avoid bias from inventory spillover. Tencent and Vinted have used this approach, but the real challenge is helping product teams understand and act on the results.
Chapters
That's a great discussion. But the tool side, the implementation, I think there's a new paper. I don't think any commercial tools, even the Python, have it implemented yet. And then how should the company actually try to apply the concept you talk about in the paper and try to discover that? Like, say, I think it depends on the ratio, right? The supply and demand ratio, and then one side is better than the other side.
So, yeah. So, a couple different things there. I think you're referring to like the 2-sided randomization sort of work, right? Yeah. So, a couple different things. You already said one thing I think is really important. I'm glad you brought that up. It's a good prompt for me to say this, Mike. I think, you know, if there's one sort of rule of thumb that I want people to take away from marketplace experimentation specifically, which now again, I mean, You're at that point where it matters. This sort of what's called interference effect matters, and you might be getting signs wrong on revenue experiments, that kind of thing. The one takeaway I really want people to think about is that you don't have to worry about those effects whenever you experiment on the side of the market that's more constrained. And the intuition for this is like, imagine that you're Netflix, which side is more constrained? Well, like I said earlier, movie supply is effectively infinite. So the constraint on Netflix is on the user side. They run all their experiments on the user side. If you think about the world of A/B testing at Netflix on the user side, like I'm trying out a new carousel for how to browse movies and I got a treatment group and a control group. What Ramesh does in the treatment group browsing through movies has no impact on Mike when he's browsing through movies. There's no interference effect between us, right? On the other hand, imagine a different world where, you know, again, like I'm Airbnb. And I run my experiments on the user side at Airbnb. Listings are supply constrained. And now there is an interference effect because if Ramesh sees a better booking flow, better profile pictures, who knows, in the treatment group, maybe he's more likely to book listings. And then that in turn reduces the available inventory to folks in the control group. What that's telling you is, you know, as long as you experiment on the side of the market that's more tightly constrained, ironically, what you're doing is you're avoiding the interference effect from the inventory constraint on the opposite side of the market. That's a great rule of thumb. Many marketplaces don't have the luxury to be able to try experiments on either side. But when you do, it's useful to think about this effect and to think to yourself, hey, like, I'm really worried about interference being a possibility with this new treatment I'm trying out. Which side am I experimenting on? Because that'll maybe mitigate the interference effect. The other thing you mentioned about 2-sided randomization, I think I saw on the Google Doc you'd asked this, and I just want to quickly point out, There are a few different companies that have tried this. There's not a lot written publicly yet, but I do believe it'll be coming out over the next year or 2. So I know that, for example, in our case, we've had some collaboration with Tencent to using 2-sided randomization in the context of ads. We did something early on with Vinted that won't be part of a paper. Vinted's a secondhand clothing marketplace that sort of was implementing something like this where you could see Changes in the estimates of treatment effects if you, if you do this 2-side randomization. 2-side randomization, in part, we suggested because it's an easy thing to do if you have sort of randomization already available to you on both sides of the market. One of the things I've learned since is that there's tech easy and human easy. I don't want anybody to underestimate this, and this is partly why, Mike, I said just get the basics right when you're starting, because The tech easy part is like, do I have the tech stack to be able to implement something where I simultaneously randomize on both sides? What I've realized since is the biggest problem actually is it's not normal to the average product manager who's consuming the output of something like this. And we don't have the playbook for how to talk about these kinds of experiments yet. So the irony I think is it's less about engineering deployment of these kinds of things, much more that we haven't built up this kind of like sort of human interface element of like org design around experimentation.
Yeah.
that allows all these innovative designs and estimators to be tried out and then presented to, you know, PMs in a decision-making loop without having to like grapple with all the nuances and technical complexity of what this new thing is doing. So I think that's really a fascinating whole area in and of itself, you know, that I just want to point out, like for those that want to try new ideas, make sure you're bringing the rest of your business along for the ride. Because I think sometimes you try these things and, you know, your product team doesn't really know how to engage with what the data is telling them.
