Ramesh Johari argues that 5-star ratings fail because scores cluster near 5 and become useless. He points to eBay's percent positive and Airbnb's blinded feedback as better designs, and suggests comparing experiences instead of absolute ratings can increase useful information.
Questions this video answers
Russ expands reputation to outcome measurement, emphasizing that tracking post‑transaction signals (repeat purchases, chargebacks, support cases) is existential for a digital marketplace. He argues that reputation systems are not just a high‑gradient tweak but a core component, and suggests moving beyond vanilla 5‑star ratings toward richer signals.
Dimitri, saw you raise your hand. Do you wanna help us kind of kick things off here?
Hi, Russ. I have a question for you about setting up reputation or review systems for marketplace. So imagine you're, you're building a marketplace, kind of given what we learned so far in kind of the management literature, how would you set up the reputation system in kind of broad strokes? And is tinkering with that kind of worth, is that like a kind of a high gradient part of your marketplace to optimize? Or is it something that's, as long as you have something pretty basic, that's as good as it's gonna get?
Yeah. So a few rapid-fire thoughts on this. First of all, on the high gradient part, let me expand from reputation to what I'm gonna call outcome measurement. Okay. Which includes what we traditionally think of as reputation, which is like the feedback somebody leaves right after a transaction, but should also include other things, chargebacks, customer support cases, you know, repeat business, right? Does somebody who transacted come back and try to buy something again? If you take that broadly, then I think there's just no way. It's not just a matter of high gradient. I think it's existential to focus on outcome measurement. And that's because if you don't do something on that front, you're basically a lead gen business. The only thing you can do is bring people in, and hope they transact with each other, you'll never see how they did after they transacted with each other. So everybody on this call, everybody kind of in this community, you are a marketplace, in my view, a digital marketplace. If you're doing some of that, if you're seeing something about that closed cycle of how the matches you make translate to good or bad outcomes, right? Okay. So let's start there now. So that means, yeah, it's like, it's a fundamental aspect of being a digital marketplace. So where does reputation and feedback specifically fit into that outcome measurement environment? Well, so it's kind of an interesting thing. I think in terms of like a couple things that everybody has observed widely now, almost any sort of off-the-shelf vanilla 5-star type rating system you would implement suffers from what's called reputation inflation, which is that over time, everybody's 4.9 to 5 stars and there's not a lot of information left. I think, you know, there's great examples of this, like you should never buy from a 99.1% seller on eBay. I apologize if any of you are, because that puts you down in like the 15th percentile of eBay sellers in terms of ratings, right? So like great examples like that on pretty much any platform. So that's a problem. And that means one way to think about that is that 5-star ratings, vanilla 5-star ratings are kind of a thing you have to make people feel comfortable. But they don't actually do anything in terms of outcome measurement. And that might be okay, because if you're getting enough other outcome signal, maybe you don't care. So eBay, for example, one paper that Steve Teddalis and others wrote focuses on what's called effective percent positive, which is that eBay, if someone doesn't leave a rating, that's actually extremely informative. It's effectively a very strong negative rating. So rather than collecting feedback, you can just let people choose whether to leave feedback. Right? Then that's great outcome measurement for them. So I think that's an important distinction. You could say, well, what problem am I really trying to solve? If the problem I'm trying to solve is make people feel comfortable with the user interface, you know, people on the outside, they look and they say, oh, everybody at eBay is 99%, that makes me comfortable to try this out. Okay, maybe that's fine. And then I got other ways that I'm gonna use to measure outcomes. Okay. I think that's unfortunately a little sad because there's a lot of room for innovation here. So just 2 points I'll make about innovation. One of them is something that came outta Airbnb as a paper which involved blinding feedback, right? So, so a very simple way to start to deal with reputation inflation is to say, I'm not gonna let you interact with the other person around feedback. Like, you're gonna have to write your review without knowing anything about what they're gonna say about you, right? That has its limits because sometimes there's still messaging back and forth leading up to the reviews on Airbnb. Maybe others have experienced this, but It's way better than the status quo before where you could like write the reviews and then edit them after they were posted and, you know, things like that. So one thing is this kind of blinding exercise of really compartmentalizing you to write the reviews, very similar to peer review in the academic literature. The second one is something I worked on with a graduate student where sort of one of the things I think that isn't done enough is to norm reviews, which basically involves saying, You know, you just had this experience. How did that experience compare to this other experience you had in the past? Right? And the reason that's important is because you can't declare every experience to be better than every other experience you've ever had in the past indefinitely. That just cognitively is dissonant for the typical individual. So, one platform that kind of implemented something like this is Belly in New York, B-L-I, and they have essentially this kind of process as even part of their onboarding. You basically are forced to rank restaurants against each other by saying, you know, was, was this place you, you ate at better or worse than that other place you told us you ate at in the past? And that kind of thing, I think, is, is a way of continuously getting valuable outcome signal. We even found in our paper that just changing the norms, the labels on the scale can provide a lot of valuable information. So, you know, actually universities have started doing this in recommendation letters. It used to be when I started at Stanford, you were asked, You know, is this individual in the top 5% of undergrads you've ever worked with? Now it's 5%, 2%, 1% among the top few undergraduates I've ever worked with at Stanford. Eventually those things inflate too, but it's a lot harder to inflate everybody up to the top few undergraduates I've ever worked with at Stanford. And so even there, like they're getting resolution between the top 2 buckets of that ranking list. In the same way, you know, that platforms that implement these kind of higher resolution scales or sort of norm shifted scales obtain feedback. So we ran an experiment where we kind of showed that was a beneficial change. So I want people to think about these kinds of things, right? Like, don't stick with the 5-star platform. If you think outcome measurement could be better, innovate on it. And if you do choose to innovate on it, by the way, my lab loves to work on these kinds of research projects. So, you know, if there's anyone out there who's like looking for partners to play with experiments on that kind of thing, we have like a track record. Probably the earliest problem I worked on when I got to Stanford was rating system design. And so remains something I'm very interested in, fascinated by. Like I said, I think it's existential for marketplaces to think about how to measure outcomes.
