A Masterclass On Marketplaces & Data Science With Ramesh Johari | EM Group Chat #177

Welcome back to Everything Marketplaces, where we talk with founders and leaders from some of today's top marketplaces. So this is episode 177, which is really a great group chat that we just had with Ramesh Johari. So Ramesh is currently a professor at Stanford University and has previously been an advisor to notable marketplaces like Airbnb, Uber, and Upwork. Ramesh is a leading mind when it comes to marketplaces and specifically data science methods and practices. So this is a great chat where we start off with Ramesh sharing what initially led him to marketplaces. What makes them great businesses? We did a deep dive into data and the role it plays in building a marketplace, discussed topics like experimentation, how AI is impacting marketplaces. Ramesh shared lots of tips for founders. We also had a great group Q&A. I really enjoyed this conversation and you're going to find it a great watch to the end. So Ramesh, I mentioned it to you right before we started recording, but you're of course a leading mind when it comes to marketplaces. Have been one of the most requested group chat guests. So I'm excited to have you join us here today and want to start off by saying a huge thanks for taking the time to do so in advance. So before we dive into lots of topics, I think it might be great if you can set off our group with sharing your background, those here that might not know you.

Yeah, I'm a professor at Stanford. I've been here for 2 decades actually, which is nuts. This is my, my second, my 20-year anniversary here. Actually, when I started at Stanford, I was really a guy basically doing math. like doing applied math and related to networks and things like that, but basically applied math. And these days, vast majority of my work, as people on this chat probably already know, relates to designing and operating online platforms and markets, and also thinking a lot about the data scientific tools that platforms and markets use. And of course, we'll be talking a lot more about that today. Thanks for having me on, Mike. That's the, that's the other thing I wanted to say.

Yeah, no, that's, it's awesome. It's a real treat for us here today. As I mentioned, you know, you've been a long requested group chat guest. So that's of course also a great background, but you know, like what initially kind of led you to marketplaces and why have you continued to focus on 'em, you know, throughout your advising and academic career?

Yeah, this is actually a good story and I think it's related to, you know, often my grad students sort of ask me like, oh, you know, what should I do as a PhD student? What should I do as like a new, you know, assistant faculty member somewhere? And the main thing I tell 'em is don't do what I did. And the reason is when I got here, like I said, I started in 2004. When I got to Stanford, I, like I said, I was doing applied math basically. I didn't touch data for the first, you know, 8 or 9 years I was here. I just built models of things and talked about what those models had to say about the world. And it was kind of a funny thing, but, you know, basically like as I got tenure and then I came up for a sabbatical, took some time, wanted to figure out, you know, what should I do? A couple of my friends were sort of telling me, you know, hey, like, You spent all this time working on theory and math. What makes you think any of that is actually meaningful or accurate? Maybe you should go talk to people who actually work with real data and sort of stress test like, hey, is theorizing about the world actually a useful thing to do? So that's what I did in 2011. I took a leave to go to what was then oDesk, it's now Upwork. And first I started out as a research scientist there and then I ended up running the data science team. It was called Data Products and Research. And, you know, really the reason I went to oDesk is because I wanted to understand if you ran a whole market, you got to see everything, you got to see matches all the way over to how matches completed. So it was basically like you were sort of almost, you know, like kind of omnisciently observing, like in The Truman Show, you know, what's going on in this entire marketplace, then do you need theory anymore? Like, do you need models? Do you need like a, you know, sort of a framework within which you reason about the market? And we'll talk more about this, but basically that's certainly what led me into marketplaces is essentially that they were from an economic standpoint, one of the most exciting ways that technology was changing human interaction, I thought, because of this kind of closed nature of the system where you could observe so much. And then as someone who you know, put so much effort into modeling and thinking about the structure of these kinds of systems, I really wanted to understand what the data that these markets were generating had to tell us about economic activity. So that was back in 2012, and it's 2024 now, and the last 12 years of my life have been shaped by that left turn. So now I tell my students, start where I started in 2012, not where I started in 2004.

No, that's great. And so you mentioned, you know, data science, of course, and that's something that I wanted to spend some time talking about. and, you know, the role it plays at marketplaces. But I guess as a starting point, could you maybe help us kind of, you know, define, you know, what we mean when we mention data science and why it's important for marketplaces?

Yeah, and I know a lot of people on the call may have either heard or like read, you know, some snippets of the podcast I did with Lenny, and I talk about this a little bit there. For the purposes of this call, I'll kind of be succinct in describing some of the points that I made there. The most important thing for a digital marketplace, like a technology-enabled marketplace is first of all, just that, you know, if you think about what markets looked like thousands of years ago, like Trajan's Market in Rome or the Agora in Athens, these were non-digital markets and any transaction that took place was lost to the ether. Like we never have a, you know, we don't have a record for posterity about like the the buyer, the seller, you know, the terms of trade, how they found each other. And digital marketplaces upend all that. We, all of these digital footprints, the digital exhaust is like recorded for us. So that, first of all, in a nutshell, why data science? It's because data is, I think, the foundation of what makes these marketplaces technologically unique as opposed to something like Trajan's Market. And what does data science mean and when does it matter? I think one could arguably make the case that basically any aspect of a modern marketplace should be informed what we do with this data ranging from marketing to product to ops. But I want to focus on a distinction between kind of 2 different types of things you could do. One type of thing you could do is something you would do even if you weren't running a marketplace. So let's take marketing as an example, right? Whether you're running Johnson& Johnson or you're running Airbnb, you know, figuring out how to run your marketing campaigns is a thing that you have to spend time on. And it wouldn't surprise me at all if J&J and Airbnb cross-recruit, you know, people that are working in marketing at the other company. There's not really a huge difference there, like mechanically in terms of the things that data scientists that work on marketing do, right? You're trying to evaluate campaigns, you're trying to figure out multichannel marketing, you know, those kinds of things. What's different in a marketplace, I think though, is the set of data scientific questions that are specifically related to having 2 scaled sides, say buyers and sellers, at least 2. Some markets have more than 2, like Instacart, DoorDash are multi-sided marketplaces. And as soon as you have multiple sides of the marketplace and you're thinking, what does data science mean there? It brings in a sort of new roster of problems for data science to grapple with, which basically involve helping each side sort through the mass on the other side to understand which are the best matches for me and why, and then to help enable those matches. And that's a very different type of problem than the kind of thing that usually arises. You can think of it as sort of the equivalent of the Netflix recommendation problem, but on both sides of the market simultaneously. And we'll talk a little bit more, I think, over the course of the chat about some of the ways in which this problem is, let's say, different than the Netflix problem. But I think fundamentally, like, one of the things that I try to draw people's attention to is what kinds of problems and data science problems in particular are different because you're running a marketplace. Which ones are just kind of what you would see no matter what kind of digital business you were running. And so I want to draw attention to the fact that finding and making good matches, I think, is the core of what innovative data science in a marketplace, what makes that look different than maybe what you would see, let's say, again, if you were working at Johnson& Johnson or something like that on the product side.

And that's a really great point. It's going to be helpful. It'll probably lead to a few questions when we get to that Q&A. But, you know, we have lots of earlier stage founders and teams here. So I guess, could you maybe share more about, you know, at what stage founders should start to think about data and maybe some of the kind of key considerations in doing so?

Yeah, actually. Okay, there's a couple of things that I want to say. One of them is that I kind of hinted at some of this in the podcast with Lenny too, that like, so on one hand, Every business is a platform today. You know, so I mentioned Johnson& Johnson. One reason I specifically mentioned them is that I run a center, I co-direct a center here called the Causal Science Center. It's focused on experimentation, causal inference. And we have a number of tech companies that we work with, but one of the companies we also work with is Johnson& Johnson. And that's sort of interesting because, you know, nobody thinks of them as like naturally a digital platform business, but Even if you're J&J, you're thinking about the digitization of reaching your consumers, of marketing your goods. So what that means is like, if you're a founder, it doesn't matter what you're starting in 2024, you are in a data-driven business. It doesn't matter if it's a marketplace or not.

Okay?

So I think when should you start thinking about data? Day one, you should start thinking about data because that is the distinguishing feature of building your business using digital technology. There's a lot of challenges that, you know, come into that. We can touch on some of those, but I wanted to separate that from what it means to think of data as a marketplace. And so I think 2 things that maybe will sound contradictory but are both true. One of them is that no business is a marketplace when it starts. And the other is that every scaled business is eventually a platform or a market. And what do I mean by this? No business is a marketplace when it starts. And what I mean by that, if you want to be like sort of drill into that comment is I think most of the interesting aspects of thinking of a market only happen when you actually have scaling on both sides of the market, not just one. When you start, you almost never have that liquidity. You never have both buyers and sellers scaled up from day one. That's very rare. There's exceptions when there's like subsidiaries, for example, that start or spin out, you know, of a larger parent organization. But otherwise, if you're just, you know, a founder starting out, Probably the way you're starting out is you're first getting users on board or you're first getting suppliers on board and then hoping they'll attract the other side of the market. So when you start, you're not a marketplace. I think it's really important because you do want to be thinking of yourself as a data-driven business, but you probably don't want to be obsessing about things like how do I build the best matching engine because you just don't have enough liquidity for that to be a first-order problem. So no business is a marketplace when it starts. What about the contradictory statement? Every business is a marketplace when it scales. Well, why am I saying that? I'm saying that because digital supply chains are almost never completely vertically integrated today. There's a very, very small number of companies, literally you can count on one hand that look that vertically integrated, right? The FAANG, basically companies like Facebook, you know, or Meta, Amazon, I guess, MANG now, Meta, Amazon, you know, Netflix, Google, et cetera, right? So there's a handful of those, but you're not that if you're starting out. And even if you imagine what scaling looks like and when you're scaled, that because the digital supply chain you're a part of is not going to be just you, it's gonna involve you with multiple inputs and outputs. Those inputs and outputs are gonna treat you like a platform right now. Sometimes it's explicitly a market like Airbnb. But even if you think about a company like Johnson& Johnson, they work with a large number of suppliers. to enable their products to get made. And they of course are selling to consumers, to businesses. And so that's the sides that they deal with. And that problem of intermediating between suppliers and users doesn't look actually that different than the kind of problem that all of you who are, you know, marketplace founders are thinking about. So I think the hard thing is to know, well, when have I gotten to a place where dealing with the challenges that come along with scaled liquidity on multiple sides is the right thing for me to spend time on. And we could talk a little bit about that. I have some sort of evolved thoughts on that that, you know, I'm hoping we'll be able to touch on as we go along.

You mentioned as far as, you know, getting into that a little bit more as far as, you know, some of the challenges that kind of come with scaled liquidity. So it would be great to spend some time maybe talking about that. And as you mentioned, some of your evolved thoughts.

Yeah, that's so one, one sort of like advertisement I want to put out there is I'm going to be teaching a couple-hour live course at Stanford sometime in February that's going to be, you know, open for people to sign up for. So look out for that. I'll probably announce it on LinkedIn at some point. And part of what I'm testing out the waters with here is some of the thoughts of things that I'll, you know, I'll try to mention on that call. And so one thing I want to talk about is metrics. That's, you know, talking about sort of what's, how do you know where you're at? as a founder, you know, so I think one way to think about it is if you start a company, what are metrics you're gonna care about, right? So one metric you're obviously gonna care about is something like conversion, right? Growth. You'll look at daily active users, you know, retention, right? So all the usual metrics. If you go like to ChatGPT, you say, I'm starting a new business, what are the top 10 metrics I should look at? The thing is those 10 metrics are gonna matter to you whether you're a marketplace or not. So what's important is what metrics are different because I'm a marketplace. And the kinds of metrics I want people to focus on are metrics that are testing the sensitivity of one side of the market to the health of the other side of the market. So what does this mean? When you start, you're gonna be in a place where if you increase the number of suppliers on the opposite side of the market, that's actually not having a huge impact on the behavior on the demand side of the market. Simple example, I mentioned UrbanSitting, I think in the podcast with Lenny, it's a babysitting marketplace. You have parents, you have babysitters. When UrbanSitter started, if you added 10 babysitters, this wasn't going to change anything about parent behavior. Parent behavior was entirely just constrained by the friction of booking and paying babysitters, not determined by the liquidity on the opposite side, right? When they get bigger, you get to a place where now suddenly you realize, oh, you know, if I can add another 100 babysitters, I'll have much better retention of parents. I'll have much better conversion of parents. And those kinds of things, they only happen when you get to a place where the limiting factor to someone's engagement with your platform is whether or not they feel like they're seeing enough on the opposite side of the market. And I think it's counterintuitive. You might imagine, well, obviously when you're young, that next babysitter is hugely valuable. The problem is that can't be the case because you're not at a place where people are coming to your platform for liquidity. They're coming to your business to solve a different problem. That's what I talk about in the podcast with Lenny. There are other transaction costs that you're dealing with, not liquidity. And so early on, ironically, the early additions you make on the opposite side of the market, they're not the reason you're getting more people. So the metrics you want to be paying attention to are how does one side of the market respond to growth on the opposite side of the market? And when does it start to seem like the response of growth to the increases on the opposite side of the market is positive? How could you see this? You can look at marketing investment and growing the opposite side and ask sort of what's the impact of that marketing investment on, let's say, the, you know, you invest on the driver side, what's the impact of that on my rider side, right? And vice versa. So these kinds of metrics are what I call cross-market metrics. And those are the things, you know, certainly if you're an economist who thinks about, you know, market design, network effects. These are the things that those papers are all dealing with every day. But I came to realize that, you know, if you were a founder, that just looks like a jumble of metrics. These don't look any different to you than conversion rates, right? In a way, it is a conversion rate. And I'm sort of asking myself as a, you know, what's the conversion rate of a user against the supply side? What I'm trying to say is, well, actually it's not, that obvious, you want to split that apart a little bit. You want to ask just what's the raw probability customer buys something from my business? But then you actually are here asking about a derivative of that with respect to the rate of change of the opposite side of the market. And I think it's that kind of thinking, this kind of cross-market elasticity is the term that an economist would use, that I think is very important to distinguishing when you are versus are not a marketplace. In the sense that I've been talking about up to now in the call.

No, that's going to be super helpful here. So definitely lots of complexities. And I think this kind of goes back and helps reinforce the point about data mattering, you know, from the beginning so that we can kind of learn and make informed decisions. And of course, with marketplaces, as you mentioned, there's kind of trade-offs to everything as well.

So yeah, I think I want to— I just want to, since you brought that up, that's another example of a metric, I think, that tells you where you're operating as a marketplace. So I just talked about like a marketing problem, right? Which is like, I'm gonna grow the opposite side. What's the returns to growing the opposite side on this side? There's a complementary aspect of that, which is what I call inventory management. So another question you can ask yourself, it's really the same question as what I just described, but I'm gonna frame it now in terms of not so much bringing in new participants, but sort of, you know, retaining and making happy the existing participants. The way to think about that is if you're Netflix and you ask yourself, you know, if Ramesh watches Juror Number 2, great movie I just saw a couple nights ago, you know, on streaming, does that mean that, you know, Dimitri can't watch Juror Number 2, right? So basically, does my watching a movie preclude Dimitri from watching that movie? You know, there was a time when it did because Netflix used to ship DVDs out. So if I had the DVD, he didn't have the DVD, right? That kind of thing was part of the original environment of Netflix where there were inventory constraints. And you could ask the question, what does one person transacting with my platform do to another person who wants to transact with my platform? The math name for this is a shadow price. That's typically what we refer to it as. And really just sort of colloquially what we're saying is the tighter the inventory constraint on the opposite side is, The more likely it is that one person doing something is precluding someone else from doing something, right? And so that's another one of those metrics that you can look at that is telling you something about the health of inventory. So most platforms that are really scaled up are going to look at this regularly. Again, you could see it's a cross-market metric because you're really asking yourself, how does this behavior across the market of one individual influence another individual? It's kind of the level of competition induced by scarcity on the opposite side of the market. This temporally varies. It varies, you know, across space. So, you know, some cities may be more constrained than others for Uber. Sometimes a day are more concerned— constrained for Uber than others, that kind of thing. So it's not as if these are just like point in time, you know, fixed once kinds of metrics. But again, it's that— it's the— it's another version of a metric you can look at. It's kind of a complementary version of the thing I was saying earlier. that tells you something about the cross-market health of your business. And in particular, are you really a market or not? So Netflix today is in a world in which Dimitri is completely uninfluenced by my viewing behavior, right? He and I can watch whatever we want. So Netflix is an example of a platform where they have infinite supply on the opposite side effectively. And it's really only thinking about how to optimize for, you know, the viewers. And You know, I think what's interesting to think about is, well, what can you, what lessons can you take from a Netflix-like recommendation system for a 2-sided market? And this is the biggest one, which is that a 2-sided market needs to be like Netflix, but with, while respecting the role of constrained inventory. And that's where some of the data science, you know, challenges happen is in translating from vanilla recommendation systems out there into recommendation systems that are appropriate for an inventory-constrained environment.

That's a really great example. So cool. So I did want to spend some time talking about experimentation, but, you know, that's a— that could be a whole other kind of conversation in itself. So I guess, you know, one of the questions that a founder actually wanted me to ask in advance, though, is, you know, what are some of the biggest mistakes that you see marketplaces make around experimentation?

Yeah, let me sort of— yeah, right. It could be a big topic and it is. It's a fun one. I just want to tell everyone, we just, I think literally today had a Harvard Business Review article that came out. So it's myself, Martin Tingley, Iavor Bojinov, David Holtz, and Sven Schmidt that's basically talking about sort of scaling experimentation. So how do you get from, you know, starting up experimentation using like third-party tools and so on up to the next phase of experimentation where you're actually kind of operating at high velocity and chaining experiments together in an intelligent way. So I think that's a good read to address some of the issues that this question brings up. I think what's complementary to the article is, you know, what are mistakes when you're starting out with experimentation? And I think there's 2 things I want to bring up. One of them is you can, I think, paralyze yourself. You know, I think It took me a long time to understand this honestly, because when I started out in the tech industry, I thought, of course you want to be data driven. Isn't that obvious, right? Like, why wouldn't you want to be data driven? But there's a funny thing about experimentation, which is it demands a level of rigor in the volume of data you're able to obtain and the precision with which you're able to estimate effects that actually can preclude you from thinking of other big things for which collecting that kind of data wouldn't be possible. For example, you're a new company. Maybe what you're doing is you're charging users per transaction and you're thinking to yourself, oh, you know, should we move to a menu? I mentioned Urban Sitter earlier, and this is one of the interesting things they did is they moved from a per transaction credit card fee billing model to something where they opened up a subscription possibility. I'm putting aside all the desiderata that go into making that choice. One of the things I wanna point out to you is you cannot A/B test that when you're a new business. That's not something you're gonna be experimenting with easily. So if you sit there and you say to yourself, I will only be data-driven in the sense that I will only do things for which I can run an experiment and validate it's a good idea, you would never even try it. And, you know, I think there's this funny sense in which, while it pains me to say it, it's true that if you rely on experimentation for everything, especially as a young business without that level of high velocity and, you know, parallelism of experimentation that larger businesses can have access to, what you'll end up doing is just looking under lampposts that are already lit and you may not end up investigating the bigger high-risk ideas. Now, I want to say that, and this is something the article does talk about, that there's a lot of evidence that's emerged in the academic literature about this sort of role of high-risk ideas in experimentation. I don't know if most people know this, but, you know, success rates at successful companies with experimentation are very low. experiments can be 10%, right? And Ron Kohavi, for example, has written about this quite a bit. So 10%. So what does that mean? Like, one option is you could say to yourself, oh, that's terrible. I should be chasing higher success rates. If our experimentation program is such that we're only succeeding 1 out of 10 times, my data scientists are doing something wrong. I should be succeeding 9 out of 10 times. The problem is if you push them to succeed 9 out of 10 times, they're gonna do it by being very incremental. And the big ideas that can really move the needle won't get tested because they're too high risk. The goal actually is to test higher risk ideas and to abandon those that don't look good quickly and to test many of them so that you figure out kind of what these big winners could be, right? And some of that is actually A/B testing in the narrow sense, but some of experimentation is actually what we sort of humans think of colloquially as experimentation, right? I'm going to try out a new fitness routine this weekend. I'm not running a randomized trial. I'm just going to go, you know, for a 6-mile run instead of a 3-mile run, right? So, I think it's important to distinguish between being quantified and data-driven that way. So, what I try to encourage people to do is try to attach numbers to your beliefs of why you think something might be a good idea and also use that to ask yourself, what evidence do you need to see from this new thing that you're trying out? You know, maybe not without a like strictly defined experiment. What do you need to see to convince yourself that, hey, maybe there's a there there, right? And when you put it that way, a lot of what founders do is experimentation. It's not A/B testing as run by an Optimizely-like platform, but it's definitely experimentation. And I think you want to view all of that as a spectrum. And especially when you're young, you do not want to paralyze yourself by like falling into the trap of only doing things where A/B tests can tell you that they were a good idea. So I think, you know, there's like the kind of points I want to emphasize there is sort of one, the role of testing high-risk ideas and two, the role of being quantified. This idea of being sort of quantified rather than quantified as a superset of being data-driven in my view.

Yeah.

I think that's going to be super helpful here and kind of framework for founders to think about it. So cool. So we couldn't have a conversation without spending some time talking about AI and I think the impact it's having on marketplaces and how you think it'll continue to do so moving forward. So I guess, you know, on that note, you know, how are you seeing AI impact marketplaces today?

Yeah, that's a, it's a great question. I think we're still in a very fluid state of what that impact looks like. There's kind of like the obvious impacts that seem to be not that great, which is like Rufus, like the Amazon chat assistant or equivalent, right? Every company has tried. One platform I talked to, they observed what I think most platforms have observed, which is like, we launch these things and most people don't use them because we're all so conditioned to using sort of short keyword searches and search bars that I can invest the effort to build a chatbot, but nobody's really chatting with it. Now, Perplexity has kind of turned that on its head by having people like sort of use Perplexity as a shopping engine. I don't have numbers on how well they're doing based on my own anecdotal observation. I don't think that's like the primary use case of Perplexity for most people. And so there's this funny thing where like, you know, if you just narrowly ask like injection of an LLM chatbot into marketplaces, like what's that doing? I don't think it's really doing a lot. In a different sense though, I think it's completely transformative. And I think the reason it's transformative has less to do with LLMs and chat interfaces narrowly and more broadly with what's actually under the hood in this revolution in machine learning. And what is under the hood? is our ability to build predictive models with far higher fidelity or, you know, lower error to use the machine learning term. And in very sort of rich, complex data environments than we were able to do, you know, pre-transformers. Okay. So why does that matter? Well, because if you look at marketplaces, I think there's 2 kinds, and I'm going to distinguish one class away, although I think You know, AI's revolution has a lot to do for those marketplaces as well. So one class is like ridesharing. The Uber example is a sort of commoditized marketplace. I think there, there's not as much heterogeneity in the goods being sold. Let me put that aside. Let me think more about like Upwork where there's a huge, huge heterogeneity of things being sold and transacted, right? Different types of jobs from fixed price to hourly development to admin work, Right? So historically on Upwork, the descriptions may even be written by people who don't quite know what they want. They're not even experts enough to write a description that describes really what needs to get done. On the other side, workers might not be in a position to understand how to do that translation for the employer, right? Now, we've gone overnight into a world in which suddenly these highly unstructured thoughts about what it is I'm trying to do or what I'm capable of doing, Can be represented in a far richer way. And I think what I'm looking forward to is, well, what is the frontier of matching, for example, based on modern language modeling? What does that look like, right? Because one of the biggest challenges we kept facing when I was at Autodesk and Upwork, and I think last into today, is taxonomy and hierarchy. How do individuals think about what kind of job they're posting for, what kind of work they're doing, what skills they have? I think all of that becomes upended. So the definition of your market, what the goods are, which is like the core of any market, actually gets completely upended. And I think we're only at the very early stages of exploring that space. There's a different, I think, revolution on the horizon for marketplaces that look like ridesharing markets or markets that you might think are more commoditized, which is that even they are operating with these like very rich streams of temporal data. If you look at the state of the Uber network, it's a very high fidelity representation of every single car and ride in, let's say, the city of San Francisco, right? Over all time since Uber's birth. So that's a very, very rich sequence of data, time series of data. And what this same technology, transformers or modern sequence modeling technologies, have opened up is the possibility of far higher fidelity forecasting and causal inference in these kinds of time series than anything we've seen before. So I'm pretty excited, I think, on that front as well, which is from a data scientific perspective that, you know, there's this push outside of the tech industry and sort of scientific research towards what are called digital twins. Basically, you can build a synthetic example. Think of an assembly line floor. I could have a digital twin of the assembly line if I'm Ford and use that before I build out a full assembly line to know what would its throughput be. Imagine digital twins for marketplaces that are leveraging this kind of modeling technology. I think we're again only in the very early days of imagining what that kind of revolution could look like. So I wanted to mention these because I think people think, you know, AI and marketplaces that your mind goes to Rufus. And I just want to point out, I feel like that's really not the right thing to focus on. You want to focus on sort of how is this underlying technology, what is powering it, and how is it changing the problems we used to think were fundamental for, you know, running a marketplace?

Definitely agree. It's super early. And then also too, as a notice, like how it's going to change. And I guess, you know, somewhat related topic is, you know, opportunities for marketplaces. And so, you know, are you seeing any specific opportunities maybe that you're excited about for marketplaces?

Yeah, it's sort of interesting, right? I think the opportunities haven't changed, and it's kind of, One thing I'm up close and personal with at Stanford is that what I always found exciting about, you know, that, so once I started working with oDesk and Upwork, that as many of you know, kind of led into the opportunity to work with a lot of different marketplaces. And if you look at my sort of LinkedIn profile, you'll see there wasn't really a lot of, you know, specificity in one vertical or another. I've actually had a chance to work with a wide range of industries over that time. And one of the reasons I find that really exciting is Marketplaces as a tech business are sort of an interesting bridge between the digital world and the physical world. Stitch Fix is retail, Airbnb is lodging, Uber is rides. And I continue to be fascinated by that bridge. And that's where I think marketplaces as businesses are really exciting. So another one that we didn't, you know, I haven't touched in my industrial work, but I have in my research work is education where basically, you know, universities are marketplaces. We're matching courses with students. And of course, there's many marketplace platforms that are trying to do the same thing. I think that's a really interesting way to think about verticals for marketplaces because so much of the push in AI is away from that human physical interface. I think one of my colleagues, for example, has pointed out there's a big gap between robotics and AI right now. AI has just completely taken off. It's a massive steep vertical. Robotics has not advanced, you know, to the same degree. You, you can't go out and buy a $10 robot that's going to fold your laundry for you. That remains a very hard problem, right? But you can, you know, go interact with L1 and prove theorems that you couldn't prove yesterday. So why that gap? And well, that gap is super interesting because that gap means there's still this interface to the physical reality of sort of what humans deal with every day that's not being addressed. And I think every vertical I find interesting You know, kind of falls in that space. I'm talking now to a company that's kind of working on food delivery, sort of prepared food delivery. You know, that's of course a super interesting space like that. I think other marketplaces like we have a dog and, you know, there's obviously marketplaces like Rover and so on for sort of pet support. So that's like another vertical like that, right? I think those are all so interesting because That remains a place where you need 2 sides. You really need somebody to provide that service on the opposite side. AI is not here yet to replace your dog walker, to replace your, you know, sort of food delivery, to replace the rides that you're getting. We're getting there on that last one, but, you know, not, I think there's so many ways in which we're not quite there yet. And I think that's how I think about the distinction right now.

I definitely agree that there's so many different interesting kind of, you know, opportunities as well. So we're gonna get into the group Q&A because I'm getting some messages from founders. Hey, Dimitri, saw you raise your hand. Do you wanna help us kind of kick things off here?

Hi, Russ. I have a question for you about setting up reputation or review systems for marketplace. So imagine you're, you're building a marketplace, kind of given what we learned so far in kind of the management literature, how would you set up the reputation system in kind of broad strokes? And is tinkering with that kind of worth Is that like a kind of a high gradient part of your marketplace to optimize? Or is it something that's, as long as you have something pretty basic, that's as good as it's gonna get?

Yeah.

So a few rapid-fire thoughts on this. First of all, on the high gradient part, let me expand from reputation to what I'm gonna call outcome measurement. Okay. Which includes what we traditionally think of as reputation, which is like the feedback somebody leaves right after a transaction. which should also include other things, chargebacks, customer support cases, you know, repeat business, right? Does somebody who transacted come back and try to buy something again? If you take that broadly, then I think there's just no way. It's not just a matter of high gradient. I think it's existential to focus on outcome measurement. And that's because if you don't do something on that front, you're basically A lead gen business. The only thing you can do is bring people in and hope they transact with each other. You'll never see how they did after they transacted with each other. So everybody on this call, everybody kind of in this community, you are a marketplace in my view, a digital marketplace. If you're doing some of that, if you're seeing something about that closed cycle of how the matches you make translate to good or bad outcomes, right? Okay. So let's start there now. So that means, yeah, it's like, it's a fundamental aspect of being a digital marketplace. So where does reputation and feedback specifically fit into that outcome measurement environment? Well, so it's kind of an interesting thing. I think in terms of like a couple things that everybody has observed widely now, almost any sort of off-the-shelf vanilla 5-star type rating system you would implement suffers from what's called reputation inflation, which is that over time, everybody's 4.9 to 5 stars and there's not a lot of information left. I think, you know, there's great examples of this, like you should never buy from a 99.1% seller on eBay. I apologize if any of you are, because that puts you down in like the 15th percentile of eBay sellers in terms of ratings, right? So like great examples like that on pretty much any platform. So that's a problem. And that means one way to think about that is that 5-star ratings, vanilla 5-star ratings are kind of a thing you have to make people feel comfortable. But they don't actually do anything in terms of outcome measurement. And that might be okay, because if you're getting enough other outcome signal, maybe you don't care. So eBay, for example, one paper that Steve Teddalis and others wrote focuses on what's called effective percent positive, which is that eBay, if someone doesn't leave a rating, that's actually extremely informative. It's effectively a very strong negative rating. So rather than collecting feedback, you can just let people choose whether to leave feedback. Right? Then that's great outcome measurement for them. So I think that's an important distinction. You could say, well, what problem am I really trying to solve? If the problem I'm trying to solve is make people feel comfortable with the user interface, you know, people on the outside, they look and they say, oh, everybody at eBay is 99%, that makes me comfortable to try this out. Okay, maybe that's fine. And then I got other ways that I'm gonna use to measure outcomes. Okay. I think that's unfortunately a little sad because there's a lot of room for innovation here. So just 2 points I'll make about innovation. One of them is something that came outta Airbnb as a paper which involved blinding feedback, right? So, so a very simple way to start to deal with reputation inflation is to say, I'm not gonna let you interact with the other person around feedback. Like, you're gonna have to write your review without knowing anything about what they're gonna say about you, right? That has its limits because sometimes there's still messaging back and forth leading up to the reviews on Airbnb. Maybe others have experienced this, but It's way better than the status quo before where you could like write the reviews and then edit them after they were posted and, you know, things like that. So one thing is this kind of blinding exercise of really compartmentalizing you to write the reviews, very similar to peer review in the academic literature. The second one is something I worked on with a graduate student where sort of one of the things I think that isn't done enough is to norm reviews, which basically involves saying, You know, you just had this experience. How did that experience compare to this other experience you had in the past? Right? And the reason that's important is because you can't declare every experience to be better than every other experience you've ever had in the past indefinitely. That just cognitively is dissonant for the typical individual. So one platform that kind of implemented something like this is Bely in New York, B-L-Y, and they have essentially this kind of process as even part of their onboarding. You basically are forced to rank restaurants against each other by saying, you know, was, was this place you, you ate at better or worse than that other place you told us you ate at in the past? And that kind of thing, I think, is, is a way of continuously getting valuable outcome signal. We even found in our paper that just changing the norms, the labels on the scale can provide a lot of valuable information. So, you know, actually universities have started doing this in recommendation letters. It used to be when I started at Stanford, you were asked, You know, is this individual in the top 5% of undergrads you've ever worked with? Now it's 5%, 2%, 1% among the top few undergraduates I've ever worked with at Stanford. Eventually those things inflate too, but it's a lot harder to inflate everybody up to the top few undergraduates I've ever worked with at Stanford. And so even there, like they're getting resolution between the top 2 buckets of that ranking list. In the same way, you know, that platforms that implement these kind of higher resolution scales or sort of norm shifted scales obtain feedback. So we ran an experiment where we kind of showed that was a beneficial change. So I want people to think about these kinds of things, right? Like, don't stick with the 5-star platform. If you think outcome measurement could be better, innovate on it. And if you do choose to innovate on it, by the way, my lab loves to work on these kinds of research projects. So, you know, if there's anyone out there who's like looking for partners to play with experiments on that kind of thing, we have like a track record. Probably the earliest problem I worked on when I got to Stanford was rating system design. And so remains something I'm very interested in, fascinated by. Like I said, I think it's existential for marketplaces to think about how to measure outcomes.

Cool. Hey, Ramesh, I actually wanted to ask a question a founder added to the Google Doc in advance, and it was around experimentation. And do you have like an experimentation toolkit maybe for, you know, for tools that you suggest, you know, founders to use for getting started with experimentation? Experimentation and testing?

Yeah. I mean, again, when we're talking about getting started, I think the key thing is not to let, not, don't get obsessed. So for example, if you go run an archive search for experimentation in marketplaces, that's not a good idea when you're starting. I just want to say upfront, I say this despite being an author of many of the papers that'll come up when you do that search. Like, don't do that. And it goes back to my earlier point. You're not a marketplace when you start. You're not a marketplace when you're early. That's not the experimentation problem you should be trying to solve. I think honestly, these days when you're starting out, the right thing to do is probably to buy a third-party service and just use it to optimize the heck out of the basic vanilla things that experimentation, A/B testing can really help you with, like app design, you know, sort of like the workflow for the user and that kind of thing. So for that, I mean, there's a lot of great tools, right, out there now. You know, one, for example, Sven Schmidt, my co-author on this HBR article, is a former student of mine. who's the head stats engineer at Eppo. There's Statsig, there's Optimizely obviously, and quite a few others. So I think that's an easy way to get started. And depending on what you're looking for, one or more of these is more or less cost effective. Some people just use the tools built into Google Analytics. Some people use Adobe's Test and Target and any number of these kinds of things, right? Where I think it gets interesting is when you say, okay, when does it, again, in line with my earlier comment about metrics, This is, I'd say, a similar question. What makes measurement uniquely marketplace-focused compared to measurement that's like sort of a thing that any company should be doing, right? And I think that comes down to when do you get to a place where running experiments might mislead you? So let me give you an example of this, right? So let's say that I'm a marketplace that has the opportunity to set prices of some kind like Uber. And I consider an intervention that slightly lowers prices. So I'm going to A/B test that on riders. So half my riders see slightly lower prices, the other half the riders don't. So what happens? Well, the riders that see slightly lower prices, they're going to be more likely to book rides. Right?

Okay, great.

Well, that's what should happen. So that, that should— what we should see is that there's a treatment effect on demand that's positive, right? But one thing that's going to be problematic here is that when they book more rides, they're actually reducing the inventory available to the control group that saw the higher prices. And so we're going to overestimate relative to what would've been happening if everybody saw lower prices. The folks that are seeing lower prices now are seeing less competition because everybody else sees higher prices. And relative to that sort of world in which all the price— everybody sees higher prices, those folks are seeing very competitive riders that are seeing lower prices. So we're going to basically overestimate our treatment effect. The treatment folks are going to look better than they would if everybody was in treatment. The control folks will look worse than they would if everybody was in control. Okay, you can say that's not that bad. I've overestimated, you know, what's the effect on demand of lowering prices? What's the big deal? At least I got the sign right. And here's the problem. The problem is you lowered prices, right? You lowered prices, demand goes up. And what do you expect is going to happen to revenue? That we don't know. Price times demand could go up or down. And one of the interesting papers we wrote recently is something that demonstrates that, hey, you know, like one problem with price experimentation in a marketplace is you might get the sign wrong on revenue. You might conclude, I lower prices, of course demand goes up. I might misread the sign of the treatment effect on revenue and conclude that actually my revenues got worse when they would've gotten better. And because that's the case, I can end up in a situation where I'm sort of moving my prices thinking I'm making my marketplace better. But in reality, you know, I'm actually moving away from a revenue optimal point. That kind of example only happens when inventory constraints start to matter on the opposite side of the market. Going back to the earlier theme, there's a very tight interconnection between metrics and measuring when are you a liquid marketplace, when do those constraints matter? And when should you worry about these kinds of effects in marketplace experimentation? So just at a macro level, I want people to understand that it's going to be, you want to know that you're really a marketplace where these effects are there before you go invest in the advanced sort of marketplace experimentation methodology. For the vast majority of early stage startups that envision themselves to be marketplaces, that's not where you're at. The right thing to worry about is just getting the brass tacks of vanilla A/B testing, right? And for that, I think a lot of third-party tools are great. And again, I'll flag the HBR article 'cause I think it does a great job of bridging the gap between where we're at today with third-party platforms and kind of what they provide and up to sort of what does the frontier of scaled A/B testing look like in sort of modern technology businesses?

That's a really great example. So I'll have to, yeah, I'll have to get the link to the paper from you as well so we can include that. So, cool.

Thanks.

Haiyan, do you want to come on?

That's a great discussion. But the tool side, the implementation, I think there's a new paper. I don't think any commercial tools, I think the platforms have it implemented yet. And then how should the company actually try to apply the concept you talk about in the paper and try to discover that? Like, say, I think it depends on the ratio, right? The supply and demand ratio, and then one side is better than the other side.

So, yeah. So, a couple different things there. I think you're referring to like the 2-sided randomization sort of work, right?

Yeah.

So, a couple different things. You already said one thing I think is really important. I'm glad you brought that up. It's a good prompt for me to say this, Mike. I think, you know, if there's one sort of rule of thumb that I want people to take away from marketplace experimentation specifically, which now again, I mean, You're at that point where it matters. This sort of what's called interference effect matters, and you might be getting signs wrong on revenue experiments, that kind of thing. The one takeaway I really want people to think about is that you don't have to worry about those effects whenever you experiment on the side of the market that's more constrained. And the intuition for this is like, imagine that you're Netflix. Which side is more constrained? Well, like I said earlier, movie supply is effectively infinite. So the constraint on Netflix is on the user side. They run all their experiments on the user side. If you think about the world of A/B testing at Netflix on the user side, like I'm trying out a new carousel for how to browse movies and I got a treatment group and a control group. What Ramesh does in the treatment group browsing through movies has no impact on Mike when he's browsing through movies. There's no interference effect between us, right? On the other hand, imagine a different world where, you know, again, like I'm Airbnb. And I run my experiments on the user side at Airbnb. Listings are supply constrained. And now there is an interference effect because if Ramesh sees a better booking flow, better profile pictures, who knows, in the treatment group, maybe he's more likely to book listings. And then that in turn reduces the available inventory to folks in the control group. What that's telling you is, you know, as long as you experiment on the side of the market that's more tightly constrained, ironically, what you're doing is you're avoiding the interference effect from the inventory constraint on the opposite side of the market. That's a great rule of thumb. Many marketplaces don't have the luxury to be able to try experiments on either side, but when you do, it's useful to think about this effect and to think to yourself, hey, like, I'm really worried about interference being a possibility with this new treatment I'm trying out. Which side am I experimenting on? Because that'll maybe mitigate the interference. In fact, the other thing you mentioned about 2-sided randomization, I think I saw on the Google Doc you'd asked this, and I just want to quickly point out there are a few different companies that have tried this. There's not a lot written publicly yet, but I do believe it'll be coming out over the next year or 2. So I know that, for example, in our case, we've had some collaboration with Tencent to using 2-sided randomization in the context of ads. We did something early on with Vinted that won't be part of a paper. Vinted's a secondhand clothing marketplace that sort of was implementing something like this where you could see changes in the estimates of treatment effects if you, if you do this 2-side randomization. 2-side randomization, in part, we suggested because it's an easy thing to do if you have sort of randomization already available to you on both sides of the market. One of the things I've learned since is that there's tech easy and human easy. I don't want anybody to underestimate this. And this is partly why, Mike, I said just get the basics right when you're starting because the tech easy part is like, do I have the tech stack to be able to implement something where I simultaneously randomize on both sides? What I've realized since is the biggest problem actually is it's not normal to the average product manager who's consuming the output of something like this. And we don't have the playbook for how to talk about these kinds of experiments yet. So the irony I think is it's less about engineering deployment of these kinds of things. Much more that we haven't built up this kind of like sort of human interface element of like org design around experimentation that allows all these innovative designs and estimators to be tried out and then presented to, you know, PMs in a decision-making loop without having to like grapple with all the nuances and technical complexity of what this new thing is doing. So I think that's really a fascinating whole area in and of itself, you know, that, that I just want to point out, like for those that want to try new ideas. make sure you're bringing the rest of your business along for the ride because I think sometimes you try these things and, you know, your product team doesn't really know how to engage with what the data is telling them.

So awesome. Those are great questions. We're going to try to squeeze in one last question. Looks like I might have a familiar face. Rob, do you want to come on?

Yes.

Thanks, Mike. Ramesh, thank you for presenting today. I'd really love your quote about no marketplace or no business is a marketplace when it starts, but all are marketplaces when they scale. I'd just love to hear your thoughts about when companies are making the transition from not being a marketplace to scaling. What are some of the investments or things that you'd recommend for companies to have more of that marketplace DNA in terms of like people or skills? Or what do you think has been— be effective for companies to kind of make that transition?

Yeah, I actually think it's interesting, you know, even for companies that, that sort of imagine themselves in marketplace. I really want to double down on the data science of marketplaces here as that investment in particular. There's a great article that Eric Colson, who used to be Chief Algorithms Officer at Stitch Fix, recently wrote. I think he wrote it maybe 2 months ago in O'Reilly about sort of data science as more than just, you know, people that analyze data. And this is a really important way that I think that that comes up and Here's kind of what I'd get at. A lot of the mental shift that has to happen is that suddenly you move from a world in which you can evaluate the health of your business entirely through the behavior of one type of individual, right? Like I said, conversions, daily active users, retained users, like these are things you won't stop looking at those. One, even if you imagine yourself to be a marketplace, as you're scaling up and you're focused on those things, it can be easy to forget that you're getting into a world in which the health of the opposite side is actually the driving factor on the health of, let's say, the user side. Data scientists are actually, especially kind of data scientists that I think come from sort of a multidisciplinary training. So this means sort of data science that involves like econometrics or economics background, operations research background. Actually, to be totally honest, even sort of folks that are trained in applied physics or neuroscience, cognitive neuroscience, like what you see in those environments is an ability to think sort of more at what I call a system level where you're basically asking yourself, what are the main drivers of health? So I might imagine my primary metric actually is retention on the user side. There's a difference between thinking of that mechanically as just let me just track that in a dashboard and thinking about it mechanistically, which is what are the things that are driving user retention, right? And what you'll see is if you break out that metric, you're going to come to a point at which one of the things that's driving retention is, are you growing liquidity on the opposite side? So I think that you want people who take the system-level viewpoint, who are willing to break metrics down and always interrogate, are we actually sort of causally affecting the knobs that are most influential for that top line metric? So in a way, you know, Rob, I think I'm cheating a bit on your question. What I believe is important is actually data science broadly is kind of that core competency that becomes fundamental when you scale up into a big marketplace. So that's where like this kind of liquidity-driven data science is happening. But I think that the point underneath that is you want what I would call a kind of system-level data scientist. You don't just want data scientists who are just purely about implementing statistical methodology and nothing else. You kind of want people who are willing to constantly ask that interrogating question about causal mechanism. And so that's why I said physicist, right? It's like the physics of a marketplace. Like you need to keep asking yourself, when am I crossing over that threshold? So you know, if you're J&J and you're starting out as a Johnson& Johnson type business, you're gonna come to a place where suppliers matter. And when that's the bottleneck, you want somebody to have identified that so that you can start shifting your mindset quickly into imagining these kind of platform types of questions rather than just single-sided optimization. One thing I wanna bring up there, I'll close with this kind of anecdote, right? You know, I, I won't name platforms or people in this story for reasons that you'll see in a second. So I was working with someone and, you know, we were working on sort of helping a PM make a decision about a liquidity management sort of feature. So liquidity management meaning like, you know, something which kind of determine what aspects of the opposite side of the market would one side see. And the, you know, I asked the guys, or I saw the memo and I was like, hey, you know, like I saw what you did here and you used some really advanced methods. It's really cool, but you're kind of not really recommending anything. You don't have like a punchline. You don't have a, you know, here's what I think is happening. Here's a decision we should make. Here's the scenario planning around different things we could do. And the guy's response was sort of like, well, hey, look, my job as a data scientist is to take the methods we know, use them on our data, and implement them and make sure that everybody knows what method I used. That's where my buck stops. Everything after that is the PM. It's the PM's job to take in what I wrote up, figure out what the implications are for decisions, and make those decisions. And I just want to say, I categorically disagree with this viewpoint. Like, that viewpoint is disastrous for data science and it's disastrous for the PM. Like it is the data scientist's job to be the one who's interpreting that, right? And so I think that's where I was going with my earlier comment, Rob. You want data scientists who don't just stop at, let me measure the metrics you're telling me to measure. You want data scientists who are willing to do things like break those metrics apart and say, here's where we're hitting a liquidity constraint, you know, versus not. So I think Hopefully that kind of answer makes sense, but I really feel like that's the type of competency. It's a human resource competency I think that you want to pay attention to as you're moving through that transition.

Awesome. I'll say a great way to wrap things up. And yeah, Ramesh, we really appreciate taking the time to join us here today. So this was long overdue and a real treat. So really appreciate joining us. And I know we also tried to pack it in, covering lots of topics here and going deep, but I'll Also, it's great, you know, to kind of circle things back to the earlier stage and, you know, what really matters here. So, you know, last but not least, time for a quick plug. Where can we keep up with you at?

That's a good question. Well, I mean, I'm an academic. I have a webpage I'm not updating regularly. So I will be updating it again starting in January. I'll post occasionally on LinkedIn. I'll put a couple of quick pointers in. One of them is that I co-direct the Stanford Causal Science Center. And we run a conference for sort of industry academic collaboration in the spring every year, usually around late May. Last year we had like 400 people there. It's a great spot to show up if you want to like interact with a community of folks that span the spectrum from research over to kind of, you know, other founders and so on. And then also, you know, yeah, I'll occasionally post on LinkedIn and things like that about events like this. I am on LinkedIn. So if you want to connect and, you know, have questions, feel free to reach out. Awesome.

I'll include all links in the recording. So once again, really appreciate taking the time to join us here today, and thanks everyone for the questions.