Marketplace Search, Data, & Ads (At Scale) Workshop

So welcome to our Everything Marketplaces community workshop today, which is going to be on marketplace search, ads, and data at scale. So if this is your first community workshop, welcome. And our workshops are where domain experts join us to walk us through topics that are relevant to us all as marketplace founders and teams here. They're typically 30-minute or so presentations followed by a group Q&A. So that said, I'm really excited to welcome on Andrew Yates today, who's going to walk us through marketplace search, ads, and data. And not just in the earlier stages, but also for marketplaces at scale. So Andrew is also the co-founder of Promoted.ai, which helps top marketplaces like Hipcamp, Outschool, and Teachers Pay Teachers with this. So right before I let Andrew take over, I do once again want to mention that these are about 30 or so minute presentations followed by a brief group Q&A. So if you have any questions, you can just post them in the Zoom chat and just use the Zoom raise hand functionality so that way we know to call on you during the Q&A part. So that said, you know, huge thanks again for taking the time to join us here today, Andrew. I'm really excited for the workshop and I'll actually just let you start with a brief intro and I'll take it over from here.

Sounds great. Thank you so much. Hey everyone, I'm Andrew Yates. I'm CEO and co-founder of Promoted.ai. We sort search and promote the best listings at the top to increase revenue, and we specialize in marketplaces and complex e-commerce. Today, I'm going to be talking about Marketplace Search Data and Ads at Scale workshop. And the emphasis of this is what happens when you scale and why technology is relevant to this. Like why are marketplace companies, online marketplace companies so frequently seen as technology companies? What is the technology doing? So here's an outline. Intro to technical scaling. Next, I'm going to go to search, a little bit about data and a little bit about ads. So first, this intro to technical scaling. Back to the original idea here. Why? Why is Airbnb a top tech company? Because look, look at their app. I mean, it's like a few listings. This is like a hackathon project. And I'm sure you've seen someone put together something that looks like a bunch of listings and a map view in not too much time. So I've got one product. So how complex is that? It's just one, right? And there's no fulfillment even. They don't even have inventory. So what is all of this technology doing? Why is Airbnb a technology company? And they absolutely are. And they're one of the best. And It has something to do with complexity theory. So some of you probably studied computer science and you have to do computational complexity in algorithm class and this idea of the cost of something growing proportionate to the size of the job. So there's this idea of linear growth, sublinear growth, and superlinear growth. The challenge of networks and marketplaces is as you are adding more and more buyers and sellers and contexts, the number of possible interactions are growing faster than the number of individual things you're adding. And so this creates something that feels like running different. It feels like it's different to run and optimize than when you're small. If it were simply a matter of selling widgets, then for every unit of sales, then you add in 0.5 units of cost of doing revenue. And you just keep doing that as much as you have sales and it's no problem. But there's something about marketplaces in particular and complex e-commerce where as you add that next unit of sales or that next unit of growth, the amount of work it takes to operate the whole thing and make it work grows faster. So like the first time it's 0.5, the next time it's 0.7, the next time it's 0.1, 2, 5, 10. And it turns out that it's more work to run the thing than you're making profits. And then the marketplace starts to unravel. So one part of it is the number of possible interactions are growing faster than the number of buyers, sellers, and contacts. The other piece of this, and for any of you who are doing integrity or seller success or anything to do with fulfillment, certainly rare exception cases become common. It's not something that every once in a while happens and you can deal with it as they come up. It's like a one in a million chance, if it's happening multiple millions of times per day, are going to happen many times per day. And then the next piece of this is what was working at small scales in terms of organization structure. Resourcing technology just fundamentally don't work anymore at larger scale. You can't just repeat that unit over and over again. So technology has solutions, but they're they're more complex, and we're going to talk about what some of these look like. So first, I'm going to start with search on scaled marketplaces. Now, when I say search, I really mean just any listings. That could be feed or recommendations, traditional search. There's sometimes like this idea of directed discovery where people like putting in categories effectively. The gist of it though, it's easier just to say, hey, look, this is search, but it applies to all of those things. And by the way, everything I'm talking about applies not just as literal technical two-sided marketplaces like Airbnb, but to Amazon or any sufficiently large e-commerce app. eventually scaled into something that looked very similar to a marketplace. So in summary here, when you're small, you're focusing on things that are relevant. You're returning a best set for everyone. Everyone can be found all of the time. It's one-sided largely. It's like, okay, these are the best sets and then you deliver it to everybody. So it's largely static and stable. So you have maybe 1,000 top queries. You could literally have someone go through and go through all of those top 1,000 queries and make sure they look pretty good and do that over a couple of months. And from computational easy, what's easier than that? You could just have a spreadsheet and run it. In fact, there's some like behind the scenes kind of things. I remember at the story at Pinterest, my background, I was engineering manager at Pinterest in ads. Things were run in a spreadsheet in the ads, like early days. You could do that. That's not true anymore. Scaled. First of all, you're not optimizing just for relevancy, you're optimizing for profitability, which can be much more challenging. We'll talk about that. Best matching. So rather than here's the best set and delivering to everyone, the network effect here, which are like, here are some buyers, here are some sellers, and you're finding the best matches in this network. And that's also much more complex. Small window into total inventory. When I showed the Airbnb, Airbnb has more than 2 listings available, but I only saw 2. Like which 2 listings? That's really important. Those 2 listings really care a lot that they were shown. You only, each individual component is only seeing a small window of the entire picture. 2-sided or more. You have brands, you have sellers, you have buyers, you have your own internal merchandising team, you have internal Different product teams, in fact, can start coming up with more and more sites. And it's necessarily dynamic. And then all of these things combined together make it computationally hard. So we're going to talk a little bit about some of the systems, about starting small and then combining some of these smaller pieces into dealing with this complexity. So at small scale, search really just needs to return, quote, relevant results. So Here are some vendors and some different technologies, and these all work great. I usually recommend Algolia, candidly. I think, hey, what do I need? Algolia. They'll do almost everything you need for a very long time until you're very, very skilled. They also have like a ranker on top. They have a new vector search thing. They're great. I have no affiliation with Algolia. I just think they're good. Elastic is also open source or open core actually, open source available actually. I think they have a different license, but also a very commonly used solution. Pinecone is new kid on the street in terms of like the new vector search stuff. SOLR is the old classic. But anyways, any of these effectively have a query, you have keys, you're indexing into it, you're returning a result set. Each of the individual results have some sort of score, you rank by the score, return it. That works. So for example, small. This is an Algolia example, literally right off Algolia's website. Product discovery. Well, this is Under Armour. And how many shoes does Under Armour have? And like, how frequently is it changing? And like, how complex is this? They only have 450 shoes, 454 exactly at the time I took this screenshot. You can do this with like a relatively straightforward information retrieval approach and with some good filtering, some reasonable ranking. Maybe you want to put in some like simple pinning for merchandising or like a recommendation at the top. It can be done with straightforward vendor technology. So some summaries of this. Everyone can be found. You could, it is not intractable to go through all 400 shoes. You, when you walk into a store, a shoe store, there are probably thousands of different shoes that you scan very quickly with your own eyes. It can be done on a screen. 400 can be done, especially with filtering. So things like best or most recent or hot or highest rated, these sorts of selection criteria work Fine. And it's easy to understand your search quality team or your product team. You may not even need any technical people or data scientists or forget machine learning people. You can take a look at the search results and do they look good or not? If it looked bad, fix it and go change it manually. And then another piece of this is because there's not too many, there's also not too many exceptions. So things like, hey, this is out of season. Hey, this is out of stock. On one side, there's not too many exceptions, and so they don't overwhelm your screen. So you don't have a whole screen of out-of-stock items because you don't have that many items. Another piece is you can also deal with it manually with operations as opposed to computation and with machine learning. But let's look at scale. So the first thing you run into when you start scaling is Everybody wants to be first. And this is especially true for marketplaces or with brands or more of a two-sided marketplace as opposed to a retailer. If you are not first and you are the brand or you are the retailer or you are the seller, then search is bad. Just bad. According to that seller, it's bad. You may think as the retailer, the marketplace, it's great. But every single person who's not first, they will yell at you. Why am I not in this query? Why am I not here? So you need to come up with some sort of solution to satisfy your seller side, or else you're not going to have sellers. Um, another part of it is you can have hundreds of thousands of incredibly relevant results. So before we were talking about finding the things that are relevant and then doing some relatively straightforward ways of filtering through that list. As you scale, you have 1,000 things that are all equally really valid to show. They're all equal good matches. You can show any one of these 1,000 things, but you can't show 1,000 things. So what do you do about that? This idea of best is changing in time and for different people. So this is when you start mixing in a lot of the personalization in particular, but also merchandising. And then it gets really hard mode in ads. And we'll talk about that at the last part. And then starvation and flooding concept. Like, oh, if you have a, if you're huge and you just show the same person at the top, going back to the Airbnb example, geez, I sure hope that person that has some availability, if they're shown to every single literal person in San Francisco, you gotta have some challenge of make sure the first person is being rotated around. And then you have the starvation challenge as well. If you have 1,000 lists and you're like ranked 950, No one will ever buy you or ever see you or discover you. So you flake out. And then personalization, exploration, novelty, diversion, seasonality, merchandising. And what if your inventory is dynamic? Some of the marketplaces that we help, it's not a fixed amount of inventory. There are things like, well, listings on Airbnb, you book it, it's gone. Well, what do you do now? It's ephemeral. So I'm going to show some examples of how ordinary search struggles with some practical scaled marketplace queries, and then I'm going to present some technical architecture to address how you would solve this in standard practice. So here are some examples. Good old Amazon. So I just picked something off the top of my head, AA batteries. This is very specific. If I go into Safeway, there is a shelf and there's maybe one at most two different selections for for triple AA batteries. Amazon has 9,237 listings for AA batteries, and really every single one of these are incredibly relevant. What do you do now? And here's and here is what the front page happened to be for me. Like why? Why did Amazon show me— gosh, if you ever have to do marketplace integrity, imagine if you are ranked 3 here, like Panasonic. How dare you show 2 listings with fewer reviews than mine? They have only 9. I have 5,700. How dare you show the GP Battery Super Alkaline and they're just blah, blah, blah. Like, this is the sort of thing that comes up in scaled marketplace search. The second one doesn't even have any ratings. What's up with that? Is that bad? Probably not. I mean, look, I think Amazon knows what they're doing. They're a pretty successful company. I made it even more specific, like, no, I want the Duracell batteries. For goodness sakes, there's even 115 results and the second result isn't even Duracell. And in fact, neither of these 2 things are organic results. They're both sponsored. And we'll talk about that in a moment here. But Like even when you get super, super specific, when you get scale, you have a lot of different results and the decisioning to decide what you're going to show can get quite sophisticated. So backing this down from like the biggest companies in the world, Hipcamp is one of our customers. I'm going to go through 3 of our customers because I know them best and talk a little bit about some of their marketplace search challenges. Hipcamp, they're— I can't get away from saying this. They are camping like Airbnb for camping. So all of the effects that you have on Airbnb, Hipcamp also has. So for example, a site can only be booked one-on-one. So I can only book one site at a time and one site can only be booked one site at a time. So therefore the best result must be necessarily always changing because they're getting booked. The better it is, the more it's booked, so less available it is. So it must be rotating. Sites with too few bookings shut down. Like, that's also a big problem of how do you deal with your seller side? New people are trying out, hey, I have a new backyard, I made it beautiful. I'm ranked 100th, no one ever sees me. How do you be fair about this? They have like no reviews, they're at the top, the people with 100 reviews are complaining. How do you deal with this? And then really anything nearby to where you're searching is relevant. So this concept of in traditional search retrieval, like if you're going to use Algolia, Algolia will correctly return or Elastic or any kind of geolocation search. Yeah, here's 100 things that are within the radius you asked for and you can rank them by rating or whatever, but that's not enough. Outschool, another one of our customers. So Their searches, I would really say it's a blend between guided discovery relevance and then sometimes people do literally search for a very specific thing, but generally it's more like, hey, oh my gosh, my kid actually has spring break on March 17th and we haven't booked a camp yet. He's going to be at home and I don't know. Okay, gosh, I need to put this kid on something. So summer science camp or spring break summer camp or something like that. So similar sorts of challenges here. Like first, classes will start and stop and fill up. There's ephemeral inventory and then limited inventory. And then really any one of these classes, they have hundreds or even thousands of possibilities that someone could attend during that time. And it also could be highly personalized. My child is in kindergarten, so things that are appropriate for a 6th grader are not going to work for a kindergartner. So this high contextual dependency on time, the classes they fill up. From Outschool, they also have like this bin packing challenge where classes that don't fill up sufficiently will not start and then you get zero. It's a big problem. Or like you have classes that don't get anyone trying to book in them and then you have the starvation problem as well. And then you have some like, some like Guided Discovery down here is Minecraft. There is no mention of summer or I think even camp. No, none of those words show up in that listing at all. Yes, but in practice, this is what my child would actually want to do.

All the other things.

Yeah, yeah, learning Minecraft, and that's how he will be occupied over summer break for sure. So like there's also this challenge of what does relevance mean? In the sort of matching marketplace situation, TPT. So this is more of a situation of scale reading comprehension. They have they have five hundred thousand results, and each of these are individual sellers. A lot of them are small businesses. Oh wow! Whoever is first wants to be first, and if they're not first, they will complain, and they will tell you, and they'll tell the operations team and me and everybody like. But there are 500,000 different results and kids are different and teachers are different. And there's all different kinds of sellers. How do you deal with this sort of resource competition fairly, and also in a way that helps both buyers and sellers? So I talked a little bit about some examples of where just a regular sort of, hey, show me the top relevant results, show them in a page. Information retrieval from either traditional text query matching or even like the fancy vector search. I've used my query, I asked ChatGPT, here's, you know, 20 different listings. That's just not solving the problems that come up in scaled marketplaces and big e-commerce. And here's some of the architecture design about how you do solve these sorts of things. So first is getting away from this concept of there's just the search system. What you're actually going to be doing is retrieving from multiple different sources that each individual are going to be achieving some sort of objective. So for example, new, best, relevant ads, merchandising as well, and personalized. And then this is what most people tend to think of as search is this retrieval section. The next piece is in industry, it's called ranking generally. So if you hear anyone hiring for a search ranking or ads ranking ML engineer, this is stage 2. This is where you've filtered the result set down to maybe 1,000 results or several hundred from maybe multiple different sources. And now you need to score them for this specific example about how you're going to show the best possible result. So this is frequently where you'll see the heavyweight sort of machine learning models. Because there's a lot of computation. It's using user features and context features and item features, mixing them all together and coming up with a probability of how good this is, maybe even down to like a probability of purchase. And then every single time you're generating this, you keep hitting refresh page or different people are coming in or it's the next day, it's generating a new set to get sort of these rotations and achieving some of these other objectives, not just most relevant. And then some systems even go above and beyond this, which is blending. Most people will do a very simple blending. The simplest blending of all is just rank, or here's my score, just rank by score. Okay, that's a very simple blend, but you can get pretty sophisticated about this. And this is where you start getting into ad load or promotion load or diversity controls or merchandising. And you can even come up with ML models for this as well. The algorithm of it is a bit more complex. Usually it's, you're limited down like maybe like 100 items or 20 or so. And then as you're inserting items, you'd be checking to see the things that have already been allocated when you're making decisions for the next thing to show. And you can do that either as usually greedy for computational reasons, but you can even try to do it in one pass for a sequence-to-slate kind of algorithm. It's a bit of a tangent, we can do that in Q&A. So in summary, retrieval, ranking, blending. Retrieval is recall, ranking is precision, and then blending is usually optimizing for some sort of concept of user LTV, the user experience, the entire experience. And this is the architecture used at most big tech companies. So I see some people nodding along. If you've been at a company, this is how they set it up. Diagram of just what I said. You have your document corpus, the millions, billions, maybe dynamic. First stage, usually first stage recall, ranking. Then you have multiple re-rankers potentially. And then finally the presentation. In our system, this is a diagram I think off of our page. We have customers who are actually using Algolia as relevance. And then we ask them to send multiple queries to Algolia for like new and search relevance. You have personalized recommendations, new item discovery, ads and sponsored search, maybe internal merchandising. And these all go into second stage ranking and then eventually third stage ranking and it goes to the user display. And then going back to something I said about second stage is typically where you see the fancy machine learning models. Here's actually a slide out of my ML lead June's presentation on our architecture, you'll have image embeddings and network embedding, text embeddings, ID embeddings. You have counter features, all sorts of other features about, is this a good match or not? And they all get mixed together and go into some sort of neural network. It comes up with a score. We use a multitask model of prediction click, prediction to conversion, but different people are doing different things. So typically the heavyweight machine learning is happening in second stage. First stage is information retrieval. And you'll see a lot of machine learning in like generating the embeddings and then fetching those. And then also some other heuristics around text matching. Third stage is more niche, but that's more of like rules or ad load you'll start seeing here, or diversity rules or things that depend on the other things that are already allocated. So I want to make sure we have enough time for questions. So these next 2 pieces are a little bit faster. I'm going to just talk a little bit about some of the challenges of when, okay, now you have this sort of matching problem as opposed to just relevance problem. What are the other artifacts of this? So data. At small scale, regular analytics for business work just fine. You just need page views and purchases really. What are people still looking at? What pages did they see? And there's a variety of different tools. I picked 3 off the top of my head, Mixpanel, Segment, Google Analytics, but there's lots of these. These all sort of work. I see scale. Now every page is dynamic and you have these questions of what's being seen. People lose track of who's first in search. Where am I in search rankings? Where am I showing up in feed and recommendations? It's not just the destinations that are being seen, but how did they get there in the full path? And if I have a list of 20 different things I'm showing to someone, where am I showing up on that list? It's not enough just to say this product showed up on this search page. And so page views are only telling you destinations or really clicks, the destination of a click, not search and feed rankings. And so search and feed ranking is a critical piece of your product. You're missing a huge piece of it. And you also have many different dimensions across, going back to all the different interactions across users and demographics, across time, seasonality, or even time of day or countries, even queries and products. So some of you've probably experienced, I don't know if you've had to do AWS cost optimization or you ever got a bill from Segment or Snowflake and such, like your data infrastructure bills will get gigantic. And queries will get huge and really slow and fail to execute. So you won't even be able to do internal analytics. Forget if you're trying to just report this back to sellers. If a query takes 2 hours to execute and half the time it fails, you have to restart it. That may fly if that's for your internal reporting team. They may not like it, but they may be able to tolerate it. But for seller reporting, external reporting, that's, that's not going to work. And so, gosh, this is actually somewhat unsolved. Facebook and Google and Amazon have solutions for this. And there also exists a variety of very high enterprise vendors like Snowflake that can help solve for this and build out an appropriate data lake and a data warehouse so you can build your applications on top of it. But what I have seen in my own practice, and this is something we kind of learned the hard way, At, at Promoted is it's not so much the super fancy machine learning that is a blocker for getting the extra efficiency. It's that lack of transparency and perceived control from not having sufficient data reporting. That's, that's a top blocker in optimizing marketplaces. Like the operational load and loss of control becomes so overwhelming for the organization to handle and also like a seller experience. That if you will not launch something like some sort of machine learning thing that increases revenue by like 3%, let's say 3%, 3% would be great. Like if you're a scaled marketplace, that's a lot of revenue. It won't get launched because people can't understand. People will complain. Like you launch it and it's like, oh, top sellers are not first anymore and we don't know why. So this lack of transparency and explanation is, we internally benchmark it at roughly 10%, which is shocking. Just disgustingly high. Like, oh my God, you have to increase total efficiency by 10% to overcome this sort of operational thrash from not having sufficient reporting and transparency. Yeah. And that's just not for vendors. That's for internal teams too, especially internal teams. Then ads. So ads are fun. So at small scale, look, how hard is it to put something at the top of a page and charge some amount of money for it? It's not hard. Anybody can do that. And there are products out there that do a great job. I always recommend Kevl if you just want to put something at the top or something relatively straightforward. You have like at most like 50 ads or something to show and you have like a banner and et cetera, et cetera, et cetera. Google Ads Manager will love to sell you something. Most people use Google Ads Manager, and if they don't like Google, they'll use someone like Kettle. And Google, of course, has their own inventory. So like, hey, you know, get extra revenue, but these work fine. Scaled ads, Amazon, $150 billion in just Q4 of last year. They're doing something differently. They're not—they're not just doing a banner and at the top. I want to talk a little bit about that. So, going back to that work example of Amazon Search, why are all of the listings ads? These are both sponsored search. And like another part of this is why—is Duracell having to buy an ad for their own search. I literally searched for exactly this product of this brand and they had to buy an ad for it. And then the second one is Energizer. Like, that's not Duracell. Why did that show up and why is it an ad? So something else is going on here than just like a typical sort of banner ad placement. There's a different, it's a different business really. And what I'd really like to communicate for the whole marketplace community who are thinking of doing ads is it's fine if you wanna do some sort of brand placement ad, you wanna do some sort of fixed placement. That's a new line of business that's really generating some extra revenue for your extra attention and is driven by sales, really sales and like brand awareness of what you're doing. But these are really not ads in some sense. Like these ads, there's, um, yes, Amazon has a sales team, but the way these are being presented and the way this fits into the product is not really like an ad that you would see on, I don't know, network television, for example. What you can think of this more as profit optimization for Marketplace. So this is taking the matching challenge that we talked about. And taking it to the next extreme of how do you come up with the most take rate out of this, these sellers to get the most profits out of your marketplace fairly. So you can think of marketplace ads or like advanced, sufficiently advanced performance ads on e-commerce as profit optimization. And so therefore it's really not a new line of business. It's really more just an extension of their existing retail business. And to say that $150 billion Amazon could just do ads, it kind of doesn't make any sense because really this is just a way for people to compete for higher volumes with a higher take rate. Ads are hard and why are they hard? So performance native ads are hard. And I talked a little bit about brand ads are technically easy. The hard part is really on the sales side. You need to have the whole brand sales side of it and relationships. Because why is it hard? You have to solve the first thing we talked about, scaled search and all of the matching problems. Then you need the data side. No one is going to buy your performance ads if you cannot solve some of the data transparency pieces that we were talking about in step 2. And not only that, you're taking people's money in advance. So they're going to be super serious that those metrics are correct. And if you're not correct, You will show up in the Wall Street Journal. You'll get sued. I mean, it happened to Facebook all the time. Like you see that every once in a while. And you're effectively raising your take rates and degrading the user experience, the buyer experience, maybe just through increased prices to get more revenue. So, wow, you better be doing that with really good reasons and carefully control that because people don't like it. Of course, your buyers don't like it. No one likes ads. People complain about sponsored search. Sellers aren't going to like it either. Why am I going to pay you higher prices when I don't have to? So the bar for being able to do this is very hard, and that's why Amazon is good at it because they're the very best. So you need to have solved scaled search and you have need to solve scaled data. And then you can start adding in some of these competitive pricing controls to extract that extra profits. And so you'll hear this from experts here in the community of Everything Marketplaces. You'll hear from your venture capitalists, you'll hear it from all sorts of people. Don't do ads until you're scaled because it's really hard. And if you haven't solved those other 2 things first, really, really, really well, you're fooling yourself. You're just going to cannibalize your marketplace and you won't grow. So if your valuation is based on growing into something bigger than you are today, doing ads too early is going to be very bad for you. If you are trying to extract profits out of your marketplace, and your valuation is based on the size you are today or roughly that size, then ads can make more sense. And this is different from just like a banner ad or brand partnership or some of these other like side things that really don't really affect that too much of your core business and how people perceive you, unless you overdo it. Don't do that too. And by the way, here's the system diagram for how we do ads and how we design it. We put it in the second stage ranker. There are a lot of components here. You have organic ranking, you have organic retrieval, you have, and then on top of that, you have an ad auction, you have like allocation calibration, bid pacing, all gets logged down, log it for pricing and later optimization, and then you return it. So this is a system diagram of what a system could look like in production. So something that we do. So I think I kept it to 36 minutes. Well, we have the rest of the time for Q&A. Love to take questions from the community.

Yeah, no, this is really great. So, you know, thanks for walking us through. So I feel like it was definitely kind of a masterclass on, you know, search and also kind of ads at scale here. So a lot for us to digest. But yeah, we had a few questions. So hey, Dimitri, do you want to come on? You had a pretty good question you asked me.

Yeah, thanks. Yeah, Andrew, I had a question about as you're iterating on, let's say, a ranking algorithm, and you're developing new versions that you hope are much better. Are there any subtleties in testing new versions versus the old? Like, for example, in your Hipcamp example, if you're using user-level randomization where some users use the old one, new one, and the new one makes it easier to find the best campsites, then that sort of affects what's available to the control group. And, um, and the second part is, do any of these solutions that you recommended have that capability sort of built in? Well, the second part is no, and it's even challenging to do it on Facebook. I mean, that was again some of my background. I worked in marketing science department for some of the time I was there and doing the sort of careful measurement that you described is both challenging to do from a data science or even just explaining it to people, but technically being able to create these systems, like you can get all kinds of systems, like you have lift studies with counterfactuals for control group. It's a lot of infrastructure. It's also not free because you have opportunity costs on the infrastructure side for replacements. You can get interleaving for A/B experimentation for different rankers, for example. I'm not aware of things out of the box that do any of these kind of things. Generally, these are all the domains of people who have in-house teams and are doing them. From our perspective in the products that we provide. So a little bit about Promoter. We focus on the enterprise market. So like, what if Algolia is great, but you need more? Like you're starting to ask questions like, oh, what about interleaving? Or like, yeah. Or lift studies. And you like know about that. So we serve that market. The way that I handle this internally is there's a lot of this experimentation that's to the service of functioning the That for operating the engineering and data science organization, and a lot of which doesn't actually serve the business. It's actually serving the organization process of launch committees and getting things launched. So the way that we handle this internally is as much as possible, we avoid doing the sort of work that would be required to do a very careful sort of experiment review and then automate it. So if you can, Actually get all of the metrics and here's what your launch objectives are, like retraining models, for example, or adding features. We just automatically re-add all of the features every day and run it in a pipeline. And yeah, it's computationally expensive, but we're good engineers. So like, it's a lot cheaper to pay the computer cost than the people cost. It's the opposite of bigger companies. Like every single feature you do, you need to say, okay, here's my roadmap. Here's the incremental gain from this individual feature. We're going to do an A/B test for this. I am credited 0.1% of increasing revenue because I did this feature engineering work. We launched it in a model. This is not exactly answering your question. I think the first one, because it's kind of trying to get away from all of the extra work that comes in a huge organization to try and justify the value of the work that you're doing. And rather just try to focus on making the best possible result all the time using machines to do it. But then you lose some of the transparency of what exactly of the signals and who did that work in terms of assigning credits for different teams. So that's as a philosophy, one way we think of it, having AutoML auto automatically launch the best possible model, always testing multiple models at the same time, but we lose the ability to say which particular thing was credited for what amount of A/B production experiment results. The other thing that we think of is You mentioned something specifically on ranker and this concept of there exists a better ranker. And then you can do inter— I mean, I said interleaving, but it sounded like you were trying to get into a little bit some of that, like the experimental design for doing statistical power or different biases that come up in measurement. We think of it differently. We try to think of economics and machine learning as software engineering. So this idea of each individual model is trying to produce independently an interface, a signal that this, this like a click prediction model, this click prediction model is just trying to predict clicks as best as it can. That doesn't mean that you want to rank by clicks necessarily. It's just responsible for predicting clicks. And then that becomes an input to another system. Like it could be an allocator system, which is trying to come up with the right mixture of things that are relevant, things that are very expensive and things that are cheap or ads. So ad click is very component, important component for like quality scores, important for like complementary bids and quality scores for ads. It can be itself a ranker. So like now you're getting away from, there's just one big black box and like a biology experiment of You can't get into the individual components and evaluate them independently and try to break it down into individual components that can be evaluated independently as quality, even though the composition of them can be dynamic. So another example of this sort of composition, like we have click prediction, we have like conversion prediction, we have like price, and then like the final allocation. It's not just a ranker, it's, oh, I have ads, I have expensive things, I have cheap things. You're trying to maximize user LTV. But I don't have to change my click model if I change my allocator because it's just consuming the click prediction as a signal. I can change the model that produces the click model, make a better click model, slightly better click model without changing the downstream consumers because the interface is consistent. So this is a thinking from software engineering that can be applied to economic systems and the machine learning systems. Economic system. Like I can still bid per click and change the bid model or change the click model. I go from a GBDT to a neural network with some new signals and I don't have to change my economic design, for example. So that's the other way we get around some of the things that you're talking about of how do you manage all of this complexity and not have to retune and put too much load on the sort of final A/B experimentation. make any kind of decisioning because that's really slow and can take a very long time. And sometimes it grinds organizations to a halt, something I've seen at bigger organizations for search and recommendations and marketplaces and social media.

Good question. Took us a little bit deeper. So cool. Hey, Colton, do you want to come on? I know you sent me a message with a question. Yeah, thanks, Mike.

So my question is more directed towards what you would recommend for smaller scale organizations around, you know, when you're in your first handful of iterations of your product and you don't have a lot of data to go off of and you're sitting here wondering what is the best search experience for my users?

What are some of the questions you ask yourself about maybe the product that you have or the users that you're serving to figure out, maybe get a little bit closer to the correct search experience and not have to go through so many learning curves?

I would say do the work, like sit down for 8 hours and go through all of your top queries and make a spreadsheet. And then you can like make some scores. Like here are some of the basic components, like how many times does this get purchased? How many reviews does this have? And you can like change the rankings in a spreadsheet and then do it manually. And literally we have this product, like we have this product, you export the search result. Into a spreadsheet. And here's like click prediction. It gets fancy. Like we have thousands of columns or whatever, trying to trim it down to 100 or something. But you can do this at smaller scales as well. I would say, try to avoid getting abstract for as long as possible and be super reductive about it. Show me my top 10 queries, probably drive 80% of your traffic or your top 10 screens. What are the things being listed out today? How about tomorrow? How about the next day? Do they look good? Can you manually fix it? And try to do that for as long as possible. That's my recommendation. Don't be tied into like, oh, you feel like, oh, we're not a serious scalable company if we're not doing all this ML stuff. I mean, you can tell your investor slide or whatever you're going to do in the future, but don't do that.

What about the difference between guided discovery and like an actual search?

They are implemented in largely the same way. The retrieval query is different. That's another thing I recommend to people is don't think of recommendations or merchandising or search as like totally separate things. You literally can implement recommendations as select for me the top things or like the top or some sort of recommender score for users. And then you apply all of the same sort of metrics and same sort of other ways you evaluate search quality on recommendations as well. Or like some sort of stratified search, like, okay, this user likes these category A, B, and C. Okay, fetch categories A, B, and C, just like literally a query in your query system for A category, B category, C, and then interleave them 1, 2, 3, 1, 2, 3, 1, 2, 3, or A, B, C, B, C, B, C. Maybe like some sort of ranking score. So I would, the blend of super specific search to homepage query-free discovery, there's a cline. So there's like all the way from like more of like exploratory sort of discovery sort of queries to like, I don't know, personalized feed to like home feed, but this technology can be shared across all of these different stacks and the way evaluating could be thought of as the same way as well.

Awesome. Well, this is such a great workshop. It's gonna make for a really good recap post in the community with the recording and notes, which I'll share kind of after this, hopefully here pretty soon. So thanks again for taking the time to join us here. And I'll also include more to promote it in your email if that sounds good.

That'd be great. Thank you so much.