How Shopify Implements AI Across Sales and Product with the Head of AI at Shopify

Please join me in welcoming two amazing speakers. We have Radhina Sassuri. She is a managing partner at Glasswing Ventures and Mike Tamir, head of AI at Shopify.

They are going to be talking about how Shopify is implementing an AI system across their sales and product team. So we are so thrilled to have both Radhina and Mike dive in a little bit more to what's going on in the ecosystem and hear some real-world examples of how Mike is doing this work at Shopify.

Thank you so much, Taylor. Hello everyone. What Mike and I thought about doing today is How does one think about AI adoption both at an individual level in a business context, as well as the critical work that Mike is doing at Shopify and has previously done at Uber, Saskahana, amongst others, as well as in the cutting-edge work he does, um, teaching at UC Berkeley. So by way of background, um, I'm managing partner at Glasswing Ventures. We are an AI originalist We in fact founded Glasswing back in 2016 with the goal of investing in AI-native companies that solve real problems for the enterprise and cybersecurity markets. Mike, with that, would you like to do a little bit more on your background? I think I would be doing it a disservice.

Oh, I think you did, you did more, more than enough. But just to recap, I'm Ed Bai, one of the distinguished ML scientists and running ML and AI functions. Here.

Lovely. So let's delve in the first part of this conversation. I will do a bit of level setting around what's going on with AI from an enterprise adoption and where we are in that cycle. An important level setting point is the fact that today there are many initiatives across large and mid-sized enterprises around AI. In fact, we tell our AI-native companies not to lead with AI when they're speaking to prospective customers because it's such a top-of-mind topic that it's a superficial indication of interest. But most of the engagement, with few notable exceptions, is still at that superficial level with early sort of measurements, if any, on the impact on the enterprise on an adoption basis. Now, we have seen a few green shoots, if you will, in the space, particularly, for example, what we saw with Klaviyo and the impact that they have had by bringing AI from a project siloed or customer interface type level usage to their core business where they automated 700 humans and over two-thirds of customer success is now AI supported and generated. And contributed $40 million to their bottom line. And it is because they brought it to their core product, to their core business. And as we expect in the mid and long term, the impact is sizable, but they are noteworthy because they are one of the few companies. And again, Mike will talk to what Shopify is doing as another leader in that category that have brought AI to the core business. So what is the mindset? What is the actual work that needs to be done around building the infrastructure to leverage AI for the core business and the highest impact? I would articulate it along three dimensions. One is software and the technology infrastructure and how, whether it's the debt, whether it's the tech decisions, whether it's the data availability, et cetera. But getting the enterprise or the facets of the business to be AI ready. The other piece is really around cleaning the data. So one is the algorithmic side of technology side. One is the data that you are going to feed. And oftentimes there is an almost naive assumption that the data will be easy to leverage. Oh, it's an asset and we'll use it and leverage it. And it turns out that oftentimes is the biggest problem. And thirdly, culture. Culture of adopting AI. And again, we'll delve more into this as, as we proceed in the discussion. So where are we heading therefore, as, as we try to adopt AI in the enterprise? I have a term that I use called ambient AI, which is basically that we, let's think of ourselves as professionals and productivity sort of driven professionals. We will be surrounded in and leveraging AI in this copilot dynamic. And for one to wrap one's mind around what does it mean for me, um, to be leveraging ambient AI, if the input is you, the business individual leader contributor, your activities and your environment, and you think of that as the input into ambient, into your existing AI tools that you've surrounded yourself for a copiloting dynamic. The outputs are insights that you wouldn't have been able to derive on your own. And secondly, the automation of the mundane, the low productivity, highly repeatable, to free one up to be more valuable. So as we think about our roles as professionals and what will make us successful going forward, worth considering that in addition to the well-tried high impact, high productivity, the people and the professionals will have one leg up. If in fact they are leveraging the tools, if in fact they are intelligent business workers, not simply in the intellectual sense, but leveraging the intelligent tools. And as we shift to the enterprise, it's really these topics of data, infrastructure, and culture. And I will use this layout and framework to turn it over to Mike to talk about the impact starting with the data. My first question stated here, in fact, how do you know if you have the right data? And is data really, the claim that I made is that oftentimes the biggest bottleneck and the biggest problem. For this business audience, not necessarily a technical audience, how do we think about that change first?

Absolutely. So you couldn't be more correct that data is often the messiest and the time that you think You're, you're, you know, while we have the data, why don't we just get to the building the models? When in reality it's one of these iceberg clichés where most of the time you spend is on gathering the data and cleaning the data and making sure that it's ready, that it's of the right quality. And even once you begin the modeling, you're not done. Modeling is really an iterative process where you train your models, you see what signal you're getting from it, you see how good your performance is in this speaks to measurement and making sure that you have the right measures. And of course, every different ML technique is going to be slightly different, but very gener— generally, um, and certainly when you hear about LLMs and a lot of the modeling techniques that are in vogue, um, it's trial and error, right? A model will see an example of this is what information we have and this is what we're trying to predict. It'll take the information that you give it of what we have ahead of time and make its own prediction and then look at how far it missed, right? How far is what it tried to predict from the truth of what it should have predicted? And then it'll take that error, right? We, we, we hear about failure is the key to learning. This couldn't be more true than in machine learning. Machine learning literally takes that error and with a bit of math tries to adjust all the little parameters, all the little calculations that it makes along the way from input to output in order to make that error just a little bit smaller the next time. And this leads into, uh, a lot of the things that you have to do with your data as best practices. So if I take all of my training data and I teach my model with my training data to close that gap between its predictions and what it should have done, then I might be overfitting. So overfitting is if your, if your model has taken that time to close the gap between its predictions and what it should have done, maybe what it's done is just learned for those specific examples that you gave it to learn from. Right. And so it has overfit to those specific examples, but we don't wanna build models that can just take the inputs you give it and give the outputs that you told it to learn. We already have that data. What we want it to do is to learn the pattern. And to learn the pattern means we have to do all of these different things in order to, to constrain the model in various ways so that it predicts not just on the data that you're giving it, but it also predicts effectively on the data that it's never seen before. And we usually call that validation data and ultimately test data in different stages for figuring out when is, uh, how is this model going to predict accurately? Out of sample, out of the training sample.

And so if I am a P&L owner in XYZ business and I've had this data for 20 years and Mike's team is helping me sort this out, why is it taking them multiple weeks? What's so hard about it? What's so hard about it?

And there's two parts of this, which also speaks to, to, I think the culture, right? Especially in a world where you can talk to an LLM and it seems like it's giving you right answers and you want to say, you only live once, let's just use whatever the answers are. And I'm not going to name any deployments that have happened in the last several months, but we all know news headlines where maybe that didn't work out so well. So having the culture of checking your results and making sure that you have the right metrics to evaluate whether it's an LLM or a more traditional model that's not generating text or generating images, but just estimating numbers or estimating classes of whatever the task is, having that culture where you are focusing on what are your metrics and you're evaluating those results is really important. On the, why is it taking so long side of things, when you take the data, and we're going to have a very nice concrete example of this towards the end of this talk, it can be very, uh, the goal, the business goal that you have might be something like, I want to make more sales, or I want to delight my customer feed and customers and see that in feedback. Whatever that metric is, we have to translate that, speaking to the culture of measuring our results. We have to translate that to a concrete metric. And so this is going to be some sort of mathematical function of saying, how bad did my prediction miss from what I said the right answer was? And hopefully it's very close and you get some, some fireworks. Now figuring out that answer is actually very complicated. It's complicated in finance. It's complicated in producing recommendations to, or returning a search for a query. There, that gap between what I want to mathematically measure, which is how the machine is going to learn based on that error, and what my actual product result is, really matters. Another area where you really have to think about what matters is how the data is actually structured. If I give a bunch of examples where of this is the kind of input and this is output I would expect, or this is the kind of output I would not expect. That's going— the model is going to learn that specific pattern. So if you give it bad examples or you give it dirty examples, either examples with the wrong answers or examples where, where a negative case doesn't— isn't as informative as an almost positive case. And we're gonna get again to a concrete example of that soon. Having that clean data is really going to be the lifeblood of how the model learns and ultimately how the product is successful.

Yeah. All right. Continuing our discussion on the infrastructure side. If you are a productivity business user, you can only affect the decisions that get made at the infrastructure level so much. What should one know? What constitutes a modern AI stack that enterprises can scale on the basis of?

Ultimately, that's going to be a little bit dependent on the use case and the context, right? Every company is going to have slightly different needs and going to have slightly different applications. When it comes to doing the, the more advanced training and deep learning models that I've been describing, you're going to need GPUs for one thing, right? Which means you may need a single GPU and you're a very small startup and you're, you're trying to just work on a budget. You may need a cluster of GPUs, in which case you need infrastructure management. In the last several years, uh, there's been obviously a very big shortage of GPUs. And so the— it has historically been, uh, the, the existence of cloud platforms, AWS, GCP, Azure, has been very helpful in sporadic needs. Oh, I need a— I need to rent a GPU for a few hours to do my training, and then I don't need it anymore. Versus purchasing or long-term rental, right, where you reserve an instance for days or for months And that's gonna end up costing thousands of dollars in often time. So being able to size up what kind of training you need and then picking what level of, are you going to need it at hook? Are you going to need it as a reserved instance? Maybe you buy your own infrastructure, which is also a, you know, often cheaper answer if you have, if you already have an infrastructure, an on-prem infrastructure system. That's really going to depend on what your AI training expectations are anticipated.

So I'm going to throw a curveball. If we think about the migration that enterprises made from on-premise to the cloud, the underlying value prop was, look, the variability is fairly cheap. You can ramp up fairly fast. In fact, I have this term that I use that it's a variable fixed cost on your P&L, meaning that you can scale it up and down. The line will always be there for you as a business, and that's the cloud cost. As we move to the, a world of AI nativeness and leveraging it, is your belief that the existing incumbents will actually, whatever you're seeing with the large foundation models providers, will become the Swiss Army knife? So I'm speaking, for example, to the AWS and Amazon approach where they're saying, listen, you wanna use OpenAI, you wanna use Anthropic. We have our offering, we have Bedrock, you don't have to switch, it's going to be one more feeder. So does this line remain or do we have a new fixed variable line?

It's, it's a very good question. And certainly when you talk to AWS, you talk to GCP, they all have partnerships and obviously Azure very much so with, with OpenAI. They have these partnerships, but, and it's worth being aware of this, using a first-party solution with, say, OpenAI or with Anthropic is going to have different performance, different latency than if you do it via an intermediary like a cloud. So you maybe get some simplicity by not having to onboard different vendors and different, different support, but there is going to be an impact in terms of performance. And that's something that might be a temporary problem, is certainly a very real consideration.

Is that a high pain point in this moment in time, the delta in performance, or do we believe it persists? And your view is?

It's right now it's worth not ignoring for the use cases that I've had. And something else, maybe hopefully transient for '24, is the option for ad hoc GPUs when you go higher than like an L4, if you need an A100 or an H100, something that can host some of these larger, large open source large language models. We talked about commodity models. What if you want to do something like a 70 billion LLaMA or MinSTROL or Mixtral or any of these? Those are going to be very hard to get on an ad hoc basis. And you're going to end up having to pay the price for a reserved instance if if you want to be able to serve something like that.

And those are, you've now touched on the trade-offs between using open source platforms versus existing walled garden incumbents. Culture. How do you effectively implement AI from a cultural process people point of view?

One of the both beautiful but also double-edged things that is happening with the ease of libraries, in the olden days, 10 years ago, if you wanted to, if you wanted to, to, to put together a model, a neural net model, you would probably have to layer it, pull it, put it up layer by layer. You'd have to design it yourself. The, for the most part now you get a pre-trained, something that is to say a model that somebody else has trained, and then you're just going to tweak its decisions a little bit to adapt it to your particular data. Or the days of coding up your own specific classification model or, or there's a whole host of supervised and unsupervised, et cetera, models that you might wanna do. The days of having to have ML experts do that by hand, or that usually doesn't happen. At most, you're doing tweaks, you're doing minor corrections. And what that means also is that you can have non-experts who are not used to a lot of the things that I've already touched on. Like, how do you make sure that your model's not overfitting? How do you make sure that your data is clean? How do you trust and have the right metrics so that you don't have an embarrassing situation where your LLM offered free X of service, and now you're on the hook for actually backing up whatever your LLM did, even though it wasn't trained or aligned, it's called, for that particular mistake. So enabling your engineers in order to leverage all of these great use cases, including the models that seem to take care of themselves sometimes deceptively, and making sure that you still have very rigorous metric supervision and evaluation of how your models are doing and when the worst cases are gonna happen. Making sure that you stratify— another technical term— that means you, you consider all the different situations carved up into little boxes so that you make sure you're not just performing well on one, you're performing across the board of all of the different use cases. Having that strong culture to back up the performance of your model before you go into release is probably one of the biggest issues in terms of having a strong ML culture when you're, as it gets easier and easier for non-experts to start leveraging these tools.

And if in a 5-minute conversation with a business owner, you have the opportunity to completely overwhelm them by throwing all sorts of technical ML terms, if that is not what they have done day in and day out, but they're very good at running the P&L and all of a sudden we are in a new paradigm. How can they stress test what you are doing? They don't speak your language. You throw the 3-letter acronyms, like where do they find themselves?

Oh, that's a wonderful question. I, I mean, I love it. Yeah. So, so it's very easy to throw technical acronyms and to maybe go deeper than is really necessary when you're coming from a technical perspective. Understanding exactly what a transformer model does or what a multi-head attention network does and why you need rotary versus non-rotary embed— embed positional embeddings or different softmax loss functions and temperature, that honestly doesn't make a big difference in terms of making the business decision of, is this model going to perform the way I want? Is it going to perform out of sample? That is to say, when we're live. Are we cleaning our data? Are we using, or do we have metrics that match what the model was trained to do? Our loss function, our leading metric, how did my prediction match the truth versus what the product is supposed to do? And asking questions about why those match is going to be the most valuable thing for you to dig in on, because that's what's ultimately going to determine the performance of how your product that's AI-driven is going to succeed.

I couldn't agree more. So let me turn it over to you to delve into some of these examples.

Yeah, Shopify is one of, one of the largest e-commerce platforms in North America. And in particular, that means making sure that we can answer a fundamental question of if a buyer has a query and is searching for a query, which products would match that query, right? Which ones? Are relevant matches and which ones are irrelevant matches. Now that becomes relevant when there's a particular merchant who has a single shop who is using Shopify and wants to— you want to— and if somebody goes to that shop and wants to search for particular product products within that shop. We also have a Shop app where you're searching much other e-commerce experience across multiple merchants and multiple products that are hosted across several different providers. And so search relevance in those search contexts, those kind of search contexts that as an end user you're very familiar with, are first and foremost very important. Now, there's also just making sure that you have high quality, right? If I search for boots and I end up with boot socks and boots and ski, ski boots and a lot of things that might not be relevant, I need to be able to evaluate that and I need to be able to evaluate that, say, on the first and different results to make sure that when we do these key searches, we're giving the end user buyers the right kinds of results. And to do that, we need to have an evaluation method, right? How does this product match with this query? And again, this goes back to what I've been banging the drum on all along is you need to care about your metrics, right? So this is as much a metric as a product itself.

Mike, if I may interject, because I think we can't stress the metric point enough. It's also not all metrics are created equal across different businesses. So what's a good enough metric? I'm going to make up a basic example. If I search for an item on Shopify, the boot example, and I get the right item 50% of the time, that might be great or not. You will tell us what's relevant in one context. but completely insufficient in another context. So tell us a little bit about not all results are created equal back to this metrics point. Context matters.

Sure. Yeah. And maybe to your point, in product search, there, there's two dimensions that there are two, two principal directions that you might want to have relevance in, right? One might be, I'm looking for a product. And so I want to see that product. I want, I'm looking for boots. I want boots. Another might be, I'm shopping for boots, and so I'm going to buy boots, and I'm going to buy socks that match the boots, and I'm going to buy maybe some cleaner that, that makes sure that my boots stay in nice condition in the weather, in the wintertime. So cross-selling and relevance, right? And these can actually be two different metrics. And maybe what you would want to do is have one metric that shows for direct relevance and another that shows complementary relevance, right? And then it is somewhat a product decision of how much you wanna balance these, but it's also something that you can balance with GM, with, with, with your, your bottom line, right? With how much do you know if we balance for, say, 50% are gonna be direct on the nose relevant, 50% are going to be more complementary. Maybe if we tweak that to 60/40 or 70/30, you might actually see a change in your bottom line. And that's the sort of, that's the way you're products, your end goal is going to be determined by certain parameters that you choose in your actual model deployment.

And so back to this interdependency between your teams and your organization and the, the P&L owner or the merchandiser, whoever is responsible for the business unit and that bottom line, how does that interaction back and forth work? Whatever tweaking you are doing on 40 or 50% or 2% improvement has ramifications in terms of their bottom line, but also the resources that you have leveraged. So if we wanted to, we expressed the metric in terms of results for the end customer in this example. What about from an ROI perspective? How much do you have to put in to get what out?

Yeah. So there's a couple aspects to that. One is as a data person, um, Obviously I love seeing the numbers and letting the numbers tell me. And so having experimentation, being able to experiment with, to take the simple example of how do you balance these two kinds of experiences is one answer. And then you need to invest in experimentation frameworks, right? And hopefully, you know, again, these things get easier and easier as there's existing open source frameworks and tools and vendors that you can leverage for that. That's one answer. The other answer is really keeping close communication with your stakeholders. And this really speaks to the, the, the culture aspect of it, right? Just having ML, ML engineers and ML scientists working in a vacuum and coming up and saying, this is your model, I'm gonna go work on the next fun technical thing, is a mistake. And it's been a mistake. And we've seen different versions of this back when, you know, my job title was called data scientist and that was all, all that was the only kind of title there was. You know, what you need is to have a pretty mix. You need to have engineering for serving, you need to have your ML scientists and engineers, and you need to have product. You need to have product every— in the trenches with your ML builders so that way they can understand what are the pros and cons of not just making a tweak in a parameter and what a metric is, but also that key, that key value that I keep driving home, which is does your end goal product metric and what you want to get there match your metrics that your model is trying to learn from. And that's going to make a huge difference in terms of what you actually see when you get your bottom line results.

Got it. Vectorization and multimodal LLMs. LLMs, most people have heard in one form or another. Multimodal is initially we started with text-to-text, now it's text to image, text to video, 2D to 3D, et cetera. Vectorization in plain English, and then how that come— there's relevance in the context of the exchange we've just had.

Yeah, we had in the good old days, it was keyword search, right? And people have learned, oh, I just put in one— in the good old days, maybe 15 years ago, when you're searching, you search for one word. You don't really phrase it in a sentence. You don't talk to Google that way. At least you used to not to do that because it's just looking for the main words. It's ignoring the, we call the unimportant words and trying to match those in the products or whatever, or whatever it is that you want to do, you want to search for, right? The documents, if you're doing, if you're doing a search on web pages. For me, it's a somewhat complex topic, which really can be boiled down to, to a very simple image, which is in a multidimensional space. And you can imagine 3, every word. Word in its context and every image in its context can be represented as a point in that space. Mm-hmm. And if you're doing your vectorization correctly, then similar words from similar context or similar images are going to end up located in that similar part of the space, right? So all your cat pictures are going to end up here, all your dog pictures are gonna end up close to the cat pictures, but not the same place. All of your, I don't know, fireworks predictions, we've seen a lot of those, are gonna end up way far away from those. And the words corresponding to those are going to be in similar, if you do it right, your image vectorization where your pictures end up as a point in that space are going to end up in the same part of that space as your words. So if I describe cat, it's gonna end up where the cat pictures are. If I, if I describe a dog, it's gonna end up where the dog pictures are. And when I say in context, what do I— what if I say bank? Once I say bank, I could say the robbers were leaving the bank and they were going at high speed and crashed into the riverbank. And I've said bank twice here, and those are very different words, right? One is going to be in the con— the second one in that context is a place by a river. The first one is a place that has money. And so modern techniques for vectorization have done a very good job of looking at the rest of the sentence, looking at that context and figuring out that the first instance of the word goes where with pictures of industry, industrial places that manage money. And the other one has pictures of rivers and that sort of thing in it.

What Mike just described is about 80% of the backbone of GenAI and why we have the ability to interact with some of the applications in plain English and get human-like interactions because leveraging these vector technologies and delivering what's called embeddings, which is really leveraging the vectorization through deep learning, just so that folks actually internalize what this all means.

Yeah. And when you hear embeddings, it's pretty much a synonym to how I'm using the word vectorization. It's embedding the word or the image into a vector space.

So Mike, you're using this on how you're constructing the, whether it's the merchandising, the images, the text, the descriptors. Pricing? Are you leveraging AI for dynamic pricing of any kind?

So dynamic pricing is a complicated area. And very generally, I know from past lives, you never want to do dynamic pricing. You want to do dynamic discounts as a, as a product framing bit of advice. Now, certainly having tools helping merchants to understand things like demand and supply curves and how that those trade-offs are, that's certainly something that, that shopify is interested in providing for merchants as, as the first-party customers of Shopify. We want to give them the tools so they can be effective entrepreneurs. Real quick, just to frame the evolution of what's going on here, right? We started with keyword matching, then we might compare vectors, right? And typically I had on the previous slide cross-encoding and Two Towers. These are just very technical neural network architectures where we actually look at the vectors at the same time. Versus looking at the vectors separately. And if you want, if I do a query, I don't want to have to compare that query to every single product across all of Shopify, let alone even maybe in a single large merchant. And because it's not scalable, if you have hundreds of millions to billions of products to search for, you can't do that.

No, scalable, we mean it's very expensive to, from a compute perspective. Yep.

Expensive. Yeah. Expensive. And, and obviously prohibitively, it would take too much time in order to return it that quickly.

Time and dollars.

So that's one side of, uh, and that, and that is in terms of serving, in terms of giving results when somebody, uh, produces a query, you're, you're going to want to sever those. Now, if you want to just look at quality and you can do just smaller quality checks, then you can look at them simultaneously and you can get a little bit more information. The same story is now true with LLMs. LLMs, if you domain adapt them, right? That is to say, you train them on your data and have them learn your specific behaviors that you want to see correctly, can be very good at also taking in a slower context. I, I look through this query, I have this product, are they relevant together? Or I'm, I have this product, what are the relevant features? What are the attributes? What are the, what are the sorts of structured features. If you have a t-shirt, you might, it might have a V-neck, it might have a, a turtleneck, it might have different colors. What sorts of things can I extract about this image and this product description that I can use in search queries down the line? And so LLMs are remarkably, almost too good to be true, effective at this sort of work. Good. What's the cons? They're very slow and they're very expensive. And if you pay a commodity LLM, which we're right now does usually beat out open source LLMs, you're going to end up paying per token and you can't scalably, if you have the right size of a business, do this for every query or every kind of use case. So what do you do? You might want to take an open source model and do what we call distillation. Distillation is a technical term. It means I have a small model, I have the answers that I've gotten my big model, I know what goes into the big model and what comes outta the big model. And so I train my little model to learn to mimic the big model answers, right?

And how's that different than fine-tuning?

Uh, so fine-tuning is you might have any size model. The parameters are already trained on a lot of data by someone else for their data. And that data might be close to yours, but it's not exactly the same data as yours. So if I'm going to fine-tune my model, I'm going to take my specific data and I'm going to adapt to my specific domain on my data, the base model. So for an LLM context, LLMs are really remarkably great at grammar and understanding the subtle, and these days understanding the subtleties of language and often doing multilingual things. And there's a lot of baseline things that I don't want to put in the work on that, right? That work has been done and it's been done very effectively and there's no re— to have no reason to have everyone be another OpenAI or another Anthropic when it comes to baseline. But when it comes to my specific data, whatever it is, I'm going to want it to be extra, extra good at those nuances. And there are all sorts of ways of fine-tuning. There's all sorts of ways of— there's a whole lot of technical literature on different ways that you can adapt your model to your specific data. That's fine-tuning is what the main one.

And so back to your example, you were going to do the LLM, the large language model. And then almost like a small language model that's separate from fine-tuning. You're fine-tuning on top of the LLM. This is above and beyond, sits on top of it. I just wanna make sure people have an image of it.

Yeah.

Okay.

Yeah. So you fine-tune your smaller open source, and when I say smaller, it might be 7 to 70 billion, not so small, just small compared to the bigger ones, right? You might fine-tune it in order to mimic the performance of the large commodity model.

Got it. I know we have some more context, but what do you wish I had asked you about your work that we didn't get to?

Maybe it'd be worthwhile just spending 60 seconds to talk about the difference, about the data issues, because we, we Easter egged that.

Yeah.

So in particular, let's think about search results. How do you get data? You want to train your model to produce good examples for a query and know how to avoid bad examples for a query. How do you train your model to do that? There's lots of ways we can do it. We can have humans do it and come up with thousands and thousands of examples, and that could be expensive. We can ask an LLM to do that, and that is a new dimension in terms of data management that's now available. But now we're trusting our LLM for the answers that we are gonna build our entire foundations on. And so you need to have other ways of validating that information, making sure that it's not wrong, that you're not just trusting the LLM.

But to avoid what's come to be known hallucin— as hallucinations —in case—

yeah, absolutely. And so with your— so you could also use your own data, right? You can have positive examples. Oh, those are the things I searched for something, I bought it, I clicked on it, I bought it. That's definitely a success. It gets really subtle when you have negative examples, right? Because if I search for boots and I show a Christmas tree, that's not gonna— the model's not gonna learn much from saying, oh, that was wrong, right? It's gonna learn a lot more by showing it sneakers or by showing it sandals and saying that's because it's closer to what would ostensibly be correct. So thinking about that data and working with product, working with how do we carve up the different space of possible answers and understanding how you can think about what's negative example versus what's a positive example in combination with maybe asking your LLM to find things that are close, but not so, not, but no cigar, so to speak. That's really where a lot of this, we talked about data work is the bottom of the iceberg. I think that takes so much time. That's where a lot of the hard work comes in.

And any prospects for automating the data prep and what data scientists and data engineers do because it's so laborious and boring for them and they're so expensive and can create so much value?

Yeah, and I, it's, it's a, a little bit of a, of a Scylla and Charybdis, right? There's a balance here between trusting your, trusting your LLM implicitly and never supervising it. Versus, versus using, you know, not using an LLM and then being stuck paying humans to do things that maybe an LLM can do for you. And so finding out different layers of maybe I'm going to generate some candidates with my LLM and then I'm gonna have humans evaluate the quality of their results. Or maybe I'm gonna have LLMs find the potential negative values and, but have humans actually curate those values and make sure that they're actually good negative values. Balances of those and ask, and then paying, you know, the, the people that you spend the big bucks hiring in order to do the statistics and make sure that this, this layering of strategies is correct. That's a great place for them to spend time rather than coding up neural networks from scratch, which is a thing of the past.

So with that, I think we started our discussion where there is a lot of trial and error, and I think we're ending with trust and verify. But thank you so much, Mike. Rodina, thank you so much for the very informative session and for all the details. Thank you.