How Shopify Implements AI Across Sales and Product with the Head of AI at Shopify

Please join me in welcoming two amazing speakers. We have Rudina Sasori, she is a Managing Partner at Glasswing Ventures and Mike Tamir, Head of AI at Shopify.

They are going to be talking about.

How Shopify is implementing an AI system.

Across their sales and product team.

So we are so thrilled to have both Regina and Mike dive in a little bit more to what's going on in the ecosystem. And here are some real world examples.

Of how Mike is doing this work at Shopify.

Thank you so much Taylor. Hello everyone. What Mike and I thought about doing today is how does one think about AI adoption both at an individual level in a business context as well as the critical work that Mike is doing at Shopify and has previously done at Uber, Susquehanna amongst others, as well as in the cutting edge work he does teaching at UC Berkeley. So by way of background, I'm Managing Partner at Glasswing Ventures. We are an AI originalist firm. We in fact founded Glasswing back in 2016 with the goal of investing in AI and native companies that solve real problems for the enterprise and cybersecurity markets. Mike, with that, would you like to do a little bit more on your background? I think I would be doing it a disservice.

Oh I. I think you did. You did more, more than enough. But just to recap, I. I'm Ed Bar, one of the distinguished ML scientists in running ML and AI functions here.

Lovely. So let's delve in the first part of this conversation. I will do a bit of level setting around what's going on with AI from enterprise adoption and where we are in that cycle. An important level setting point is the fact that today there are many initiatives across large and mid sized enterprises around AI. In fact we tell our AI native companies not to lead with AI when they're speaking to prospective customers because it's such a top of mind topic. That is a superficial indication of interest. But most of the engagement, with few notable exceptions, is still at that superficial level with early sort of measurements, if any, on the impact on the enterprise on an adoption basis. Now we have seen a few green shoots if you will, in the space, particularly for example what we saw with Klaviyo and the impact that they have had by bringing AI from a project siloed or customer interface type level usage to their core business where they automated 700 humans with and over two thirds of customer Success is now AI supported and generated and contributed 40 million to their bottom line. And it is because they brought it to their core product, to their core business. And as we expect in the mid and long term the impact is sizable, but they are noteworthy because they are one of the few companies and again Mike will talk to what Shopify is doing as another leader in that category that have brought AI to the core business. So what is the mindset? What is the actual work that needs to be done around building the infrastructure to leverage AI for the core business and the highest impact? I would articulate it along three dimensions. One is software and the technology infrastructure and how whether it's the debt, whether it's the tech decisions, whether it's the data availability, et cetera, but getting the enterprise or the facets of the business to be AI ready, the other piece is really around cleaning the data. So one is the algorithmic side of technology, side one is the data that you are going to feed. And oftentimes there is a almost naive assumption that the data will be easy to leverage. Oh, it's an asset and we'll use it and leverage it. And it turns out that oftentimes is the biggest problem. And thirdly culture culture of adopting AI. And again we'll delve more into this as. As we proceed in the discussion. So where are we heading therefore as. As we try to adopt AI in the enterprise? I have a term that I use called ambient AI which is basically that we let's think of ourselves as professionals and productivity sort of driven professionals. We will be surrounded in and leveraging AI in this co pilot dynamic. And for one to wrap one's mind around what does it mean for me to be leveraging ambient AI if the input is you, the business, individual, leader, contributor, your activities and your environment and you think of that as the input into ambient into your existing AI tools that you've surrounded yourself or copilot in dynamic. The outputs are insights that you wouldn't have been able to derive on your own. And secondly the automation of the mundane, the low productivity, highly repeatable to free one up to be more valuable. So as we think about our roles as professionals and what will make us successful going forward, worth considering that in addition to the well tried high impact, high productivity people and the professionals will have one leg up if in fact they are leveraging the tools, if in fact they are intelligent business workers, not simply in the intellectual sense, but leveraging the intelligent tools. And as we shift to the enterprise it's really these topics of data infrastructure and culture. And I will use this layout and framework to turn it over to Mike to talk about the impact starting with the data. Mike, first question stated here in Fact, how do you know if you have the right data? And is data really. The claim that I made is oftentimes the biggest bottleneck and the biggest problem for this business audience, not necessarily a technical audience. How do we think about that change first?

Absolutely. So you couldn't be more correct. That data is often the messiest and the time that you think you're. While we have the data, why don't we just get to the building the models, when in reality it's one of these iceberg cliches where most of the time you spend is on gathering the data and cleaning the data and making sure that it's ready, that it's of the right quality. And even once you begin the modeling, you're not done. Modeling is really an iterative process where you train your models, you see what signal you're getting from it, you see how good your performance is. And this speaks to measurement and making sure that you have the right measures. And of course, every different ML technique is going to be slightly different, but very generic generally. And certainly when you hear about LLMs and a lot of the modeling techniques that are invoked, it's trial and error, right? A model will see an example of this is what information we have, and this is what we're trying to predict. It'll take the information that you give it of what we have ahead of time and make its own prediction. And then look at how far it missed, right? How far is what it tried to predict from the truth of what it should have predicted. And then it'll take that error, right. We, we, we hear about failure is the key to learning. This couldn't be more true than in machine learning. Machine learning literally takes that error and with a bit of math, tries to adjust all the little parameters, all the little calculations that it makes along the way from input to output in order to make that error just a little bit smaller the next time. And this leads into a lot of the things that you have to do with your data as best practices. So if I take all of my training data and I teach my model with my training data to close that gap between its predictions and what it should have done, then I might be overfitting. So overfitting is if you're, if your model has taken that time to close the gap between its predictions and what it should have done, maybe what it's done is just learned for those specific examples that you gave it to learn from, right? And so it has overfit to those specific examples. But we don't want to build Models that can just take the inputs you give it and give the outputs that you told it to learn. We already have that data. What we wanted to do is to learn the pattern. And to learn the pattern means we have to do all of these different things in order to, to constrain the model in various ways so that it predicts not just on the data that you're giving it, but it also predicts effectively on the data that it's never seen before. And we usually call that validation data and ultimately test data. Different stages for figuring out when is how is this model going to predict accurately out of sample, out of the training sample.

And so if I am a PNL owner in X, Y, Z business and I've had this data for 20 years, and Mike's team is helping me sort this out, why is it taking them multiple weeks? What's so hard about it? What's so hard about it?

And there's two parts of this which also speaks to, I think, the culture, right? Especially in a world where you can talk to an LLM and it seems like it's giving you right answers and you want to say, you only live once. Let's just use whatever the answers are. And I'm not going to name any deployments that have happened in the last several months, but we all know news headlines where maybe that didn't work out so well. So having the culture of checking your results and making sure that you have the right metrics to evaluate whether it's an LLM or a more traditional model that's not generating text or generating images, but just estimating numbers or estimating classes of whatever the task is. Having that culture where you are focusing on what are your metrics and you're evaluating those results is really important. On the why is it taking so long side of things when you take the data and we're going to have a very nice concrete example of this towards the end of this talk, it can be very. The goal, the business goal that you have might be something like, I want to make more sales or I want to delight my customer, feed customers and see that in feedback, whatever that metric is, we have to translate that. Speaking to the culture of measuring our results, we have to translate that to a concrete metric. And so this is going to be some sort of mathematical function of saying, how bad did my prediction miss from what I said the right answer was? And hopefully it's very close and you get some fireworks. Now, figuring out that answer is actually very complicated. It's complicated in finance. It's complicated in producing recommendations to or returning a search for a query there that gap between what I want to mathematically measure, which is how the machine is going to learn based on that error, and what my actual product result is really matters. Another area where you really have to think about what matters is how the data is actually structured. If I give a bunch of examples where of this is the kind of input and this is output I would expect or this is the kind of output I would not expect that's going the model is going to learn that specific pattern. So if you give it bad examples or you give it dirty examples, either examples with the wrong answers or examples where where a negative case doesn't isn't as informative as or almost positive case and we're going to get again to a concrete example of that soon. Having that clean data is really going to be the lifeblood of how the model works and ultimately how the product is successful.

All right, continuing our discussion on the infrastructure side, if you are a productivity business user, you can only affect the decisions that get made at the interest in the infrastructure level so much. What should one know what constitutes a modern AI stack that enterprises can scale.

On the basis of ultimately that's going to be a little bit dependent on the use case and the context, right? Every company is going to have slightly different needs and going to have slightly different applications when it comes to doing the more advanced training and deep learning models that I've been describing. You're going to need GPUs for one thing, right? Which means you may need a single GPU and you're a very small startup and you're, you're trying to just work on a budget. You may need a cluster of GPUs in which case you need infrastructure management. In the last several years there's been obviously a very big shortage of GPUs and so the, it has historically been the the existence of cloud platforms, aws, gcp, Azure has been very helpful in sporadic needs. Oh I need a, I need to rent a GPU for a few hours to do my training and then I don't need it anymore versus purchasing or long term rental, right where you reserve an instance for days or for months and that's going to end up costing thousands of dollars in oftentimes. So being able to size up what kind of training you need and then picking what level of are you going to need it ad hoc? Are you going to need it as a resurgence? Maybe you buy your own infrastructure, which is also a often cheaper answer if you have, if you already have an infrastructure, an on prem infrastructure system, that's really going to depend on what your AI training expectations are anticipated.

So I'm going to throw a curveball. If we think about the migration that enterprises made from on premise to the cloud, the underlying value prop was log. The variability is fairly cheap, you can ramp up fairly fast. In fact I have this term that I use that it's a variable fixed cost on your P and L meaning that you can scale it up and down the line will always be there for you as a business and that's the cloud cost. As we move to the world of AI nativeness and leveraging, it is your belief that the existing incumbents will actually whatever you're seeing with the large foundation models providers will become the Swiss army knife. So I'm speaking for example to the AWS and Amazon approach where they're saying listen, you want to use OpenAI, you want to use Anthropic, we have our offering, we have bedrock, you don't have to switch, it's going to be one more feeder. So does this line remain or do we have a new fixed variable line?

It's, it's a very good question and certainly when you talk to aws, you talk to gcp, they all have partnerships and obviously Azure very much so with, with OpenAI they have these partnerships but and it's worth being aware of this using a first party solution, let's say OpenAI or with anthropic is going to have different performance, different latency than if you do it via an intermediary like a cloud. So you maybe get some simplicity by not having to onboard different vendors and different, different support. But there is going to be an impact in terms of performance and that's something that might be a temporary problem is certainly a very real consideration.

Is it a high pain point in this moment in time, the delta in performance or do we believe it persists.

And your view is it's right now it's worth not ignoring for the use cases that I've had and something else maybe hopefully transient for 24 is the option for ad hoc GPUs when you go higher than like an L4 if you need an A100 or an H100, something that can host some of these larger open source large language models. We talked about commodity models. What if you want to do something like a 70 billion llama or minstrel or mix or any of these? Those are going to be very hard to get on an ad hoc basis and you're going to end up having to pay the price for a reserve needs and if you want to be.

Able to serve something like that, and those are you've now touched on the trade offs between using open source platforms versus existing walled garden incumbents. Culture how do you effectively implement AI from a cultural process people point of view?

One of the both beautiful but also double edged things that is happening with the ease of libraries in the olden days 10 years ago if you wanted to. If you want to put together a model, a neural net model, you would probably have to layer it, pull it, put it up layer by layer. You'd have to design it yourself for the most part. Now you get a pre trained something that is to say a model that somebody else has trained and then you're just going to tweak its decisions a little bit to adapt it to your particular data or the days of coding up your own specific classification model or there's a whole host of supervised and unsupervised et cetera models that you might want to do. The days of having to have ML experts do that by hand or over, that usually doesn't happen. At most you're doing tweaks, you're doing minor corrections. And what that means also is that you can have non experts who are not used to a lot of the things that I've already touched on. Like how do you make sure that your model's not overfitting? How do you make sure that your data is clean? How do you trust and have the right metrics so that you don't have an embarrassing situation where your LLM offered free X of service and now you're on the hook for actually backing up whatever your LLM did, even though it wasn't trained or aligned, it's called for that particular mistake. So enabling your engineers in order to leverage all of these great use cases, including the models that seem to take care of themselves sometimes deceptively and making sure that you still have very rigorous metric supervision and evaluation of how your models are doing and when the worst cases are going to happen. Making sure that you stratify another technical term that means you consider all the different situations carved up into little boxes so that you make sure you're not just performing well on one, you're performing across the board of all of the different use cases. Having that strong culture to back up the performance of your model before you go into release is probably one of the biggest issues in terms of having a strong ML culture when you're as it gets easier and easier for non experts to start leveraging these tools.

And if in a five minute conversation with a business owner you have the opportunity to completely overwhelm them by throwing all sorts of technical ML terms if that is not what they have done day in and day out, but they're very good at running the P and L and all of a sudden we are in a new paradigm, how can they stress test what you are doing? They don't speak your language. You throw the three letter acronyms like where do they find themselves?

It's a wonderful question. I, I mean I love it. Yeah, so. So it's very easy to throw technical acronyms and to maybe go deeper than is really necessary. You when you're coming from a technical perspective understanding exactly what a transformer model does or what a multi head attention network does and why you need rotary versus non rotary embed positional embeddings or different softmax loss functions and temperature, that honestly doesn't make a big difference in terms of making the business decision of is this model going to perform the way I want? Is it going to perform simple? That is say when we're live, are we cleaning our data? Are we using are do we have metrics that match what the model was trained to do? Our loss function, our leading metric, how did my prediction match the truth versus what the product is supposed to do? And asking questions about why those match is going to be the most valuable thing for you to dig in on because that's what's ultimately going to determine the performance of how your product that's AI driven is going to succeed.

I couldn't agree more. So let me turn it over to you to delve into some of these examples.

Yeah, Shopify is one of one of the largest e commerce platforms in North America in particular. That means making sure that we can answer a fundamental question of if a buyer has a query and is searching for a query, which products would match that query? Right. Which ones are relevant matches and which ones are irrelevant matches. Now that becomes relevant when there's a particular merchant who has a single shop who is using Shopify and wants you want to and if somebody goes to that shop and wants to search for particular project products within that shop. We also have a Shop app where you're searching much other e commerce experience across multiple merchants and multiple products that are hosted across several different providers. Right. And so search relevance in those search contexts, those kind of search contexts that as an end user you're very familiar with are first and foremost very important. Now there's also just making sure that you have high quality. Right. If I search for boots and I end up with boot socks and boots and ski, ski boots and a lot of things that might not be relevant, I need to be able to evaluate that and I need to be able to evaluate that, say on the first and different results to make sure that when we do these key searches, we're giving the end user buyers the right kinds of results. And to do that we need to have an evaluation method. Right. How does this product match with this query? And again this goes back to what I've been banging, banging the drum on all along is you need to care about your metrics. Right. So this is as much a metric as a product itself.

Mike, if I may interject because I think we can't stress the metric point enough. It's also not all metrics are created equal across different businesses. So what's a good enough metric? I'm going to make up a basic example. If I search for an item on Shopify, the boot example, and I get the right item 50% of the time, that might be great. Great or not, you will tell us what's relevant in one context but completely insufficient in another context. So tell us a little bit about not all results are created equal. Back to this metrics point. It context matters.

Sure, yeah. And maybe to your point in product search, there, there's two dimensions. There are two, two all principal directions that you might want to have relevance in. Right. One might be I'm looking for a product and so I want to see that product. I want, I'm looking for boots. I want boots. Another might be I'm shopping for boots and so I'm going to buy boots and I'm going to buy socks that match the boots and I'm going to buy maybe some cleaner that, that make sure that my boots stay in nice condition in the weather in the winter time. So cross selling and relevance. Right. And these can actually be two different metrics. And maybe what you would want to do is have one metric that shows for direct relevance and another that shows complementary relevance. Right. And then it is somewhat a product decision of how much you want to balance these, but it's also something that you can balance with GM with, with your, your bottom line. Right. With how much do you know if we balance for say 50% are going to be direct on the nose relevant, 50% are going to be more complementary. Maybe if we tweak that to 60, 40 or 70, 30, you might actually see a change in your bottom line. And that's the sort of, that's the way your product, your end goal is going to be determined by certain parameters that you choose in your actual model deployment.

And so back to this interdependency between your teams and your organization and the P and L owner or the merchandiser, whoever is responsible for the business unit. And that bottom line, how does that interaction back and forth work? Whatever tweaking you are doing on 40 or 50% or 2% improvement has ramifications in terms of their bottom line, but also the resources that you have leveraged. So if we wanted to, we expressed the metric in terms of results for the end customer in this example. What about from an ROI perspective? How much do you have to put in to get what out?

Yeah, so there's a couple aspects to that. One is, as a data person, obviously I love seeing the numbers and letting the numbers tell me. And so having experimentation, being able to experiment with. To take the simple example of how do you balance these two kinds of experiences is one answer. And then you need to invest in experimentation frameworks, right? And hopefully, you know, again, these things get easier and easier as there's existing open source frameworks and tools and vendors that you can leverage for that. That's one answer. The other answer is really keeping close communication with your stakeholders. And this really speaks to the culture aspect of it, right? Just having ML, ML engineers and ML scientists working in a vacuum and coming up and saying this is your model, I'm going to go work on the next fun technical thing is a mistake. And it's been a mistake. And we've seen different versions of this. Back when my job title was called data scientist and that was all, that was the only kind of title there was, you know, what you need is to have a pretty mixed, you know, you need to have engineering for serving, you need to have your ML scientists and engineers and you need to have product. You need to have product every in the trenches with your ML builders. So that way they can understand what are the pros and cons of not just making a tweak in a parameter and what a metric is, but also that key that, that key value that I keep driving home, which is does your end goal product metric and what you want to get there match your, your metrics that your model is trying to learn from and that's going to make a huge difference in terms of what the, what you actually see when you get your bottom line results.

Got it. Vectorization and Multimodal LLMs. LLMs. Most people have heard in one form or another. Multimodal is initially we started with text to text, now it's text to image, text to video, 2D to 3D, et cetera, itemization in plain English. And then how that come bears relevance in the context of the exchange we've just had.

Yeah, we had. In the good old days, it was keyword search, right? And people have learned, oh, I just put in one. In the good old days, maybe 15 years ago, when you're searching, you search for one word. You don't really phrase it in a sentence. You don't talk to Google that way. At least you used not to do that because it's just looking for the main words. It's ignoring that we call the unimportant words and trying to match those in the products or whatever or whatever it is that you want to do. You want to search for. Right, the documents. If you're doing, if you're doing a search on webpages, for me is, it's a somewhat complex topic which really can be boiled down to, to a very simple image which is in a multidimensional space. And you can imagine three, every word in its context and every image in its context can be represented as a point in that space. And if you're doing your vectorization correctly, then similar words from similar context or similar images are going to end up located in that similar part of the space, right? So all your cat pictures are going to end up here. All your dog pictures are going to end up close to the cat pictures, but not the same place. All of your fireworks predictions, we've seen a lot of those are going to end up way far away from those, and the words corresponding to those are going to be in similar. If you do it right, your image vectorization, where your pictures end up as a point in that space are going to end up in the same part of that space as your words. So if I describe cat, it's going to end up where the cat pictures are. If I, if I describe a dog, it's going to end up where the dog pictures are. And when I say in context, what do I, what if I say bank? Once I say bank, I could say the robbers were leaving the bank and they were going at high speed and crashed into the river bank. And I've said bank twice here and those are very different words, right? One is going to be in the con. The second one in that context is a place by a river. The first one is a place that has money. And so modern techniques for vectorization have Done a very good job of looking at the rest of the sentence, looking at that context and figuring out that the first instance of the word goes where with pictures of industry, industrial places that manage money, and the other one has pictures of rivers and that sort of thing in it.

What Mike just described is about 80% of the backbone of Gen AI and why we have the ability to interact with some of the applications in plain English and get human like interactions. Because leveraging these vector technologies and delivering what's called embeddings, which is really leveraging the vectorization through deep learning just so that folks actually internalize what this all means.

Yeah, and when you hear embeddings, it's. It's pretty much a synonym to how I'm using the word vectorization. It's embedding the word or the image into a vector space.

So Mike, you're using this and how you're constructing the. Whether it's the merchandising, the images, the text, the descriptors, pricing. Are you leveraging AI for dynamic pricing of any kind?

So dynamic pricing is a complicated area and very generally, I know from past lives you never want to do dynamic pricing, you want to do dynamic discounts as a, as a product framing. Bit of advice. Now certainly having tools helping merchants to understand things like demand curves and supply curves and how that those trade offs are, that's certainly something that, that Shopify is interested in providing for merchants. As the first party customers of Shopify, we want to give them the tools so they can be effective entrepreneurs real quick. Just to frame the evolution of what's going on here. Right. We started with keyword matching, then we might compare vectors. Right. And typically I had on the previous slide cross encoding and two towers. These are just very technical neural network architectures where we actually look at the vectors at the same time versus looking at the vectors separately. And if you want, if I do a query, I don't want to have to compare that query to every single product across all of Shopify, let alone even maybe in a single large merchant. And because it's not scalable, if you have hundreds of millions to billions of products to search for, you can't do that.

No, it's scalable. We mean it's very expensive to you from a compute perspective.

Yep, expensive. Yeah, expensive and, and obviously prohibitively, it would take too much time in order to return it that way. So that's one side of. And that is in terms of serving, in terms of giving results, when somebody produces a query, you're going to Want to sever those now if you want to just look at quality and you can do just smaller quality checks, then you can look at them simultaneously, you can get a little bit more information. The same story is now true with LLMs. LLMs, if you domain adapt them, right, that is to say you train them on your data and have them learn your specific behaviors that you want to see correctly can be very good at. Also taking in a slower context. I, I look through this query, I have this product. Are they relevant together? Or I'm I have this product. What are the relevant features, what are the attributes, what are the, what are the sorts of structured features? If you have a T shirt you might, it might have a V neck, it might have a turtleneck, it might have different colors. What sorts of thing can I extract about this image and this product description that I can use in search queries down the line? And so LLMs are remarkably, almost too good to be true effective at this sort of work. Good. What's the cons? They're very slow and they're very expensive and if you pay a commodity LLM, which we're right now does usually beat out open source LLMs, you're going to end up paying per token and you can't scalably if you have the right size of a business do this for every query or every kind of use case. So what do you do? You might want to take an open source model and do what we call distillation. Distillation is a technical term. It means I have a small model, I have the answers that I've gotten my big model, I know what goes into the big model and what comes out of the big model. And so I train my little model to learn to mimic the big model answers. Right.

And how's that different than fine tuning?

So fine tuning is you might have any size model, the parameters are already trained on a lot of data by someone else for their data and that data might be close to yours, but it's not exactly the same data as yours. So if I'm going to fine tune my model, I'm going to take my specific data and I'm going to adapt to my specific domain on my data, the base model. So for an LLM context, LLMs are really remarkably great at grammar and understanding the subtle and these days understanding the subtleties of language and often during multilingual things. And there's a lot of baseline things that I don't want to put in the work on that. Right. That work has been done and it's been done very effectively. And there's no re to have no reason to have everyone be another OpenAI or another anthropic when it comes to baseline. But when it comes to my specific data, whatever it is, I'm going to want it to be extra, extra good at those nuances. And there are all sorts of ways of fine tuning. There's all sorts of ways of. There's a whole lot of technical literature on different ways that you can adapt your model to your specific data. That's fine tuning is what the main one.

And so back to your example, you were going to do the LLM, the large language model. And then almost like a small language model that's separate from fine tuning, you're fine tuning on top of the LLM. This is above and beyond sits on top of it. I just want to make sure people get an image of it. Yeah, okay.

Yeah. So you fine tune your smaller open source, and when I say smaller, it might be 7 to 70 billion. Not so small, just small compared to the bigger one. Right. You might fine tune it in order to mimic the performance of the large commodity model.

Got it. I know we have some more context, but what do you wish I had asked you about your work that we didn't get to.

Maybe it'd be worthwhile just spending 60 seconds to talk about the difference, about the data issues, because we, we, we Easter egg that. Yeah. So in particular, let's think about search results. How do you get data? You want to train your model to produce good examples for a query and you know how to avoid bad examples for a query. How do you train your model to do that? There's lots of ways we can do it. We can have humans do it and come up with thousands, thousands of examples. And that could be expensive. We can ask an LLM to do that. And that is a new dimension in terms of data management that's now available. But now we're trusting our LLM for the answers that we're going to build our entire foundations on. And so you need to have other ways of validating that information, making sure that it's not wrong, that you're not just trusting the LLM to avoid what's.

Come to be known hallucination. As hallucinations engage.

Absolutely. And so with your. So you could also use your own data. Right. You can have positive examples. Oh, those are the things. I searched for something, I bought it, I clicked on it, I bought it. That's definitely a success. It gets really subtle when you have negative examples. Right. Because If I search for boots and I show a Christmas tree that's not going to. The model's not going to learn much from saying, oh, that was wrong. Right. It's going to learn a lot more by showing its sneakers or by showing its sandals and saying that's because it's closer to what would ostensibly be correct. So thinking about that data and working with product, working with how do we carve up the different space of possible answers and understanding how you can think about what's negative example versus what's a positive example in combination with maybe asking your LLM to find things that are close but not so not but no cigar, so to speak. That's really where a lot of this we talked about data work is the bottom of the iceberg. I think that takes so much time. That's where a lot of the hard work comes in.

And any prospects for automating the data prep and what data scientists and data engineers do because it's so laborious and boring for them and they're so expensive and can create so much value.

Yeah. And I, it's, it's a, a little bit of a, of a skilla and probdis. Right. There's a balance here between trusting your, trusting your LLM implicitly and never supervising it versus versus using, you know, not using an LLM and then being stuck paying human to do things that maybe an LLM can do for you. And so finding out different layers of may. I'm going to generate some candidates with my LLM and then I'm going to have humans evaluate the quality of their results. Or maybe I'm going to have LLMs find the potential negative values and. But have humans actually curate those values and make sure that they're actually good negative values. Balances of those and ask and then paying you the, the people that you spend the big bucks hiring in order to do the statistics and make sure that this, this layering of strategies is correct. That's a great place for them to spend time rather than coding up neural networks from scratch, which is a thing of the past.

So with that, I think we started our discussion where there is a lot of trial and error and I think we're ending with trust and verify. But thank you so much. Mike or Dina, thank you so much for the very informative session and for all the details.

Thank you.