SaaStr AI Day: How Shopify is Implementing a Practical AI System Across their Sales and Product T…

AI Day 2024. We are so thrilled to be here with one of our closing sessions and to help have you here for another exciting session. So please join me in welcoming two amazing speakers. We have Radhina Sassuri. She is a managing partner at Glasswing Ventures and Mike Tamir, head of AI at Shopify. They are going to be doing an awesome session that's going to be talking about how Shopify is implementing an AI system across their sales and product teams. So we are so thrilled to have both Radhina and Mike dive in a little bit more to what's going on in the ecosystem and hear some real-world examples of how Mike is doing this work at Shopify. We are so glad to have you both here. And before we get going, I just wanna remind everyone that this session is being recorded. It will be immediately available on our SaaStr YouTube page, so please make sure you're subscribed there. And the slides will also be available on our SlideShare. So I'll drop the links down here. We're also going to have plenty of time for Q&A, and Radhina and Mike would love to hear from you. So encourage you to have your cameras on, to drop your questions in the chat, and we will work on getting you into the queue when we get to Q&A. So without further ado, Radhina and Mike, welcome so much to AI Day.

Thank you so much, Taylor. I'm happy to be here.

Good to be here.

Great. Well, I will let you both take it away.

Thank you. So what Mike and I— hello, everyone. What Mike and I thought about doing today is talk— sharing a bit of our backgrounds very briefly, but really diving into how does one think about AI adoption, both at an individual level in a business context, as well as the critical work that Mike is doing at Shopify and has previously done at Uber, Saskahanna, amongst others, as well as in the cutting-edge work he does teaching at UC Berkeley. So by way of background, I'm managing partner at Glasswing Ventures. We are an AI originalist firm. We in fact founded Glasswing back in 2016 with the goal of investing in AI-native companies that solve real problems for the enterprise and cybersecurity markets. One of our approaches as a first capital-in investor is to actually build a firm of what we call building partners, wherein each of our partners have been previously operators. In fact, if you look at Glasswing and how we're composed, we mirror the leadership team of an AI-native technology company. And equally importantly, who we associate and affiliate ourselves with, that being our advisory councils, is one of those groups where we're very lucky to have Mike Tamir as one of our advisors. Mike, with that, would you like to do a little bit more on your background? I think I would be doing it a disservice.

Oh, I, I think you did, you did more, more than enough. But, uh, just to recap, I, uh, I'm, I'm at, uh, Shopify, uh, one of the distinguished ML scientists and running, uh, ML and AI functions here.

Lovely. So let's, um, delve in the first part of this conversation. I will do a bit of level setting around what's going on with AI from an enterprise adoption and where we are in that cycle. And then Mike will bring it to life with his actual work and how Shopify as a best-in-class adopter is leveraging it. An important level-setting point is the fact that today there are many initiatives across large and mid-sized enterprises around AI. In fact, we tell our AI-native companies not to lead with AI when they're speaking to prospective customers because it's such a top-of-mind topic that it's a superficial indication of interest. But most of the engagement, with few notable exceptions, is still at that superficial level with early sort of measurements, if any, on the impact on the enterprise on an adoption basis. Now, we have seen a few green shoots, if you will, in the space, particularly, for example, what we saw with Klaviyo and the impact that they have had by bringing AI from a project siloed or customer interface type level usage to their core business where they automated 700 humans and over 2/3 of customer success is now AI supported and generated and contributed $40 million to their bottom line. And it is because they brought it to their core product, to their core business. And as we expect in the mid and long term, the impact is sizable. But they are noteworthy because they are one of the few companies— and again, Mike will talk to what Shopify is doing as another leader in that category— that have brought AI to the core business. So what is the mindset? What is the actual work that needs to be done around building the infrastructure to leverage AI for the core business and the highest impact? Well, I would articulate it along three dimensions. One is software and the technology infrastructure and how, whether it's the debt, whether it's the tech decisions, whether it's the data availability, et cetera, but getting the enterprise or the facets of the business to be AI ready. The other piece is really around cleaning the data. So one is the algorithmic side of technology side. One is the data that you are going to feed. And oftentimes, you know, there is a, almost naive assumption that the data will be easy to leverage. Oh, it's an asset, and we'll use it and leverage it. And it turns out that oftentimes is the biggest problem. Mike, please back me on this. And I know you'll talk about what you are doing. And thirdly, culture, culture of adopting AI. And again, we'll delve more into this as we proceed in the discussion. So where are we heading, therefore, as we try to adopt AI in the enterprise? Well, I have a term that I use called ambient AI, which is basically that we, let's think of ourselves as professionals and productivity sort of driven professionals. We will be surrounded in and leveraging AI in this copilot dynamic. And for one to wrap one's mind around what does it mean for me, you know, to be leveraging ambient AI, If the input is you, the business individual leader contributor, your activities and your environment, and you think of that as the input into ambient, into your existing AI tools that you've surrounded yourself for a copiloting dynamic, the outputs are insights that you wouldn't have been able to derive on your own. And secondly, the automation of the mundane. The low productivity, highly repeatable to free one up to be more valuable. So as we think about our roles as professionals and what will make us successful going forward, worth considering that in addition to the well-tried high impact, high productivity, the people and the professionals will have one leg up if in fact they are leveraging the tools, if in fact they are intelligent business workers, not simply in the intellectual sense, but leveraging the intelligent tools. And as you— we shift to the enterprise, it's really these topics of data, infrastructure, and culture. And I will use this layout and framework to turn it over to Mike to talk about the impact starting with the data. Mike, first question stated here, in fact, how do you know if you have the right data? And is data really sort of the claim that I made is that oftentimes the biggest bottleneck and the biggest problem. For this business audience, not necessarily a technical audience, how do we think about that change first?

Absolutely. So the, you know, couldn't be, couldn't be more correct that data is often the messiest and, and the time that you think you're, you know, while we have the data, why don't we just get to the building the models? When in reality, it's one of these iceberg clichés where most of the time you spend is on gathering the data and cleaning the data and making sure that it's ready, that it's of the right quality. And even once you begin the modeling, you're not done. You know, modeling is really an iterative process where you train your models, you see what signal you're getting from it, you see how good your performance is. And this speaks to measurement and making sure that you have the right measures.

I'm going to interject. What does— I mean, we all know what training the model means, but what does training the model mean? Is it like, oh, here feeds the data, goes into the model? I actually want you to give us, without getting too technical, the nitty-gritty of how hard that is.

Sure. And of course, every different ML technique is going to be slightly different, but very generally, and certainly when you hear about LLMs and a lot of the modeling techniques that are in vogue, it's trial and error, right? A model will see an example, an example of this is what information we have and this is what we're trying to predict. It'll take the information, that you give it of what we have ahead of time and make its own prediction and then look at how far it missed, right? How far is what it tried to predict from the truth of what it should have predicted? And then it'll take that error, right? We hear about failure is the key to learning. You know, this couldn't be more true than in machine learning. Machine learning literally takes that error and with a bit of math tries to adjust all the little parameters, all the little calculations that it makes along the way from input to output in order to make that error just a little bit smaller the next time. And so this leads into a lot of the things that you have to do with your data as best practices. So if I take all of my training data and I teach my model with my training data to close that gap between its predictions and what it should have done, then I might be overfitting. Right? And so you need to be very meticulous, not just in keeping good training data.

Overfitting.

Overfitting. A part of my job is going to be anytime you throw a technical term, by the way, for the broader audience, that delta, if anyone cares, the technical term is zero loss function, but then we use the term overfitting. So what does that, that overcorrection, what does that truly mean in business?

Yeah. So overfitting is if you're, if your model has taken that time to close the gap between its predictions and what it should have done. Maybe what it's done is just learned for those specific examples that you gave it to learn from, right? And so it has overfit to those specific examples. But we don't want to build models that can just take the inputs you give it and get the outputs that you told it to learn. We already have that data. What we wanted to do is to learn the pattern, and to learn the pattern means we have to do all of these different things in order to you know, to constrain the model in various ways so that it predicts not just on the data that you're giving it, but it also predicts effectively on the data that it's never seen before. And we usually call that validation data and ultimately test data in different stages for figuring out when is, how is this model going to predict accurately out of sample, out of the training sample.

And so if I am a P&L owner in XYZ business and I've had this data for 20 years and, you know, Mike's team is helping me sort this out. Why is it taking them, you know, multiple weeks? Like, what's so hard about it? What's so hard about it?

And there's two parts of this which also speaks to, I think, the culture, right? Especially in a world where you can talk to an LLM and it seems like it's giving you right answers off the bat and you want to say, well, you only live once, let's just use whatever the answers are. And I'm not going to name any, any LLM deployments that have happened in the last several months. But we all know news headlines where maybe that didn't work out so well. So having the culture of checking your results and making sure that you have the right metrics to evaluate, whether it's an LLM or a more traditional model that, that's not generating text or generating images, but just estimating numbers or estimating classes of, of whatever the task is. Having that culture where you are focusing on what are your metrics and you're evaluating those results is really important. On the why is it taking so long side of things, when you take the data, and we're going to have a very nice concrete example of this towards the end of this talk, it can be very, you know, the goal, the business goal that you have might be something like I want to make more sales. Or I want to delight my customer feed and customers and see that in feedback, whatever that metric is, we have to translate that. Speaking to the culture of measuring our results, we have to translate that to a concrete metric. And so this is going to be some sort of mathematical function of saying how bad did my prediction miss from what I said the right answer was. And hopefully it's very, very close and you get some fireworks. Now, figuring out that answer is actually very complicated. It's complicated in finance. It's complicated in producing recommendations to, or returning a search for a query. There, that gap between what I want to mathematically measure, which is how the machine is going to learn based on that error, and what my actual product result is, really matters. Another area where you really have to think about what matters is how the data is actually structured. If I give a bunch of examples where, um, of this is the kind of input and this is kind of output I would expect, or this is the kind of output I would not expect, that's going— the model is going to learn that specific pattern. So if you give it bad examples or you give it dirty examples, either examples with the wrong answers or examples where where a negative case doesn't— isn't as informative as an almost positive case, and we're going to get again to a concrete example of that soon, having that clean data is really going to be the lifeblood of how the model learns and ultimately how the product is successful. Got it.

All right, so continuing our discussion on the infrastructure side, without getting too technical, if you are a productivity business user, you can only affect the decisions that get made at the infrastructure level evolved so much. What should one know? What constitutes sort of a modern AI stack that enterprises can scale on the basis of?

Ultimately, unfortunately, that's going to be a little bit dependent on, on the, on the use case in the context, right? Every company is going to have slightly different needs and going to have slightly different applications. When it comes to, you know, doing the, the more advanced training and deep learning models that I've been describing, you're going to need GPUs, for one thing, right? Which means you may need a single GPU and you're a very small startup and you're trying to just work on a budget. You may need a cluster of GPUs, in which case you need infrastructure management. In the last several years, there's been obviously a very, a very big shortage of GPUs. And so the, you know, it has historically been the existence of cloud platforms— AWS, GCP, Azure— has been very helpful in sporadic needs. You know, oh, I need to rent a GPU for a few hours to do my training and then I don't need it anymore versus purchasing or long-term rental, right, where you reserve an instance for days or for months and that's gonna end up costing thousands of dollars in oftentimes. So being able to size up what kind of training you need and then picking what level of, you know, are you going to need it ad hoc, are you going to need it as a reserved instance, maybe you buy your own infrastructure, which is, also, you know, often cheaper answer if you have, if you already have an infrastructure, an on-prem infrastructure system. That's really going to depend on what your AI training expectations are anticipated.

So I'm going to throw a curveball. This is not a question that we discussed, but all is fair. So if we think about the migration that enterprises made from on-premise to the cloud, sort of the underlying value prop was, look, the variability is fairly cheap. You can ramp up fairly fast. In fact, I, I have this term that I use that it's a variable fixed cost on your P&L, meaning that you can scale it up and down. The line will always be there for you as a business, and that's the cloud cost. As we move to the, a world of, you know, AI nativeness and leveraging it, is your belief that the existing incumbents will actually, whatever you're seeing with the large foundation models providers, will become the Swiss Army knife. So I'm speaking, for example, to the AWS and Amazon approach where they're saying, listen, you want to use OpenAI, you want to use Anthropic, we have our offering, we have Bedrock, you don't have to switch, it's going to be one more feeder. So does this line remain or do we have a new fixed variable line in my Rodina lingo?

It's, it's a very good question. And certainly when you talk to AWS, you talk to GCP, they all have partnerships and obviously Azure very much so with, with OpenAI, they have these partnerships. But, and it's worth being aware of this, you know, using a first-party solution with, say, OpenAI or with Anthropic is going to have different performance, different latency than if you do it via an intermediary like a cloud. So you maybe get some simplicity by not having to onboard different vendors and different, different support. But there is going to be an impact in terms of performance. And that's something that, while might be a temporary problem, is certainly a very real consideration. It's March 27th right now.

And that was going to be my next question. Is that a high pain point in this moment in time, the Delta in performance, or do we believe it persists? And your view is?

It's right now, it's worth not ignoring for the use cases that I've had. And something else, you know, maybe hopefully transient for March 27th, 2024 is the option for ad hoc GPUs when you go higher than like an L4, if you need an A100 or an H100. Something that can host some of these larger, large open source large language models. We talked about commodity models. What if you want to do something like a $70 billion Llama or Minstrel or Mixtral or any of these? Those are going to be very hard to get on an ad hoc basis and you're going to end up having to pay the price for a reserved instance if you want to be able to serve something like that.

And those are, you've now touched on the trade-offs between using open source platforms versus, you know, existing walled garden incumbents. Culture. How do you effectively implement AI from a cultural process people point of view?

Yeah. One of the both beautiful but also kind of double-edged things that is happening with the ease of libraries, you know, in the olden days, 10 years ago, if you wanted to put together a model, a neuron net model, you would probably have to layer it pull it, put it up layer by layer. You'd have to design it yourself. The— for the most part now you get a pretrained, something that is to say a model that somebody else has trained, and then you're just going to tweak its decisions a little bit to adapt it to your particular data. Or, you know, the days of coding up your own specific classification model, or, you know, or there's a whole host of supervised and unsupervised, etc., models that you might want to do. The days of having to have ML experts do that by hand are over. That usually doesn't happen. At most, you're doing tweaks, you're doing minor corrections. And what that means also is that you can have non-experts who are not used to a lot of the things that I've already touched on, like how do you make sure that your model is not overfitting? How do you make sure that your data is clean? How do you trust and have the right metrics so that you don't have an embarrassing situation where your LLM offered free X of service and now you're on the hook for actually backing up whatever your LLM did, even though it wasn't trained or aligned, it's called, for that particular mistake. So enabling your engineers in order to leverage all of these great use cases, including the models that seem to kind of take care of themselves sometimes, deceptively so, and making sure that you still have very rigorous metric supervision and evaluation of how your models are doing and when the worst cases are going to happen. Making sure that you stratify, another technical term, that means you consider all the different situations carved up into little boxes so that you make sure you're not just performing well on one, you're performing across the board of all of the different use cases. Having that strong culture to back up the performance of your model before you go into release is probably one of the biggest issues in terms of having a strong ML culture. When you're— as it gets easier and easier for non-experts to start leveraging these tools.

And, and if in a 5-minute conversation with a business owner, you have the opportunity to completely overwhelm them by throwing all sorts of technical, um, ML terms, if that is not what they have done day in and day out, but they're very good at running the P&L, and all of a sudden we are in a new paradigm, How can they stress test what you are doing? They don't speak your language. You throw the 3-letter acronyms, like, where do they find themselves?

Ah, it's a wonderful question.

I'm blindsiding you right and left.

Go ahead. Yeah, I love it. Yeah, so it's very easy to throw technical acronyms and to maybe go deeper than is really necessary when coming from a technical perspective, understanding exactly what a transformer model does or what a multi-head attention network does and why you need rotary versus non-rotary embed— positional embeddings or, you know, different softmax loss functions and temperature, you know, that honestly doesn't make a big difference in terms of making the business decision of is this model going to perform the way I want? Is it going to perform out of sample, that is to say when we're live, are we cleaning our data right? Are we using, or do we have metrics that match what the model was trained to do? Our loss function, our leading metric, how did my prediction match the truth versus what the product is supposed to do? And asking questions about why those match is going to be the most valuable thing for you to dig in on because that's what's ultimately going to determine the performance of how your product that's AI-driven is going to, is going to succeed.

I couldn't agree more. So keeping an eye for the time, I will skip— actually, this is a proprietary framework that we're going to open source, but I will skip this because I do want to get into the actual initiatives at Spotify and bring this all to life. So let me turn it over to you to delve into some of these examples.

Yeah, so, you know, Shopify is one of, you know, one of the largest e-commerce platforms in North America. And in particular, that means making sure that we can answer a fundamental question of if a buyer has a query and is searching for a query, which products would match that query, right? Which ones are relevant matches and which ones are irrelevant matches? Now that comes up to, or that becomes relevant when, you know, there's a particular merchant who has a single shop who is using Shopify and wants to, you want to, and if somebody goes to that shop and wants to search for particular products within that shop. We also have a shop app where you're searching, you know, much like other e-commerce experience across multiple merchants and multiple products that are hosted across several different providers, right? And so So search relevance in those search contexts, those kind of search contexts that as an end user you're very familiar with, are first and foremost very important. Now there's also just making sure that you have high quality, right? If I search for boots and I end up with boot socks and boots and skis, ski boots, and a lot of things that might not be relevant, I need to be able to evaluate that. And I need to be able to evaluate that, say, on the first and different results to make sure that when we do these key searches, we're giving the end user buyers the right kinds of results. And to do that, we need to have an evaluation method, right? How does this product match with this query? And again, this goes back to what I've been banging, banging the drum on all along is you need to care about your metrics, right? So this is as much a metric as a product itself. Question.

Now, so, Mike, if I may interject, because I think we can't stress the metric point enough, so I'm going to double-click on it. It's also not all metrics are created equal across different businesses. So what's a good enough metric? I'm going to make up a basic example. If I search for an item on Shopify, the boot example, and I get the right item 50% of the time, that might be great or not. You will tell us what's relevant in one context, but completely insufficient in another context. So tell us a little bit about like not all results are created equal. Back to this metrics point, context matters.

Sure. Yeah. And maybe to your point, in product search, there's kind of like two dimensions that are two principal directions that you might want to have relevance in, right? One might be, I'm looking for a product, and so I want to see that product. I want— I'm looking for boots, I want boots. Another might be, I'm shopping for boots, and so I'm going to buy boots, and I'm going to buy socks that match the boots, and I'm going to buy, you know, maybe some cleaner that makes sure that my boots stay in nice condition in the wintertime. So cross-selling and relevance, right? And these can actually be two different metrics. And maybe what you would want to do is have one metric that shows for, you know, direct relevance and another that shows complementary relevance, right? And then it is somewhat a product decision of how much you want to balance these. But it's also something that you can balance with GM, with your bottom line, right? With how much do, you know, if we balance for, you know, say 50% are gonna be direct on the nose relevant, 50% are going to be more complementary. Maybe if we tweak that to 60/40 or 70/30, you might actually see a change in your bottom line. And that's the sort of, that's the way your product, your end goal is going to be determined by certain parameters that you choose in your actual model deployment.

And so back to this interdependency between your teams and your organizations, and the P&L owner, or, you know, the merchandiser, whoever is responsible for the business unit and that bottom line. How does that interaction back and forth work? Whatever tweaking you are doing on 40% or 50% or 2% improvement has ramifications in terms of their bottom line, but also the resources that you have leveraged. So if we wanted to, we express the metric in terms of results for the end customer in this example. What about from an ROI perspective? How much do you have to put in to get what out? What does the return look like, both maybe in whether you can quantify it in a multiple, but also timeframe? Is it within budget? Is it outside of budget? How do we think about that?

Yeah, so there's a couple aspects to that. One is, as a data person, obviously I love seeing the numbers and letting the numbers tell me. And so having experimentation, being able to experiment with, to take the simple example of how do you balance these two kinds of experiences is one answer. And then you need to invest in experimentation frameworks, right? And, you know, hopefully, you know, again, these things get easier and easier as there's existing open source frameworks and tools and vendors that you can leverage for that. That's one answer, right? The other answer is, really keeping close communication with your stakeholders. And this really speaks to the culture aspect of it, right? Just having ML engineers and ML scientists working in a vacuum and coming up and saying, this is your model, I'm going to go work on the next fun technical thing, is a mistake. And it's been a mistake. And we've seen different versions of this back when my job title was called data scientist. And that was all— that was the only kind of title until there was. You know, what you need is to have a pretty mix. You need to have engineering for serving, you need to have your ML scientists and engineers, and you need to have product. You need to have product every— in the trenches with your ML builders so that way they can understand what are the pros and cons of not just making a tweak in a parameter and what a metric is, but also that key that key value that I keep driving home, which is, does your—

Fireworks.

Does your end goal product metric and what you want to get there— I'm going to put my hands down— match your metrics that your model is trying to learn from? And that's going to make a huge difference in terms of what you actually see when you get your bottom line results.

Got it. Vectorization and multimodal LLMs. You know, LLMs, most people have heard in one form or another. Multimodal is initially we started with text-to-text. Now it's text-to-image, text-to-video, 2D to 3D, etc. Vectorization in plain English and then how that come bears relevance in the context of the exchange we've just had.

Yeah. So, so we had in the good old days, it was keyword search, right? And people have learned is, oh, I just put in one, you know, in the good old days, maybe 15 years ago, you know, when you're searching, you search for one word, you don't really phrase it in a sentence. You don't talk to Google that way. At least you used to not to do that because it's just looking for the main words. It's ignoring the, we call the unimportant words and trying to match those in the products or whatever it is that you want to do, you want to search for, right? The documents, if you're if you're doing a search on web pages. For vectorization, vectorization is, you know, it's a somewhat complex topic which really can be boiled down to a very simple image, which is in a multidimensional space. And you can imagine 3, you know, every word in its context and every image in its context can be represented as a point in that space. And if you're doing your vectorization correctly, then similar words from similar contexts or similar images are going to end up located in that similar part of the space, right? So all your cat pictures are going to end up here. All your dog pictures are going to end up close to the cat pictures, but not the same place. All of your, I don't know, fireworks predictions, we've seen a lot of those, are going to end up way far away from those, from those. And the words corresponding to those are going to be in similar— if you do it right, your image vectorization, where your pictures end up as a point in that space, are going to end up in the same part of that space as your words. So if I describe cat, it's going to end up where the cat pictures are. If I, if I describe a dog, it's going to end up where the dog pictures are. And when I say in context, what do I mean? Well, what if I say bank? If I say bank, I could say, you know, the robbers were leaving the bank and they were going at high speed and crashed into the riverbank. And I've said bank twice here, and those are very different words, right? One is going to be in the— the second one in that context is a place by a river. The first one is a place that has money. And so modern techniques for vectorization have done a very good job of looking at the rest of the sentence, looking at that context and figuring out that the first instance of the word goes where with pictures of, you know, of industry, industrial places that manage money. And the other one has pictures of rivers and that sort of thing in it.

Sorry, just interject for the benefit of the audience. What, what Mike just described is about 80% of the backbone of GenAI and why we have the ability to interact with some of the applications in plain English and get human-like interactions because leveraging these vector technologies and delivering what's called embeddings, which is really leveraging the vectorization through deep learning. Now I threw terms, but just so that folks actually internalize what this all means.

Yeah. And when you hear embeddings, it's pretty much a synonym to how I'm using the word vectorization. It's embedding the word or the image into a vector space. It's so— yeah.

So, Mike, you're using this, you know, in how you're constructing the— whether it's the merchandising, the images, the text, the descriptors. Pricing— are you leveraging AI for dynamic pricing of any kind, and is it something you'd like to weigh in on?

So dynamic pricing is a complicated area, and, you know, very generally, I know from past lives, you never want to do dynamic pricing. You want to do dynamic discounts.

How does that work for Uber exactly?

As a product framing bit of advice. Now, you know, certainly having tools helping merchants to understand things like demand curves and, you know, supply curves and how those trade-offs are, that's certainly something that Shopify is interested in providing for merchants as the first-party customers of Shopify. We want to give them the tools so they can be effective entrepreneurs. Doing dynamic pricing per se is not something that we work on at Shopify, or dynamic discounting for that one.

So multimodal LLMs, and then we'll bring it full circle and turn it over to Taylor after some learnings.

So, so real quick, just to, just to frame the evolution of what's gone on here, right? We started with keyword matching, then we might compare vectors, right? And typically, you know, I had on the previous slide cross-encoding and Two Towers. These are just very technical neural network architectures where we actually look at the vectors at the same time versus looking at the vectors separately, right? And if you want, if I do a query, I don't want to have to compare that query to every single product across all of Shopify, let alone even maybe in a single large merchant. And because it's not scalable, if you have hundreds of millions to billions of products to search for, you can't do that. So you need—

but not scalable, we mean it's very expensive to, from a compute perspective.

Yeah, expensive. Yeah, expensive. And honestly, prohibitively, it would take too much time in order to return it back.

Time and dollars.

So that's one side of— and that is in terms of serving, in terms of giving results when somebody, somebody produces a query, you're going to want to sever those. Now, if you want to just look at quality and you can do just smaller quality checks, then you can look at them simultaneously. You can get a little bit more information. The same story is now true with LLMs. LLMs, if you domain adapt them right, that is, say you train them on your data and have them learn your specific behaviors that you want to see correctly, can be very good at also taking in, in a slower context, I look for this query, I have this product, are they relevant together, right? Or I'm, I have this product, what are the relevant features? What are the, what are the attributes? What are the, you know, what are the sorts of structured features like If you have a t-shirt, you might, it might have a V-neck, it might have a turtleneck, it might have different colors. What sorts of thing can I extract about this image and this product description that I can use in search queries down the line? And so LLMs are remarkably, almost too good to be true, effective at this sort of work. Good. What's the cons? They're very slow and they're very expensive. And if you pay a commodity LLM, which right now does usually beat out open source LLMs. You're going to end up paying per token, and you can't scalably, if you have the right size of a business, do this for every query or every, every kind of use case. So what do you do? Well, you might want to take an open source model and do what we call distillation. Distillation is a technical term. It means I have a small model, I have the answers that I've gotten my big model. I know what goes into the big model and what comes out of the big model. And so I train my little model to learn to mimic the big model answers, right?

And how's that different than fine-tuning?

So fine-tuning is you might have any size model. The parameters are already trained on a lot of data by someone else for their data. And that data might be close to yours, but it's not exactly the same data as yours. So if I'm going to fine-tune my model, I'm going to take my specific data and I'm going to adapt to my specific domain on my data, the base model. So for an LLM context, LLMs are really, you know, remarkably great at grammar and understanding the subtle— in these days, understanding the subtleties of language and often doing multilingual things. And there's a lot of baseline things that I don't want to put in the work on that, right? That work has been done and it's been done very effectively. And there's no reason to have everyone be another OpenAI or another Anthropic when it comes to that baseline. But when it comes to my specific data, whatever it is, I'm going to want it to be extra good at those nuances. And there are all sorts of ways of fine-tuning. There's all sorts of ways of, you know, there's a whole lot of technical literature on different ways that you can adapt your model to your specific data, that's fine-tuning is what the main line is.

And so back to your example, you were, I'm thinking of it in the terms of a tech stack. You, you were going to do the LLM, the large language model, and then almost like a small language model that, that's separate from fine-tuning. You're fine-tuning on top of the LLM. This is above and beyond, sort of sits on top of it. I just want to make sure people have an image of it.

Yeah. Okay. Yeah. So you, you fine-tune your your smaller open source. And when I say smaller, it might be 7 to 70 billion, not, not so small, just small compared to the bigger ones, right? You might fine-tune it in order to mimic the performance of the large commodity model.

Got it. I know we have some more context, but I, I, I want to make sure that we open it up to, to the audience for questions. What do you wish I had asked you about your work that we didn't get to?

Well, maybe it'd be worthwhile just spending 60 seconds to talk about the difference, about the data issues, because we, we Easter egged that. Yeah. So in particular, let's think about search results. How do you get data? You want to train your model to produce good examples for a query and know how to avoid bad examples for a query. How do you train your model to do that? Well, there's lots of ways we can do it. We can have humans do it and, you know, and come up with thousands and thousands of examples, and that could be expensive. We can ask an LLM to do that, and that is a new dimension in terms of data management that's now available. But now we're trusting our LLM for the answers that we're going to build our entire foundations on. And so you need to have other ways of validating that information, making sure that it's not wrong, that you're not just trusting the LLM.

To avoid what's come to be known as hallucinations in case— Hallucinations.

Yeah, absolutely.

—God deals with that term. Yep.

And so with your— so you could also use your own data, right? You can have positive examples. Oh, those are the things I searched for something. I bought it. I clicked on it. I bought it. That's definitely a success. It gets really subtle when you have negative examples, right? Because if I search for boots and I show you know, a Christmas tree, that's not going to— the model's not going to learn much from saying, oh, that was wrong, right? It's going to learn a lot more by showing it sneakers or by showing it sandals and saying that's wrong because it's closer to what would ostensibly be correct. So thinking about that data and working with product, working with, you know, how do we carve up the different space of possible answers and understanding how you can think about what's negative example versus what's a positive example in combination with maybe asking your LLM to find things that are close, but not so— not— but, but no cigar, so to speak. That's really where a lot of this we talked about data work is the, is the bottom of the iceberg, the thing that takes so much time. That's where a lot of the hard work comes in.

And any prospects for automating the data prep and what data scientists and data engineers do because it's so laborious and boring for them and they're so expensive. And can create so much value.

Yeah, and I, you know, it's a little bit of a, of a Scylla and Charybdis, right? There's a balance here between trusting your LLM implicitly and never supervising it versus, you know, using, you know, not using an LLM and then being stuck paying humans to do things that maybe an LLM can do for you. And so finding out different layers of, well, I'm going to generate some candidates with my LLM, then I'm going to have humans evaluate curate the quality of the results, or maybe I'm going to have LLMs find the potential negative values and but have humans actually curate those values and make sure that they're actually good negative values, balances of those, and ask, and then paying the people that you spend the big bucks hiring in order to do the statistics and make sure that this, this layering of strategies is correct. That's a great place for them to spend time rather than coding up neural networks from scratch, which is a thing of the past.

So with that, I think we started our discussion where there is a lot of trial and error, and I think we're ending with trust and verify. But thank you so much. Let me turn it over to Taylor as she's monitoring the questions, and I think she's going to pick a few of them.

Yeah, thank you guys so much for the great conversation. And Mike, thank you so much for all your amazing insights. I actually think you guys covered a lot of the questions that came through the chat that last went Someone specifically asked for your opinion, Mike, about data cleaning. Why can't you teach AI to do that rather than needing human labor? So I think that was a great answer there. We do have someone who's still on the Zoom with us. Deb, are you still there? You have a great question about ROI. We may close with that one if Deb's still with us.

Yeah, I'm here. I just wanted to know, I'm a bit disgusted Jason M. Lemkin, you know, you are implementing all those AI things inside Shopify. Want to know a little bit about the ROI. Where are you guys primarily implementing AI in terms of the processes inside Shopify? Are you looking at the cost side of things? Are you looking at the revenue side of things? Where the business pressure is for you guys on the engineering side?

Well, we are doing quite a lot of ML and I would say it's growing, right? So there's the classic e-commerce use cases that I just talked about, search and personalization. So recommendations, understanding buyers, understanding merchants, understanding products, and seeing how all those things fit together in those interactions is obviously very, very core to the business. There's a lot of kind of frontline LLM copilot type work that we're doing as well. If you're inclined to look, I think we have some promotional videos about our Sidekick product, which helps merchants, you know, have kind of a copilot experience with how they operate their Shopify store and how they set it up to be successful or change it dynamically. Or, you know, it's a natural language interface. For administrating your store. And there's a lot of also kind of pretty standard stuff that you need for e-commerce. So security, fraud, you know, making sure that eligible products and ineligible products, products that are not safe for work, that sort of thing, are being monitored and content moderated in the right way. We have a program that enables entrepreneurs who may not be able to get access to capital in the same way than you know, than through traditional methods, let's, let's say, can come to Shopify because Shopify has insight into how strong their business actually is in, in, you know, access to information that maybe traditional, you know, sources of capital wouldn't access. So we can open the doors for these merchants in ways that maybe the doors weren't open before. And that's another rich area of, you know, sort of like financial ML that we can, that we can, you know, apply ML strength to. I think I'll stop there, but certainly there's a lot of opportunity for ML, and that's why I'm here.

I might add a statistic that we have been trained, sort of tracking. On average, enterprises have at least 2 to 3 projects going around ML, some more in depth than others, and about 87% of the, of the people leading those projects are reporting on sort of hitting ROI as planned and doing so either in the time that they forecasted or earlier, which is an interesting notion.

Yeah, I think looking at all those use cases you mentioned, Mike, then the ROI, you know, when we implement the AI projects, the ROIs are also going to be varied and there will be a lot of KPIs. How do you guys measure the ROI inside the company? Just not a financial KPI anymore, right? I mean, if you're looking at cybersecurity threat, It could be, you know, time to response if you're looking at something else. So the ROIs are going to vary for all those AI use cases, I guess.

Absolutely, right? Every metric is gonna be different and every loss function is gonna be an important question of how we are actually tracking that loss function for the particular use case. Okay, all right, thanks.

Mike, Radhina, thank you so much for the very informative session and for all the details. We appreciate it so much. Thank you everyone for joining this amazing session and also AI Day. So We really appreciate you. The recording is live on our YouTube, and we will see you at the next SaaStr event. So thanks everyone so much. Mike and Regina, thank you so much for an amazing session.

Thank you.