#241 Getting Generative AI Into Production with Lin Qiao, CEO and Co-Founder of Fireworks AI

Hi, I'm Maggie, VP of Curriculum at DataCamp, and I know firsthand how overwhelming it can be to keep up with the rapid changes in data and AI. But staying ahead doesn't have to be hard. That's why our curriculum team has curated the best learning content to help you and your team stay ahead, covering everything from data and AI literacy for busy executives to cutting-edge generative AI for developers and everything in between. Whether you're upskilling yourself or your team, we make learning practical, engaging, and flexible. It just works. Power your data and AI transformation journey on datacamp.com.

I think that's just the nature of AI. It requires a lot of experimentation and there are multiple things to experiment on. Experimentation is the top of the funnel. You need to experiment on how you frame the problem, right? How you frame the problem you want to solve. Is the framing right? Because framing will directly determine which technology you're going to pull into the stack to solve the problem.

Welcome to Data Framed, this is Richie. Playtime is over for generative AI. There's increasing pressure to make sure that all those experiments and prototypes you've been building get put into production. That's easier said than done, so today we're going to have a look at how to do it. In particular, a lot of the challenges around making sure you have good enough performance to satisfy your customers while keeping the costs low enough to satisfy your CFO. There are some technical solutions, but you also need to change your processes too if you want to create a reliable pipeline for GenAI products and features. Naturally, we'll consider both angles in this show. Our guest is Lin Qiao, the CEO and co-founder at the generative AI inference platform Fireworks.ai. That is, Lin helps developers create and deploy generative AI models. Before founding Fireworks.ai in 2022, Lin had an extensive career in technology, most notably leading the PyTorch team at Meta, where she played a critical role in developing AI frameworks and platforms such as Caffe2 and PyTorch. So I'm excited to hear her advice. Hi, Lin. Thank you for joining me on the show.

Hi. Thanks for having me. I'm super excited to have a chat with you today.

Yeah, this is going to be fun. So just to begin with, can you talk me through what are the most common uses of generative AI that you're seeing today?

Oh, that would be a— it would take 1 hour by itself. So yeah, so it is absolutely— this technology is so empowering and so disruptive. We have absolutely seen a lot of interesting, innovative applications and products built on top of our service tier. Of course, Firehouse offers inference engine for GNI models. So I will kind of bucketize them in different ways. So one big bucket is assistance. So we have seen people building medical assistant to address the shortage of doctors or nurses. And help those medical workers become productive. We have been seeing people building, many people building educational systems, target different cohorts. For example, there are people building applications for students to learn, for researchers to do research, and for people to learn foreign languages. So it kind of cross-border, various different applications, education. There are also legal assistance where they find people to do case study and because kind of lawyers are expensive or help lawyer to become more productive. So I think that's a big chunk of application. And then we see a lot of chat, just kind of interesting chat-based application. And usually those chat-based application combine voice with it. So it's not just, hey, you have to type. You can voice in, voice out, and those are also very popular. Another big bucket is API generation or code generation. It's not for end consumers, it's for another program. So this is important because we are seeing the compounding effects of people building application on top of not just single model but multiple models or multiple modalities or from multiple modalities to additional APIs. So there are example, the totality of knowledge I can drill down more to that because single model has limited, very limited knowledge by itself. And the specific use case are like generating business workflows, generating SQLs, generating charts about the production usage, monitoring observability. And so on. So that's a huge bucket. Those are kind of across the board where we're super excited of what's happening.

I feel like if you continued, you could probably just listed like every industry that there is, but that's really interesting that you got stuff that is applying to end users. So you mentioned things like productivity improvements and training, then you've also got that sort of middleware layer. So it's for other people building technology as well. Since today we're going to talk about getting things into products and into production, I'd like to know just at a high level, what are the different steps for getting generative AI into a product?

First of all, I would say application built on top of GenAI, on GenAI now, or application pre-GenAI, there are a few differences. One is pre-GenAI application built on top of CPUs. It's heavily commoditized. The stack, software stack is very mature. And the tools are kind of abundant, so it's kind of easy and low cost. With GenAI, it shifted, right? GPU is new to most of people, and the GPU is very expensive. GPU is power hungry, power expensive. Power generates heat, heat requires cooling, cooling is expensive. So the whole cost structure of productionized application on top of GenAI significantly changed. Right now. That's one. And second, the— between like before and after, whether you're building on GNI or not, one thing that doesn't change is most of those applications is consumer-facing, is developer-facing. It needs to be very interactive. So latency is a critical part of product experience. Holding the latency bar doesn't change before and after. Another interesting thing that is different is before GenAI, the logic, the reasoning logic is coded in the application itself or is coded in the surrounding infrastructure, so it's deterministic and you can debug and the logic is there. With GenAI, the GenAI models have multiple layers and with multiple layers of prediction, and it's probabilistic. Probabilistic means if you ask the generative model to produce something, it will do it regardless whether it's true or not. That directly introduced a new phenomenon, hallucination, which application developers never need to deal with. Now they are scratching their head like, "Oh, this is bad, and I need to learn how to control or minimize hallucination." in my product, in my application. So those are the kind of one invariant and two variants that's happening in kind of pre-GNI and then current GNI era. And because of that, that introduced new production challenges for our app developers. For the invariant, low latency, right? As we know, the GNI models is the extremely big side of machine learning models. And because the model is so big, and the latency will be high, and not having a low-latency product experience, it will be break or make it scenario for the app developers. So that is extremely important, we need to address that. And the cost is also important factor. Even if you have a viable product, if the cost is very high, and the nature of consumer-facing, developer-facing, apps, they scale quickly if you hit a product market fit. When you scale quickly, you're losing money at a small scale, you're gonna bankrupt at a large scale. So controlling cost is extremely important to go into production. And also to produce great product quality and content, you have to control hallucination and learning the skill set how to do that is extremely important. So I think those are the kind of 3 critical parts. To bring GenAI technology into production.

Lots done back then, lots of interesting changes. So certainly you mentioned like the switch from using CPUs to GPUs. NVIDIA's done very well out of this switch, I think.

Absolutely. Definitely.

And then the idea of going from deterministic programming to probabilistic stuff. I feel there's very different reactions there. Like all the software engineers are panicking, and then you speak to data scientists and like, isn't that normal for code to be probabilistic? So yeah, very interesting changes in differences in opinion there. We'll maybe get back into talking about latency later on, but for now, I'd like to talk about fine-tuning since this seems like quite an important sort of stage in making sure that you have high-quality results. So can you just give me a quick introduction to like why you might want to fine-tune a large language model rather than just taking something off the shelf?

So here I want to talk a little bit about the process of model training, right? So this is before fine-tuning. To train a foundation model, or train a model in general, before you train, the researchers, they need to be opinionated on what is the objectives for the end result they want to achieve. So that whole process will determine what kind of problem you care the most, what kind of problem you are okay with, and what kind of problem you don't even care. From there, you go and curate your training data. And for the areas you care the most, you want the most high-quality data, you want the most diversified data, and it requires a lot of care. And for areas you just want to have some coverage and it's okay, then you don't have to spend too much time, but the volume needs to be there. For areas you don't care, then you don't really actively source data to cover those areas. And the end result of training will reflect that, right? So, That's interesting because then there is— when people build foundation models, the concept of foundation model is create a basis of solving a lot of problems. But you see, a lot of problems in real life, in enterprise, for example, it may be slightly different or very different from when the base model is built. So that means after model is trained, you need to drive alignment for that foundation model to align towards your problem statement better. So it's called uptraining or instruction tuning. And then fine-tuning is kind of built on top of that. So that's kind of the reason why to do fine-tuning is basically align a base model better towards a specific problem. So specific workload patterns or specific vocabularies you have within your company for your application, for your new application, for your new product to deliver better results.

That's really interesting that you said you've gotta think about the goal of what you want your model to do with it. So, it sounds obvious when you say it out loud, but I think it's interesting that a lot of these sort of foundation models, you think of like GPT and the Anthropic, like the Claude models, they're designed to be able to write any form of text, but you don't necessarily want that in a business context. Like, actually you just want it to do sentiment analysis on financial statements or something like that, then it's a much narrower focus there. So that's pretty interesting. Do you want to talk me through what's the process for actually going through fine-tuning a model then? So you've decided on your goal, you've got your base model, how do you actually go about fine-tuning it?

So first, you need to create your evaluation dataset. As you said, the gold standard what is the ideal content you want to produce with a given prompt, right? So that is one that you need to establish. Without that, it's really hard to judge that you put all this effort fine-tuning, is that, you know, reaching your goal, progressing towards your goal, or going the other way. So that is very important to establish. And the second is you just kind of run your workload through whatever model you have and then be able to identify what are the failure cases. For the failure cases, then there are multiple possible next steps. One is you take the failure cases and you manually label what is the right outcome, and that becomes your fine-tuning dataset, right? This is supervised fine-tuning. Or with the failure cases, you can ask stronger model or ask user to give you what is the preference between two answers. Do you prefer this versus that? And then that build the preference dataset and then you can use that to do DPO preference-based tuning or you can just in your UI do thumbs up, thumbs down. It basically ask user to give you scores of whether they like the answer or not and use that to tune the base model. That's KTO, or there are many other different kind of tuning methods that you can use based on how close you are towards your end goals. And then you tune, and then you have your eval dataset. You never tune with your eval dataset. You have to keep that aside because that's your golden dataset to judge. And then you can see here how, you know, are you progressing in the right direction? Sometimes you fix one kind of problem, but the other problem, the other side stands still or the other side regresses and you kind of have to figure out. So those are kind of, I just talk about the mechanical part of fine-tuning or tuning in general, but they're actually much deeper than that because sometimes, hey, there's a failure case. It requires product design. It's a failure case because people haven't thought about how to handle it. In the right way. So it kind of impose a question. Even a product manager need to come in and make a decision. What is the right action if you see this beta case? So it's actually very interesting, as in fine-tuning is not just a mechanical process that machine learning engineer need to power through. It is actually a process that the product managers or product engineers along with machine learning engineer, they have to work together to make decisions.

That's fascinating. I suppose, yeah, you can think of it as like, we just gotta get a true/false answer for like, is this a good answer or not? But then from the product point of view, you need to decide what do you do if you get a bad answer? So it's gonna have to be built right into your product rather than just being this sort of abstract theoretical thing.

Okay.

That's interesting.

I forgot one thing actually. One very important thing is when you fine-tune, you also may need to make decisions because part of fine-tuning, as you said, is about how to align the model better towards your problem. So that's the delta part. But you also need to think about what kind of, for the base model, it's very capable of doing many things. What are the interesting characters you want to preserve from the base model, right? So some base model is really good at doing chat, some good base model is really good at long context and all these different capabilities. One thing is the model is really good at also forgetting during the training process. If your training data doesn't cover those captures you really want to preserve, it will forget about them. It will only learn the new skill set. So it's very important also to think through how you want to— like, the end result of the model, what kind of strength you want to train it to. Drive, and then you need to mix your label data for fine-tuning for your specific use case alignment along with some other dataset helps preserving other characters of the base model. So blending that properly is extremely important.

Do you have a sense of how much effort this all takes and how much benefit you're going to get from it? Like, what's the sort of performance increase that you're likely to get and how much effort do you have to put in to get that performance increase?

This is a really good question because typically people want to increase quality. They can go through two paths and it's kind of different level of effort. One path is prompt engineering. You can put instructions in the prompt to help align the model. So that's one approach. It's very quick. It's instant. You can Put the instruction there, and then you can test it and see how the model responds. It's very quick. Versus fine-tuning, it's a long process because it's also a multi-step process. You have to define evaluation data, as I mentioned, collect label data for fine-tuning, decide which approach you're going to fine-tune, and you're going to fine-tune and make sure fine-tuning will run. Sometimes it ooms, sometimes you got stuck somewhere. Also into conversion, all this. And then finally you get a model, then you need to test it again, use your eval data, and then you need to bring the model into production setting and send the prompt in. So it's a kind of much longer process than prompt engineering. So we have seen like at the early phase of pre-product market fit, and people have serious question, should I even spend time doing all this nine yards of fine-tuning? It's kind of I don't even know if you know where I'm going. So like people tend to use a much more powerful model so they don't worry about other production-related things, or they just prompt engineer their way out and validate if there's a path to work. And after they hit the product-market fit, and then all the other concerns like low latency and cost becomes an issue, and then they can spend more time to fine-tune. So we are seeing these patterns in terms of kind of be the most efficient from pre-power market fit to power market fit in production.

That's interesting. So I suppose then you've got a trade-off between, do I use this larger model, which is going to have probably higher performance out of the box, but it's going to cost more to run, versus do I use the smaller model and just pay the money to put the effort into fine-tuning and get the same performance, but then it's going to be cheaper to run in the long term. You've got this short-term, long-term trade-off cost then.

The interesting dynamics in the industry in the modeling space is the model quality is also changing, right? So the largest model, their quality keeps improving, but the small model, I call it small model stack, large model stack, small model stack, the small model stack quality is improving very fast as well. So almost every week there's a new small model that being announced for various different model provider, and they're kind of getting really close to those large model from various different kind of benchmark results. And that progress is very impressive. So it, it'll be interesting to observe how these dynamics play out that I will say when customer come to us and it's not clear they're still in the kind of product market fit stage, I will actually suggest them actually to try on the large model first. But with this new advancement, I think the small models are becoming more and more appealing even for pre-product market fit stage.

Okay, that's interesting that the small models are catching up. Actually, I'm not sure what the term for those is because small large language model doesn't seem to make any sense. Is there a proper word for that?

Yeah, so it's just a language model, maybe. Yeah, they're small, but they're actually very large. They're just small relatively to each other.

Okay, fair enough. Can you give some examples of tasks where you can get away with using one of these smaller models versus a task where you really do need that large best-in-class model?

Yeah, so we have seen a lot of advancement in, for example, code generation. Recently, I think Mistral announced the CodeStraw. That's their code generation model. It's very impressive. And about a week or two later, there's a new code generation model, DeepSeq Coder V2, that got announced. It beats everybody. It demonstrates like GPT-4O level quality. Of course, I want to caution everybody, benchmark comparison is one thing and what's real is another thing. So I will caution everyone to actually test it out by yourself. And that's kind of the beauty of the open source community. They are very generous in sharing their findings. So I think we also have a lot of people using us. We actually have a lot of feedback and we're always on the front of enable those cutting edge models for folks to try out. So far, like that's our observation is in the coding space, there's the relatively small models, like 70 billion models are getting really, really good. And the corresponding 7 billion and 8 billion models are also trading the 7 billion model quality pretty closely, but it can also significantly reduce latency in the call. So I think in the kind of small model space, it's getting more interesting now.

There's a bit more competition and progress is being made fast. So you said that you need to test your model on something just to make sure it actually works. Is that a case of typing lots of prompts, having a chat with the model, or is there something more thorough and robust you can do to test whether it works or not?

Yeah, so that goes back to your eval dataset, right? So I, I would strongly encourage the app builders or data scientists, they are like into GenAI space to prioritize having your golden dataset for your quality testing because The modeling space is fast moving. Whether you're on the big model stack or small model stack, there will be constantly new advancement that's happening. And if you want to stay on top of all this, you need to have your own dataset to validate how the new model advancement is giving back benefits to you. Sometimes it doesn't, right? Sometimes it even regresses. So I will say everyone's workload is different. So don't blindly trust the benchmark results, except for one thing. I have really good confidence the benchmark that lmsys.org is running, because they have humans just trying multiple models side by side and collect the ELO score. And that is more, it's kind of subjective, but also it's actually at the same time more objective than just purely the benchmarking results.

So this is the, the Chatbot Arena, and you can see which, which ones are the, uh, the top models.

Okay.

You mentioned this idea of an eval dataset. So suppose you've got, you're creating a chatbot or something. Is this going to be like an example of like the top 100 questions your users are going to ask, and you just see what the, the bot is going to generate for each of those?

Every chatbot probably have different focus. And the different kind of niche they want to tackle. For example, for some case, some chatbot really care about multilingual support and what they have specific language they want to cover, not just English, and that's important for them. For some chatbot/assistant, they really care about long context. Long context is not just a chat context, multi-term, but also And the process of generating response is the app will feed RAG as the input to help the response to be more relevant to the particular individual or particular scenario. So for some chat, like, long context is very important. And for some chat, they also want to not just doing text-based chat, they want to call out into other APIs to get the totality of the additional information. For example, get some kind of search result and get some stock price and get weather information, generate the image, and so on and so forth. So that is the direction we're moving forward, is to kind of help people build a compound AI system where you are not just using GenAI single model at a time. You want to be able to assemble multiple things together to solve a problem. We see that complexity in the chatbot, so we're, we're kind of building the system to make that job easier. So that coming back, it really depends. Again, if you're building an assistant to solve specific problem, whether it's medical, legal, educational, other things, coding assistant, so on, you need to have your golden evaluation dataset to sit on top of all these moving targets of models.

So really make sure you've got performance in the specific language you want, the specific industry or domain that you want, and just check that rather than just trying to test it on generic queries.

Okay.

Now, you mentioned that these smaller models are catching up in terms of performance, and that's obviously going to save money compared to using a larger model. Since a lot of companies are worried about the cost of putting generative AI into production, do you have any other money-saving tips? What else can you do beyond using smaller models to make things cheaper?

Yeah, so coming back to why it's so costly, right? So I briefly mentioned because the model size is just so big, it's a light of speed. Hey, you know, it's just kind of processing. The computational cost is just high, and that's one thing. But the other aspect is that, like, the GPUs are very powerful, right? So most of the generative models, though, they run on GPUs. Some run on Custom ASICs too. So those hardware are very powerful as well. And the other side of the cost is how utilized are those hardware? Are you pumping computation out of them constantly? Of course, ideal state is 100%. Of course, we cannot reach 100%, but getting close to 100% would be the best. So in order to drive utilization, then we need to kind of take a deeper look at the system resource consumption during the inference time. And the interesting part is, of course, the model has multiple components, and each component's bottleneck at different things. Like prompt processing is bottlenecked by just by flops, and generation, the content generation part is bottlenecked by memory bandwidth. So if you think about scaling, like increasing the throughput, that reduce cost by increasing throughput, you cannot increase throughput to address two bottlenecks at the same time. Does that make— you can, but it's much harder, right? So it's much easier to tackle one bottleneck at a time. So that requires us to think about the runtime of inference differently. So you kind of have to scale different parts of a model differently. And that's why we have— and sometimes you also need to go across beyond one node. You need to kind of have multiple GPU nodes, each node have 8 cards and 8 GPUs. You need to have multiple nodes to scale out to kind of be most efficient. So that's why we, we specialize in kind of the best scaling algorithm to kind of significantly reduce latency and increase throughput.

That's absolutely fascinating that, so you said the prompting side of things is limited by compute and then the generation side is limited by memory. So very different problems. So how much do individual companies or organizations need to worry about this infrastructure side of things? And how much is just sort of taken care of them by the platforms that they're using?

I think whoever is building these GenAI applications, it depends on the skill set. I think we work with many product teams, whether they are startups innovating on the product space or product team within incumbents. Enterprise. So the product team really want to move focus on product development, is focus on kind of helping the adoption and make their user happy. And they want someone else to handle infrastructure problems. And here it's not just about low latency and the cost efficiency. It's also about operation because inference is 24/7. It's just kind of, you have to make sure you have high quality of service and requires a team to kind of babysit the stack and so on. So usually it's very important for product team to have an infrastructure partner to help them. And so that's where we come in, right? So hey, we can be our infrastructure partner. But more interestingly, we're also seeing the ML/Infra team will like to accelerate the progress they can support their company with various different product teams building on top of them by kind of bringing vendors, bringing the state-of-the-art technologies, and blend the state-of-the-art technology into their existing infrastructure pieces. So we have been working with those big enterprise ML/Infra teams to help them move faster. And I think the fundamental reason is across the whole entire industry, from application stack to kind of platform stack to infrastructure stack, everything is moving so fast. And people recognize that. People recognize that. They would rather leverage and build on top of other people's work instead of taking on the full stack by themselves. And I, I actually have a huge respect of, of those engineers making those decisions because it's kind of the proud of being engineers. Hey, I am capable of doing everything. And it's very easy, actually easy to say, hey, there are open source libraries and the repos we can use to build the whole entire stack. We can have the full control, but there are leaders, they just so sharp, they understand where they want to focus and they understand where to bring other expertise in to help them. So it's a win-win. They can move like 2x, 10x faster that way. So I've seen a lot of those.

It sounds like the trade-off is if you've got an in-house AI engineering team on hand, then, you know, probably they can do some of this themselves. But for everyone else, you're going to want to partner with someone else, with another company that's going to help out with that infrastructure side of things.

Yeah. So even with in-house ML/Infra team, I think what I've seen them actively integrating the best technology from us, for example, in order to kind of supporting their other product teams better.

So it sounds like there can be quite a few teams involved in creating an AI product or adding AI to some product. So beyond the product team, you mentioned that some sort of infrastructure requirements, who else is needed? So which teams or roles need to be involved in creating an AI product?

We definitely see, I mean, the product team themselves, but usually it's the ML/Infra team. And the ML/Infra team, typically in the digital native enterprise, they already exist to bring small ML machine learning models into production, and they're kind of getting into larger deep learning models. And this GenAI is a huge leap forward. This model size is much bigger. It really needs beefier GPUs, the most state-of-the-art GPUs, and that kind of put a lot of stress on the infrastructure. But I think typically the ML team is another stakeholder.

That's interesting. And before you get started, like, I suppose if your CEO says, okay, we need to get some— get in on this AI game, we need to add AI to our product. What do you need to put in place before you can do this? So, what sort of processes need to be put in place? What other things, infrastructure needs to be put in place before you can actually be successful at this?

It depends on, like, AI is a very broad term. AI could mean you are doing simple classification, or it can be you're doing forecasting, you're doing ranking recommendation, or you are doing GenAI. So, the kind of characteristics of those workloads are quite different from each other. And we have seen kind of different companies are at different stage of this journey of the kind of AI evolution. So if we talk about the high end, like in GenAI space, then for the big company, typically, like here I'm talking about like Fortune 500 companies, and they may either use a GenAI API provider to get things going. Or they have already procured GPU, so they kind of procure GPU in-house in their cloud account or on-premise, and then they will bring different software running on top of those GPUs and to serve GenAI for them, whether it's fine-tuning or it's inference.

So basically you can get started quite quickly with just using one of these API services for generative AI, but you can afford the hardware, then you buy your own GPUs, and then you can start doing things in-house as well. I'm curious as to what success looks like. So there's a lot of companies where they've gone, okay, let's add some AI features to our products. How do you know if that's a good thing and if it's worked?

Again, there are different stages. I think that's just the nature of AI. It requires a lot of experimentation, and there are multiple things to experiment on. At the top, experimentation is the top of the funnel. You need to experiment on, hey, is this how you frame the problem, right? How you frame the problem you want to solve, is the framing right? Because framing will directly determine which technology you're going to pull into the stack to solve the problem. So that's kind of number one is testing out your framing and then you need to test out which model you want to use. You want to use big model or small model and then you would you may have some gaps either way. So then you need to kind of experiment. Do you prompt or do you fine-tune? So there are like, I would say it's a bushy tree of experimentation. You need to try in the kind of the top of funnel stage, but once you get into good quality and also kind of the product makes sense, then you kind of go down to the next level of the funnel. You go into early stage production, right? So by early stage production, then you need to worry about production quality metrics, including latency, including— may or may not be cost because it's not the scale, may not be there, but you want to start to kind of convert from experimentation code into production quality code, and then going to high-scale production where cost, it will become an issue. So yeah, going down this funnel, I think different stage you need to kind of work on different aspects.

So, there's quite a few stages to it. And yeah, once you've got things in production, are there any, have you got any concrete examples of like a metric that says, okay, this is what we measure and we know if you succeeded? Is it gonna be about like usage or is it gonna be about like quality of results? How do you go about saying this is good or not?

Again, so if you have, you can kind of dissect what you want to validate. The most important thing is validate the product, right? So by product, it's, you know, you have user, you have end consumer or developer using it. Then typically it goes by the user engagement metrics. Let's again, taking Coding Copilot as one example. So it's kind of if you generate code, then the acceptance rate of code, people that describe the quality, or just in general people's interaction, right? Even they decline, it's also interaction. It will count as interaction, but rather than people just ignore it and keep typing, it doesn't matter, I don't care, like, what you generate, right? So there are different ways to measure user engagement, and I think pump up user engagement is an important direction to focus on. And second is user adoption, right? So there are more people using it because they are fascinated by this new product experience, and they want to get onto it. So it's a natural expansion. Of the user base. So that's also very important. But then you go— that's all at the product level. Then you go one level down to validate the inference stack, right? So which support the product itself, the GenAI inference stack. Then it boils down to, hey, quality. But quality really depends on how you define your evaluation dataset and how the kind of changing inference stack is affecting it and how the inference stack help you reducing latency, or is the latency regress, so it affect user engagement, and how the inference stack help you be cost efficient across these two different areas, all these different areas. So it will be multiple layers of metrics, usually a product team have to watch on.

I like that, it's just simple stuff like, are users actually engaging with this, right through to, have they got a great experience, and do they actually enjoy the results? Have you seen any success stories from this? What's your favorite AI success story?

Oh yeah. So we have many assistants, including Copilot applications run on top of us, and we are also the leading provider in terms of our inference engine. Our latency is really, really good. And when people migrate to us, their latency drops by 5 times and their user engagement immediately goes up. They show us the curve. It's strongly correlated to user engagement. Pumped 2 times, 3 times, just the day switch over to us. That's very, very encouraging. So yeah, so we have plenty of those examples. And also when our customer— we also constantly help them upgrade to better models, and they show us, hey, with better models it generates much stronger content, and that makes us happy as well.

On the flip side, are there any common mistakes that organizations make when they're trying to make use of AI in their products?

I think one challenge we see across those, like this wave of adoption is they may move into customization too early where it's unclear what's their objectives, and then they're just busy going through the process of procure data, fine-tuning, which take a long time, get to a point they are super frustrated. It's unclear if they're moving the needle because the needle was never defined. but also it takes them a lot of time, and at the end it may not come out very strongly. Or the objective can be clear, but the process of curating data and learning all the kind of nitty-gritty details of how to fine-tune is too overwhelming, and then kind of they give up in the middle of doing that. And that's where we want to make it super easy as a kind of platform provider to remove those tedious work and the kind of time-consuming work and help our user focus on the kind of the intelligent parts where you can get the best ROI.

I like that the biggest problem is people not planning things and not knowing what they actually want the AI to do. So it just seemed like, yeah, spending a bit of time upfront and figuring out what you want is going to be pretty important to make sure it actually works. Earlier, you mentioned open-source models a few times. So do you have a sense of when open source models are better than their sort of the closed source equivalents or vice versa? Like when should you prefer one or the other?

So this is a really good question. I think it's not 100% correct, but I will coarsely correlate open source model with small models and the proprietary model, closed source with big models, right? We kind of coarsely, it's correct. So. It's— there's no question the big models are like just better quality overall because it's big and it kind of, it provides this ease of use, sense of ease of use because off the shelf you can have it do many things. So it's very easy to bootstrap on, on the big models. And I think that's why OpenAI did a great job and it's kind of their popularity speaks volumes of it. And then the kind of the obvious big models, it has a gig and it will be slower, it'll be more expensive. There's a, it's just like loss of physics there. And then the, of course the advantage of small model is, yeah, it's smaller and then it's faster, much, much faster and, and more cost efficient. But then the quality is not there, right? So quality has a gap. And then the most important thing is, yes, latency and the cost is very important for production. And production scaling, but if quality is not there, it doesn't make sense also, right? So, so how to shrink down the quality gap is very important before you can start to use small models.

Okay. So really, you're juggling a few different sort of competing criteria then around cost and performance, and you mentioned latency again, so like the speed of generation of output. Just to wrap up, what are you most excited about in the world of AI?

In the world of AI, definitely excited about generative AI as a very unique category, but within generative AI, I'm very excited about applications built on top of it, and there are really great applications that I have seen already for like consumer-facing, developer-facing products, but recently there's emerging applications that focus on having agents, specialized agents, on solving specific problems really well to have them coordinated with each other and to solve a more complex problem. I think that direction is super exciting because what I'm seeing that from AlphaGo experience, AlphaGo can learn and improve the skills of playing Go by playing Go with itself. And that level of autonomous learning is going to now get into the agentic world and that will pump up the intelligence. Even more. So, so I'm, I'm super excited about that direction.

Yeah, that's pretty cool. Just being able to learn by playing against yourself. So yeah, it's cool that AI can improve itself. Mildly scary in some respects, but I think it's how a lot of distributions are structured.

We're far away, we're far, far, far away from there, but yeah, that, that direction is very interesting.

Yeah, absolutely. So, and do you have any final advice for organizations wanting to put AI into their products?

I've been working with many organizations. They are kind of very wise in talking to the experts in the area. I personally spent a lot of time talking with organizations, one is kind of for me to see what's important for them, but also for them to kind of, like, if I have any suggestion for them, I will kind of interact back with them and help them solve problems. I think that pattern, like, tend to work out really well, especially where at the early stage of this evolution. And I will kind of strongly encourage organizations think about adopting the cutting-edge technology to build a rapport with technical expertise in, in the space. It can be a roundtable, it can be some kind of discussions, just kind of help you create a good mental picture of where it makes most sense for you to invest in and what are the commercial opportunities for your business and where you should put your biggest effort behind.

Okay. Yeah. I love that idea of just having a discussion about what you actually want, whether it's with like bringing those two involved in the commercial side of things and the technical side of things together. So yeah, that just seems incredibly important. All right. Super. Thank you so much for your time then. It was a great discussion.

Awesome. It's very nice talking with you too.