#241 Getting Generative AI Into Production with Lin Qiao, CEO and Co-Founder of Fireworks AI

Hi, I'm Maggie, VP of Curriculum at datacamp. And I know firsthand how overwhelming it can be to keep up with the rapid changes in data and AI. But staying ahead doesn't have to be hard. That's why our curriculum team has curated the best learning content to help you and your team stay ahead, covering everything from data and AI literacy for busy executives to cutting edge generative AI for developers, and everything in between. Whether you're upskilling yourself or your team, we make learning practical, engaging and flexible. It just works. Power your data and AI transformation journey on datacamp.com.

I think that's just the nature of AI. It requires a lot of experimentation and there are multiple things to experiment on. Experimentation is the top of the funnel. You need to experiment on how you frame the problem right. How you frame the problem you want to solve is the framing right? Because framing will directly determine which technology you're going to pull into the stack to solve the problem.

Welcome to Data Framed. This is Richie. Playtime is over for generative AI. There's increasing pressure to make sure that all those experiments and prototypes you've been building get put into production. That's easier said than done. So today we're going to have a look at how to do it. In particular, a lot of the challenge is around making sure you have good enough performance to satisfy your customers, while keeping the costs low enough to satisfy your cm. There are some technical solutions, but you also need to change your processes too, if you want to create a reliable pipeline for Genai products and features. Naturally, we'll consider both angles in this show. Our guest is lin Kiao, the CEO and co founder at the Generative AI Inference Platform. Fireworks AI, I.e. lyn, helps developers create and deploy generative AI models. Before founding Fireworks AI in 2022, Lynne had an extensive career in technology, most notably leading the Pytorch team at Meta, where she played a critical role in developing AI frameworks and platforms such as Cafe2 and PyTorch. So I'm excited to hear her advice. Hi Linh, thank you for joining me on the show.

Hi, thanks for having me. I'm super excited to have a chat with you today.

Yeah, this is going to be fun. So, just to begin with, can you talk me through what are the most common uses of generative AI that you're seeing today?

Oh, that would be a. It would take one hour myself. So yeah, so it is absolutely. This technology is so empowering and so disruptive. We have absolutely seen a lot of interesting, innovative applications of Products built on top of our service tier. Of course BIOS offer inference engine for JNI models. So I will kind of bucketize them in different ways. So one big bucket is assistance. So we have seen people building medical assistance to address the shortage of doctors or nurses and help those multiple workers become productive. We have been seeing people building, many people building educational systems target different cohort. For example there are people building applications for students to learn, for researchers to do research and for people to learn foreign languages. So it kind of cross border various different applications education. There are also legal assistance where they find people to do case study and because kind of lawyers are expensive or help lawyer to become more productive. So I think that's a big chunk of application and then we see a lot of chat, just kind of interesting chat based application. And usually those chat based application combine voice with it. So it's not just hey, you have to type, you can voice in, voice out. And those are also very popular. Another big bucket is API generation or code generation. It's not for end consumers, it's for another program. So this is important because we are seeing the compounding effects of people building application on top of not just single model but multiple models or multiple modalities, or from multiple modalities to additional APIs. So there are example, the totality of knowledge. I can drill down more to that because single model has limited, very limited knowledge by itself. And the specific use case are like generating business workflows, generating SQLs, generating charts about the production usage, monitoring, observability and so on. So that's a huge bucket. Those are kind of across the board where we're super excited of what's happening.

I feel like if you continued you could probably just listed like every industry that there is. But that's really interesting that you got stuff that is applying to end users. So you mentioned things like productivity improvements and training. Then you've also got that sort of middleware layer. So it's for other people building technology as well. Since today we're going to talk about getting things into products and into production. I'd like to know just at a high level, what are the different steps for getting generative AI into a product?

First of all, I would say application built on top of gen now or application pre gen. There are a few differences. One is pre geni application built on top of CPUs. It's heavily commoditized. The stack software stack is very mature and the tools are kind of abundant. So it's kind of easy and low cost. With gen AI shifted right. GPU is new to most of people and the GPU is very expensive. GPU is power hungry power expensive power generate heat, Heat requires cooling, cooling is expensive. So the whole cost structure of productionized application on top of Genai significant changed. Right now that's one and second between before and after. Whether you're building on JNI or not. One thing that doesn't change is most of those application is consumer facing, is developer facing. It needs to be very interactive. So latency is a critical part of product experience. Holding the latency bar doesn't change before and after. Another interesting thing that is different is before Genai the logic, the reasoning logic is coded in the application itself or is coded in the surrounding infrastructure. So it's deterministic and you can debug and the logic is there with Geni. The Geni models have multiple layers and with multiple layers of prediction and it's probabilistic. Probabilistic means if you ask the general model to produce something, it will do it regardless whether it's true or not. That directly introduce a new phenomenon, hallucination which application developer never need to deal with. Now they are discussing their head like oh this is bad and I need to learn how to control or minimize hallucination in my product, in my application. So those are the kind of one invariant and two variants that's happening in kind of pre Geni and the current Geni era. And because of that that introduced new production challenges for our app developers. For the invariant, low latency as we know the Geni models is the extremely big side of machine learning models. And because the model is so big and the latency will be high and not having a low latency product experience will be break or make it scenario for the app developers. So that is extremely important, we need to address that. And the cost is also important factor. Even if you have a viable product, if the cost is very high and the nature of consumer facing developer facing apps, they scale quickly. If you hit a product market bid when you scale quickly, you're losing money at a small scale, you're going to bankrupt at a large scale. So controlling cost extremely important to go into production and also to produce great product quality and content, you have to control hallucination and learning the skill set how to do that extremely important. So I think those are the kind of three critical parts to bring Genai technology into production.

Lots to unpack then lots of interesting changes. So certainly you mentioned like the switch from using CPUs to GPUs Nvidia's done very well out of this switch.

Absolutely, definitely.

And then the idea of going from deterministic programming to probabilistic stuff, I feel there's very different reactions there. Like all the software engineers are panicking and then you speak to data scientists and like, isn't that normal for code to be probabilistic? So yeah, very interesting changes in differences in opinion there. We'll maybe get back into talking about latency later on, but for now I'd like to talk about fine tuning since this seems like quite an important sort of stage in making sure that you have high quality results. So can you just give me a quick introduction to like, why you might want to fine tune a large language model rather than just taking something off the shelf.

So here I want to talk a little bit about the process of model training. Right, so this is before fine tuning to train a foundation model or train a model in general. Before you train the researchers, they need to be alternated on what is the objectives for the end result they want to achieve. So that whole process will determine what kind of problem you care the most, what kind of problem you are okay with, and what kind of problem you don't even care. From there you go and curate your training data. And for the areas you care the most, you want the most high quality data, you want the most diversified data, and it requires a lot of care. And for areas you just want to have some courage and it's okay, then you don't have to spend too much time, but the volume needs to be there. For the areas you don't care, then you don't really actively source data to cover those areas and the end result of training will reflect that. Right. So that's interesting because then when people build foundation models, the concept of foundation model is created basis of solving a lot of problems. But you see a lot of problems in real life. In enterprise, for example, it may be slightly different or very different from when the base model is built. So that means after model is trained, you need to drive alignment for that foundation model to align towards your problem statement better. So it's called up training or instruction tuning. And then fine tuning is kind of built on top of that. So that's kind of the reason why to do fine tuning is basically align a base model better towards specific problems or specific workload patterns or specific vocabularies you have within your company for your application, for your new application, for your new product to deliver better results.

That's really interesting that you said you've got to think about the goal of what you want your model to do with it. So it sounds obvious when you say it out loud, but I think it's interesting that a lot of these sort of foundation models you think of like GPT and anthropic, like the Claude models, they're designed to be able to write any form of text, but you don't necessarily want that in a business context. Like actually you just wanted to do sentiment analysis on financial statements or something like that. Then it's a much narrower focus there. So that's pretty interesting. Do you want to talk me through what's the process for actually going through fine tuning a model then? So you've decided on your goal, you've got your base model. How do you actually go about fine tuning it?

So first you need to create your evaluation data set. As you set the gold standard, what is the ideal content you want to produce with a given prompt? Right. So that is one that you need to establish. Without that it's really hard to judge that you put all this effort fine tuning is that reaching your goal, progressing towards your goal, going the other way. So that is very important to establish. And the second is you just kind of run your workload through whatever model you have and be able to identify what are the failure cases. For the failure cases, then there are multiple possible next steps. One is you take the failure cases and you manually label what is the right outcome and that becomes your fine tuning data set. This is supervised fine tuning. Or with the failure cases you can ask stronger model or ask user to give you what is the preference between two answers. Do you prefer this versus that and then that build the preference data set and then you can use that to do DPO preference based tuning. Or you can just in your UI do thumbs up, thumbs down. It basically ask user to to give you scores of whether they like the answer or not and use that to tune the base model. That's kto or there are many other different kind of tuning methods that you can use based on how close you are towards your end goals. And then you tune and then you have your eval data set. You never tune with your eval data. You have to keep that aside because that's your golden data set to judge. And then you can see here how are you progressing in the right direction. Sometimes you fix one kind of problem, but the other side stands still or the other side regress and you kind of have to figure out. So those are kind of. I just talk about the mechanical part of fine tuning or Tuning in general, but they're actually much deeper than that because sometimes, hey, there's a failure case, it requires part of design. It's a failure case because people haven't thought about how to handle it in the right way. So it kind of impose a question. Even a product manager need to come in and make a decision what is the right action if you see this better case. So it's actually very interesting as in, fine tuning is not just a mechanical process that machine learning engineer need to power through. It is actually a process that the product managers or product engineers along with machine learning engineers, they have to work together to make decisions.

That's fascinating. I suppose. Yeah, you can think of it as like we just got to get a true false answer for like is this a good answer or not? But then from the product point of view, you need to decide what you do if you get a bad answer. So it's going to have to be built right into your product rather than just being this sort of abstract theoretical thing. Okay, that's interesting.

I forgot one thing. Actually, one very important thing is when you fine tune, you also may need to make decisions because part of fine tuning data set is about how to align the model better towards your problem. So that's the delta part. But you also need to think about what kind of for the base model, it's very capable of doing many things. What are the interesting characters you want to preserve from the base model? So some base model is really good at doing checks. Some good base model is really good at long context and all these different capabilities. One thing is the model is really good at also forgetting during the training process. If your training data doesn't cover those captures you really want to preserve, it will forget about them. You only learn the new skill set. So it's very important also to think through how you want to the end result of the model, what kind of strength you want to try drive. And then you need to mix your label data for fine tuning for your specific use. Case alignment along with some other data set helps preserving other characters of the base model. So blending that properly is extremely important.

Do you have a sense of how much effort this all takes and how much benefit you're going to get from it? Like what's the sort of performance increase that you're likely to get and how much effort do you have to put in to get that performance increase?

This is a really good question because typically people want to increase quality. They can go through two paths and it's kind of different level of effort. One path is prompt engineering. You can put instructions in the prompt to help align the model. So that's one approach. It's very quick, it's instant. You can put the instruction there and then you can test it and see how the model responds. It's very quick versus fine tuning. It's a long process because it's also multi step process. You have to define evaluation data as I mentioned, collect label data for fine tuning, decide which approach you're going to fine tune and you're going to fine tune and make sure fine tuning will run. Sometimes it ooms, sometimes it got stuck somewhere. Also into convergence and all this and then finally get a model, then into test it again, use your eval data and then you need to bring the model into production setting and send the prompt in. So it's a kind of much longer process than prompt engineering. So we have seen like at the early phase of pre product market fit and people have serious question should I even spend time doing all this nine yards of fine tuning? It's kind of, I don't even know if you know where I'm going. So people tend to use a much more powerful model so they don't worry about other production related things or they just prompt engineering their way out and validate if there's a path to work. And after they hit the product market fit and then all the other concerns like low latency and cost becomes an issue and then they can spend more time to fine tune. So we are seeing these patterns in terms of kind of be the most efficient from pre product market fit to part of market fit in production.

That's interesting. So I suppose then you've got a trade off between do I use this larger model which is going to have probably higher performance out of the box, but it's going to cost more to run versus do I use a smaller model and just pay the money to put the effort into fine tune and get the same performance. But then it's going to be cheaper to run in the long term. You've got this short term, long term trade off cost.

Then the interesting dynamics in the industry in the modeling space is the model quality is also changing, right? So the largest model, their quality keeps improving. But the small model, I call it small model stack, large model stack, small model stack. The small model stack quality is improving very fast as well. So almost every week there's a new small model that being announced for various different model provider and they are kind of getting really close to those large model from various different kind of benchmark Results and that progress is very impressive. So it will be interesting to observe how these dynamics play out that I would say when customer come to us and it's not clear they are still in the kind of product market fit stage. I will actually suggest them actually to try on the large model first. But with this new advancement I think the small models are becoming more and more appealing even for pre product market fit stage.

Okay, that's interesting that the small models are catching up. Actually I'm not sure what the term for those is because small, large language model doesn't seem to make any sense. Is there a proper word for that? Yeah, so it's just a language model maybe?

Yeah, they're small, but they're actually very large. They're just small relatively to each other.

Okay, fair enough. Can you give us some examples of tasks where you can get away with using one of these smaller models versus a task where you really do need that large best in class model.

Yeah, so we have seen a lot of advancement in e.g. cO generation. Recently I think Mistral announced the code straw. That's their code generation model is very impressive. And about a week or two later there's a new cogeneration model, Deepseek Coder V2 that got announced. It beats everybody. It demonstrates like GPT4O level quality. Of course I want to caution everybody, benchmark comparison is one thing and what's real is another thing. So I will caution everyone to actually test it out by yourself. And that's kind of the beauty of the open source community. They are very generous in sharing their findings. So I think we also have a lot of people using us. We actually have a lot of feedback and we're always on the front of enable those cutting edge models for folks to try out. So far like that's our observation is in the coding space, the relatively small models, like 70 billion models are getting really, really good. And the corresponding 7 billion 8 BNA models are also trading the 7 billion model quality pretty closely. But it can also significantly reduce latency in the cost. So I think in the kind of small model space, it's getting more interesting.

Now there's a bit more competition and progress is being made fast. So you said that you need to test your model on something just to make sure it actually worked. Is that a case of typing lots of prompts, having a chat with the model, or is there something more thorough and robust you can do to test whether it works or not?

Yeah, so that goes back to your eval data set. So I would strongly Encourage the app builders or data scientists that are into Genai space to prioritize having your golden data set for your quality testing because the modeling space is fast moving. Whether you're on the big model stack or small model stack, there will be constantly new advancement that's happening. And if you want to stay on top of all this, you need to have your own data set to validate how the new model advancement is giving back benefits to you. Sometimes it doesn't, right? Sometimes even regress. So I would say everyone's workload is different, so don't blindly trust the benchmark results. Except one thing. I have really good confidence the benchmark that LLMSYS.org is running because they have humans just trying multiple models side by side and collect the ELO score. And that is more. It's kind of subjective, but also it's actually at the same time more objective than just purely the benchmarking results.

So this is the chatbot arena and you can see which ones are the top models. Okay, you mentioned this idea of an eval data set. So suppose you've got, you're creating a chatbot or something. Is this going to be like an example of like the top 100 questions your users are going to ask and you just see what the bot is going to generate for each of those.

Every chat bot probably have different focus in the different kind of niche they want to tackle. For example, for some case, some chatbots really care about multilingual support and they have specific language they want to cover, not just English and that's important for them. For some chatbot assistant they really care about long context. Long context is not just a chat context multi term but also and the process of generating response is the app will feed rag as the input to help the response to be more relevant to the particular individual or particular scenario. So for some chat like long context is very important. And for some chat they also want to not just doing text based chat, they want to call out into other APIs to get the totality of the additional information. For example, get some kind of search result and get some stock price and get weather information, generate the image and so on and so forth. So that is the direction we're moving forward is to kind of help people build a compound AI system where you are not just using genai single model at a time. You want to be able to assemble multiple things together to solve a problem. We see that complexity in the chatbot, so we're kind of building the system to make that job easier so that coming Back, it really depends. Again, if you're building a system to solve specific problem, whether it's medical, legal, educational, other things, coding assistance on, you need to have your golden evaluation data set to sit on top of all these moving targets of models.

So really make sure you've got performance in the specific language you want, the specific industry or domain that you want, and just check that rather than just trying to test it on generic queries. Okay. Now you mentioned that these smaller models are catching up in terms of performance and that's obviously going to save money compared to using a larger model. Since a lot of companies are worried about the cost of putting generative AI into production. Do you have any other money saving tips? What else can you do beyond using speed smaller models to make things cheaper?

Yeah. So coming back to why it's so costly. Right. So I briefly mentioned because the model size is just so big, it's a line of speed. Hey, you know, it's just kind of processing. The computational cost is just high and that's one thing. But the other aspect is like the GPUs are very powerful, right? So most of the Geni models though, they run on gpu. Some run on custom ASICS too. So those hardware are very powerful as well. And the other side of the cost is how utilized are those hardware? Are you pumping competition out of them constantly? Of course, ideal state is 100%. Of course we cannot reach 100%, but getting close to 100% would be the best. So in order to drive utilization then we need to kind of take a deeper look at the system resource consumption during the inference time. And the interesting part is of course the model has multiple components and each components are bottlenecked at different things like prompt processing is bottlenecked by just by flops and generation. The content generation part is a bottleneck by memory bandwidth. So if you think about scaling like increasing the throughput that reduce cost by increasing throughput, you cannot increase throughput to address two bottlenecks at the same time. Does that mean you can? But it's much harder, Right? So it's much easier to tackle one bottleneck at a time. So that requires us to think about the runtime of inference differently. So you kind of have to scale different parts of a model differently. And that's why we have. And sometimes you also need to go across beyond one node. You indicate have multiple GPU nodes. Each node have eight cards and eight GPUs. You need to have multiple nodes to scale out to kind of be most efficient. So that's why we specialize in kind of the best scaling algorithm to kind of significantly reduce latency and increase throughput.

That's absolutely fascinating. So you said the prompting side of things is limited by compute and then the generation side is limited by memory. So very different problems. So how much do individual companies or organizations need to worry about this infrastructure side of things and how much is just sort of taken care of them by the platforms that they're using?

I think whoever is building these genius applications, it depends on the skill set. I think we work with many product teams whether they are startups innovating on the product space or product team within incumbents enterprise. So the product team really want to move focus on product development. It's focused on kind of helping the adoption and make their user happy and they want someone else to handle infrastructure problems. And here it's not just about low latency and cost efficiency, it's also about operation because inference is 24 by 7. It's just kind of you have to make sure you have high quality of service and requires a team to kind of babysit the stack and so on. So usually it's very important for product team to have an infrastructure partner to help them. So that's where we come in, right? So hey, we can be our infrastructure partner. But more interestingly we're also seeing the ML info team will like to accelerate the progress they can support their company with various different product teams building on top of them by kind of bringing vendors bringing the state of our technologies and blend the state of our technology into into their existing infrastructure pieces. So we have been working with those big enterprise ML infra team to help them move faster. And I think the fundamental reason is across the whole entire industry from application stack to kind of platform stack to infrastructure stack, everything is moving so fast and people recognize that, people recognize that they would rather leverage build on top of other people's work instead of taking on the full stack by themselves. And I actually have a huge respect of those engineers making those decisions because it's kind of the proud of being an engineer is hey I am capable of doing everything. And it's very easy actually easy to say hey there are open source libraries and the repos we can use to build the whole entire stack. We can have the full control. But their leaders, they just so sharp, they understand where they want to focus and they understand where to bring other expertise in to help them. So it's a win win. They can move like 2x10x faster that way. So I've seen a Lot of those.

It sounds like the trade off is if you've got an in house AI engineering team on hand, then probably they can do some of this themselves. But for everyone else, you're going to want to partner with someone else with another company that's going to help out with that infrastructure side of things.

Yeah. So even with in house ML infra team, I think we have seen them actively integrating the best technology from us, for example, in order to kind of supporting their other product teams better.

So it sounds like there can be quite a few teams involved in creating an AI product or adding AI to some product. So beyond the product team, you mentioned that some sort of infrastructure requirements, who else is needed? So which teams or roles need to be involved in creating an AI product?

We definitely see, I mean the product team themselves, but usually it's the ML info team and the Mr. Info team. Typically in the digital native enterprise they already exist to bring small ML machine learning models into production and they are kind of getting into larger deep learning models. And the gen AI is a huge leap forward. This model size is much bigger. It really needs beefier GPUs, the most data large GPUs and that kind of put a lot of stress on their infrastructure. But I think typically my info team is another stakeholder that's interesting.

And before you get started, like I suppose if your CEO says okay, we need to get some getting on this AI game, we need to add AI to our products. What do you need to put in place before you can do this? So what sort of processes need to be put in place? What other things infrastructure needs to be put in place before you can actually be successful at this?

It depends on like AI is a very broad term. AI could mean you are doing simple classification or it can be you're doing forecasting, you're doing ranking, recommendation, or you are doing genai. So the kind of characteristics of those workloads are quite different from each other. And we will have seen kind of different companies are at different stage of this journey of the kind of AI evolution. So if we talk about the high end like in gen AI space then for the big company, typically like here I'm talking about Fortune 500 companies and they may either use a Geni API provider to get things going or they have already procured gpu. So they kind of procure GPU in house, in their cloud account or on premise and then they will bring different software running on top of those GPUs and to serve Geni for them, whether it's fine tuning or it's inference.

So basically you can get started quite quickly with just using one of these API services for generative AI. But for you can afford the hardware, then you know, you buy your own GPUs and then you can start doing things in the house as well. I'm curious as to what success looks like. So there's a lot of companies where they've gone, okay, let's add some AI features to our products. How do you know if that's a good thing and if it's worked?

Again, there are different stages. I think that's just the nature of AI. It requires a lot of experimentation and there are multiple things to experiment on at the top. Experimentation is the top of the frenell you need to experiment on. Hey, is this how you frame the problem? Right? How you frame the problem you want to solve is the framing? Right? Because framing will directly determine which technology you're going to pull into the stack to solve the problem. So that's kind of number one is testing out your framing and then you need to test out which model you want to use. You want to use big model or small model and then you would, you may have some gaps either way. So then you need to kind of experiment. Do you prompt or do you fine tune? So there are like, I would say it's a bushy tree of experimentation. You need to try in the kind of the top inferno stage. But once you get into good quality and also kind of the product makes sense, then you kind of go down to the next level. You go into early stage production. Right? So by early stage production production then you need to worry about production quality metrics including latency, including may or may not be cost because it's not. The scale may not be there, but you want to start to kind of convert from experimentation code into production quality code and then going to high scale production where cost it will become issue. So yeah, going down this funnel, I think different stage, you need to kind of work on different aspects.

So there's quite a few stages to it. And yeah, once you've got things in production, are there any, have you got any concrete examples of like a metric that says, okay, this is what you measure and we know if you succeeded, is it going to be about like usage or is it going to be like quality of results? How do you go about saying this is good or not?

Again, so if you have, you can kind of dissect what you want to validate. The most important thing is validated product. Right? So by product it's you know, you have user, you have end consumer or developer using it. Then typically it goes by the user engagement metrics. Let's again taking coding copilot as one example. So it's kind of if you generate code then the acceptance rate of code people that describe the quality or just in general people's interaction. Right. Even they decline. It's also interaction, it will count as interaction. But rather than people just ignore it and keep typing, it doesn't matter, I don't care what you generate. So there are different ways to measure user engagement. And I think pump up user engagement is important direction to focus on. And second is user adoption. So there are more people using it because they are fascinated by this new product experience and they want to get onto it. So it's a natural expansion of the user base. So that's also very important. But then you go that's all at the power level. Then you go one level down to validate the inference stack. Right. So which support the product itself the gen inference stack. Then it boils down to hey quality. But quality really depends on how you define your evaluation data set and how the kind of changing inference stack is affecting it. And how the inference stack help you reducing latency or is the latency regress so it affect user engagement and how the inference stack help you be cost efficient across these two different areas. All these different areas. So it will be multiple layers of metrics. Usually a product team have to watch on.

I like that. It's just simple stuff like are users actually engaging with this right through to have they got a great experience and do they actually enjoy the results? Have you seen the success stories from this? What's your favorite AI success story?

Oh yeah. So we have many assistants coding copilot applications run on top of us and we are also the leading provider in terms of our inference engine. Our latency is really really good. And when people migrate to us their latency drop by five times and their user engagement immediate goes up. They show us the curve. It's strongly correlated user engagement pump diet two times, three times just the day switch over to us. That's very, very encouraging. So yeah, so we have plenty of those examples. And also when our customer we also constantly help them upgrade to better models and they show us hey with better models it generates much stronger content and that makes us happy as well.

On the flip side, are there any common mistakes that organizations make when they're trying to make use of AI in their products?

I think one challenge we see across those like this wave of adoption is they may move into customization too early where it's unclear what's their objectives and then they're just busy going through the process of procure data fine tuning which take a long time get to a point they are super frustrated. It's unclear if they are moving the needle because the needle was never defined. But also it takes them a lot of time and at the end it may not come out very strongly or the objective can be clear, but the process of querying the data and learning all the kind of nitty gritty details of how to fine tune is too overwhelming. And then kind of they give up in the middle of doing that. And that's where we want to make it super easy as a kind of platform provider to remove those tedious work and the kind of time consuming work and help our user focus on the kind of the intelligent part where you can get the best roi.

I learned that the biggest problem is people not planning things and not knowing what they actually want the AI to do. So it just seemed like yeah, spending a bit of time up front and figuring out what you want is going to be pretty important to make sure it actually works. Earlier you mentioned open source models a few times. So do you have a sense of when open source models are better than the closed source equivalents or vice versa? Like when should you prefer one or the other?

So this is a really good question. I think it's not 100% correct, but I will coarsely correlate open source model with small models and the proprietary model close source with big models. Right, we kind of coarsely it's correct. So there's no question the big models are like just better quality overall because it's big and it provide this ease of use sense of ease of use because off the shelf you can have it do many things. So it's very easy to bootstrap on other big models and I think that's why OpenAI did a good great job. And it's kind of their popularity speaks volume of it and then the kind of obvious big models because it's big and it will be slower, it will be more expensive. It's just like loss of physics there. And then of course the advantage of small model is yeah, it's smaller and then it's faster, much, much faster and more cost efficient. But then the quality is not there. Right. So quality has a gap and then the most important thing is yes, latency and cost is very important for production and production scaling. But if quality is not there, it doesn't make sense also. Right. So how to shrink down the quality gap is very important before you can start to use small models.

Okay, so really you're juggling a few different sort of competing criteria then around cost and performance and you manage latencies again. So like the speed of generation of output just to wrap up. What are you most excited about in the world of AI?

In the world of AI, definitely excited about generative AI as a very unique category. But within generative AI I'm very excited about applications built on top of it. And there are really great application that I have seen already for like consumer facing, developer facing products. But recently there's emerging application that focus on have agents, specialized agent on solving specific problem really well to help them coordinate with each other and to solve a more complex problem. I think that direction is super exciting because what I'm seeing that from AlphaGo experience, AlphaGo can learn and improve the skills of playing GO by playing goal with itself. And that level of autonomous learning is going to not get into the agentic world and that will pump up the intelligence even more. So I'm super excited about that direction.

Yeah, that's pretty cool just being able to learn by playing against yourself. So yeah, it's cool that AI can improve itself. Mildly scary in some respects. I think it's how a lot of.

We'Re far away, we're far, far, far away from there. But yeah, that direction is very interesting.

Yeah, absolutely. So and do you have any final advice for organizations wanting to put AI into their products?

I've been working with many organizations. They are kind of very wise in talking to the experts in the area. I personally spend a lot of time talking with organizations. One is kind of for me to see what's important for them but also for them to kind of like if I have any suggestion for them I will kind of interact with them and help them solve problems. I think that pattern tend to work out really well, especially we're at the early stage of this evolution. And I will kind of strongly encourage organizations think about adopting the cutting edge technology to build a rapport with technical expertise in the space. It can be a roundtable, it can be some kind of discussions just kind of help you create a good mental picture of where makes most sense for you to invest and what are the commercial opportunities for your business and where you should put your biggest effort behind.

Okay. Yeah, I love the idea of just having a discussion about what you actually want, whether it's with like bringing those doing both on the commercial side of things and the technical side of things together. So, yeah, that does seem incredibly important. All right, super. Thank you so much for your time then. It was a great discussion.

Awesome. It's very nice talking with you, too.