Building AI Infrastructure with Benchmark-backed Fireworks AI Co-Founder Lin Qiao

when companies like startups or hobbyists, they don't know, they just want to have some fun and play with something or validate the biggest hypothesis is, is LLM even the solution to my problem? Right. And the best starting point is to start with the most powerful LLM and see if it can solve the problem or not. So because of that, many people start from OpenAI and the inflection point is when they hit product market fit, right? When their product starts making sense, and then all the product operation comes into the picture. I think the unit economy is going to really break this one-size-fits-all model. That's where our philosophy is opposite of OpenAI. Like, we don't believe, hey, one model is going to magically solve all the problems with the best quality, with the best latency, with the best TCO. It's just That doesn't work. That word doesn't exist. Instead, we're going the opposite direction. We believe in specialized customized model for each individual business task. And we'll offer hundreds of those models. We're going to put a really great router that sits on top of hundreds of them. And eventually to our user, they don't need to understand, oh, which model now I have a problem, which model should I pick to solve my problem? I have a discovery problem.

Welcome to Autopilot, a podcast where we discuss the past, present, and future of AI and automation.

From Atlanta, Fulton County, it's one small step for man.

We explore how AI is creating and disrupting large industries in conversations with top founders, investors, and historians. I'm your host, Will Summerlin. Let's dive in. Lin, thank you for joining the Autopilot Podcast. So today we're going to have a great conversation about open source versus closed source, AI infrastructure, lots of other interesting topics. You just raised a Series A from Sequoia and Benchmark, which I think speaks to quality of you as a founder and the opportunity around what you're building. But before we get into all that, maybe we can start with you. Can you just take a minute and share your background and how you got to found Fireworks?

Hi, Will. Thanks for having me. So yeah, so we started Fireworks September 2022. Before then, in the latest 5 years, I was head of PyTorch at Meta, working together with my co-founders at Fireworks. Mainly our focus is— at that time, Meta was ahead of the industry. Typically, that's the case in terms of adopting AI and being an AI-first company. Our mission is to transition the whole company to build the product massively using AI technology. As part of that, we built PyTorch. PyTorch is currently the dominating AI framework in the industry. As well as building the infrastructure purpose-built around PyTorch from ground up and supporting a huge variety of AI workload at Meta. By the time we left, we were running more than 50 trillion inference per day across a massive number of data centers globally. So that's who we are. And interestingly, through many kind of open source engagement, we we see the trend that many big companies, big or small, across industry, they are also going through the AI-first transition. We saw excruciating pain they're going through because they don't have the right talent, they don't know the know-hows yet, they don't have the right software stack set up. Operating GPU in production is painful. So across the board, we see a huge contrast of desire to move really fast to be at first and obstacles to get there. So these big contracts makes us believe we're going to land a huge amount of impact by creating Fireworks in the industry. So that's our founding story.

That's awesome. Yeah, 50 trillion inferences a day. That's absurd. When you think back about that, like what was the most complex problem you guys dealt with? Trying to operate at that scale at Meta?

Um, I would say actually the complex problem is operating production is simplify the complexity. That's actually, it is the complexity. Um, because at the Meta scale, it's 24/7. Uh, there are a lot of, a lot of consumers, billions of consumers using the product. And 1 minute of downtime is massive amount of impact. So how to build a resilient system? A few things also come into picture, right? Because it's a, it's a B2C, Nomada is probably the biggest B2C app and product. So it requires very low latency. The experience that users have need to be instantaneous. And so that's a characteristic of many consumer, prosumer developer. Product, and we know how to bring it down drastically, how to optimize for production workload. And also, it's not one size fits all, because from ads, from newsfeed, from video, from site integrity, from many AR/VR effects, the workload patterns are all different. And we develop a system that customized towards workload pattern in terms of produce low latency. So one size fits all, there's work. That's kind of one big lesson we learned. And the second is for all these consumer, prosumer, developer-facing apps and products, TCO is very important because the volume is so big and your cost scale with your volume, right? Yeah. So if you're not careful, you can quickly get to a stage, oh, my money is running out because I'm burning way faster than I got money in. Sure. And at that scale, you don't have any control. So tightly controlled TCO is very important. And of course, the quality of the product— what's the content you generate and push to users— Meta is a huge personalization machine, right? So the content is highly relevant. That's all important. So low latency, low TCO, high-quality content— that's a lesson learned, carry on from Meta. And interestingly, I think I want to talk about GenAI. I think that's also— we're going to touch on this topic. We are currently under the wave, a tidal wave of GenAI, and this tidal wave is going to revolutionize the applications of products in the industry. And guess what? We are running into exactly the same set of problems: low latency, low TCO, high quality content. So, so that's kind of— we are very confident Fireworks understands those problems really well from years ago. And we are laser-focused on tackling this problem, providing the best solution to the industry.

It makes a ton of sense. And with GenAI, everybody's talking about training costs, right? Like you're spending $100 million to train these gigantic models. Fewer people are talking about inference costs, which are the scarier costs because they can sneak up on you. When you get to scale, they probably make training costs look— I wouldn't say de minimis, but, you know, a smaller percentage of the overall cost structure. I was talking to a friend recently that was building an AI application company, and he was bootstrapping it, and he wasn't charging yet. He just launched like a beta version of the product, and somehow it hit a viral loop and just took off. And he, you know, he, he like, at the end of the— literally like within a 24-hour period, he turned around and had a bill for like I don't know what it was, like $20,000, $25,000. And, uh, which, I mean, great, the product's viral, but it's, uh, you know, it certainly sneaks up on you. So I'm curious, as we look at like these gigantic models versus smaller models, you know, the trend because of scaling law has been like the bigger the model, the more data it's trained on, the more compute used to train it, the better the model performs generally across a broad range of tasks. Curious, if you look at these really big models, has inference costs scaled relatively linearly to the size of the model, or is that relationship less correlated?

Yeah, so it's roughly correlated, but there are a lot of nuances towards it. The inference cost is also relevant to the context window size, which is a very important concept of large language models. Large language models have the capability of remembering things. That's called context window. And the longer— right now, the mainstream LLMs, they are using attention mechanism, and the attention mechanism basically remembers the pairwise relationship of the context, which is quadratic. So that means, hey, if you want to drive better quality, you want to, hey, remember longer context so the content you generate is more relevant, you're going to pay for it, and the cost you're going to pay for is quadratic latency penalty, right? So, so that is that. But also interestingly, we find out large language model, when it comes to— this is kind of typical latency and, and quality trade-off. We will constantly talk about that. We find out a lot of applications they cannot just send this specific request to a large language model and expect it generate great, great result for multiple reasons. One reason is typically we run into a lot of customers, they have very strict latency requirements. If they don't get the result back in half a second or one second, product experience is out of the window, and they, they will lose the viral effects, and it's not a viable product. So in order to get that latency, they have to use a smaller model, and the smaller model of course is less capable than large model, right? So then you need to teach a smaller model, hey, here's the instruction to follow. So hey, that is in the system prompt, increase the context window size. And you need to give the relevant information to help the model answer the question within some bounds, right? And these bounds make the application more interesting, more relevant. So then we often see the prompt length is long. The prompt length is long not because the question length is long. It's because you put instruction there, you put the the context there, and then the result will be much, much better. So then remember, attention is quadratic, and then it has kind of latency penalty. And we have hand-drawn a proprietary kernel called the FireAttention to address that specifically. So you don't have to pick and choose, uh, you know, hey, here's a trade-off, pick one. You pick two. You pick quality, latency, you always pick two. That's where kind of, uh, we really want to alleviate that concern for our customers.

Makes, makes total sense. I would imagine it also, like, the size of the model you want to use also depends on the, the use case that you're trying to solve, right? Like, if you're trying to have an open-ended chatbot that could talk about any subject, like, you probably, I would assume, need a larger model. If you're trying to do something within very narrow bounds, perhaps you can get away with a smaller model. Is that true as well?

That's absolutely true. And in reality, we run into many of pretty much like this kind of classification of problems. And I think the industry will start from, hey, let's try the open-ended because I don't know. Know which question is more interesting. This becomes a product design, right? Uh, the product initially designed for general, um, generalized, um, Q&A, uh, and then the more people use it, they find, oh look, there's a pattern, uh, this particular problem people like to kind of, uh, this kind of— they start to see a skewed distribution of, of questions, uh, and then pick the heavy hitters, start to customize for it. Then we see the pattern people migrate from using a large, larger model which has worse latency and much worse TCO into specialized smaller model, customize it with the data those enterprise developers have, make it much better quality, and guess what, much lower latency and lower TCO. So, and then still maintain a long tail pool of open-ended support product experience, but the volume is much lower. So over time, we'll see kind of those companies adopting GenAI start to, from early stage, they don't know what it was, the product market fit, evolving to clear product market fit, going towards customization, specialization with smaller model. So that's kind of the, the trend we're seeing.

That makes total sense. Do you think that same sort of trend will hold as we move to multimodal models and the use cases expand beyond text and go to audio, video, images?

I definitely think so. I will say multimodal models are still at earlier stage. Last year is kind of LLM, but this year, that's where we believe multimodal is important because As we in real life, right? So a lot of time we just don't interact with each other using text, right? So yeah, in the past it's always text as the input, text-to-text, text-to-image, text-to-video, text-to-audio, right? But as we are doing this interview, I'm looking at the screen, it's frame of— it consists of frame of images, and then I can extract information from there. And images actually carry much nuanced denser information. So it's actually much richer for content generation, and I think that's the future. But when it comes to multimodal, basically it's mixed text and image to answer questions. I also see the trend of people start from open-ended, right? But more and more we start to onboard customers, they have very specialized request. They want to, for example, people want to extract structured information from receipts, from medical bills, from insurance policies, and so on. So those are very— becomes kind of very specialized, and more and more people ask to fine-tune multimodal models. So I think the same trend will happen to multimodality.

And if we look at kind of the, the state of open source versus closed source, as you see companies on this arc of starting with big, probably commercial models and then moving towards smaller models, like, do they typically start with like GPT-4 or something off the shelf and then move to maybe an open source model as they optimize for, for TCO? Or, or is there a different kind of arc you're seeing on the commercial versus open source side?

Yeah, I do see one trend, one pattern is when companies like startups or hobbyists, they don't know, they just want to have some fun and play with something or validate the biggest hypothesis is, is LLM even the solution to my problem, right? And the best starting point is to start with the most powerful LLM and see if it can solve the problem or not. Because of that, many people start from OpenAI, and the inflection point is when they hit product market fit, right? When that product starts making sense, and then all the product operation comes into the picture. Hey, as I said, most of these products are going to be consumer, prosumer, developer-facing. So where is my interactiveness? right? If, if I generate the output for Coding Copilot, if I generate, uh, the coding suggestion or code review, uh, feedback 5 seconds later, uh, it's— which is not very long, but you already lose.

It's useless if you're trying to code, right?

Yeah, exactly. Um, so, um, there are people building medical assistant, there are people building educational assistant, uh, legal assistant, all these assistant Customer service, it has to be instant, right? The response has to be instant, it has to be relevant. So latency is always a problem. And again, cost is always a problem, as we just mentioned. Many startups start with free tier offering, right? They quickly see, oh my God, I'm burning money like this crazy. It's kind of, how do I even sustain this before I can charge people? Once you charge people, it may be subscription-based, and then they have upper bound. So even at incumbents, so we also support very large e-commerce companies. At incumbents, they already have a huge base of consumers to deploy this product based on GenAI 2. That's even worse for them because they instantly get onto this large scale and they look at, oh my God, we need to use these many GPUs. Tens of millions of dollars. Where is my ROI of a product? Can I justify it? Everyone starts to kind of panic once they really think about the path to production. And that's why kind of lowering TCO is not, oh, this is just yet another thing. It is actually business critical.

Yeah, it's absolutely necessary. It's funny, we talk about like the innovator's dilemma that Alphabet is facing. Because of this problem, right? The inference cost on an LLM that would deliver high-quality search results is, from numbers I've seen, up to 30 times higher than the inference cost for their core search capabilities today. And so it's like this really difficult question of, okay, you probably have to become, you know, a knowledge engine if you want to compete with Perplexity and everybody else and, you know, offer a modern solution or a modern product to customers. But the inference costs of that are so much higher. Like, how do you, how do you make it work? And I think the, the actual underlying business model is also at risk because the need for ads is, is less relevant for a Perplexity-like experience than for Google's current experience. And so you have this challenging problem where your inference costs are going to go up dramatically and like the entire underlying business model for how you monetize is, is being challenged. And they're not the only ones. Like, there are, to your point, a lot of companies that are facing the same dilemma. And so it seems like, you know, if I were, if I were executive at one of these companies, like TCO would be really one of my probably top 3 priorities and trying to wrangle.

Yeah, I think the challenge here is the way to think about the problem is the open-air approach is we're going to build a model one size fits all. But the challenge of one size fits all is you cannot customize. You cannot customize, and you're going to use— no matter what is your problem, you're going to use that model. But if you— if we look at our everyday life, uh, it's never the case, right? Because the, the unit of economy is going to push for diversity of the tools you use. For example, just, okay, every single family has multiple, um, ways to commute, right? We have bikes, we have scooters, we have all different sizes of cars, because if I'm just going to school, I just, I can just ride a bike, right? That's kind of really solved my problem, and I don't need a truck to go to school all the time, or I don't need a Ferrari, you know. So, and if we look at the kitchen, right, we have so many cookware, one for each type of cooking, and So that's kind of, I think the unit economy is going to really push, really break this one-size-fits-all model. And that's where our philosophy is opposite of OpenAI. Like, we don't believe, hey, one model is going to magically solve all the problems with the best quality, with the best latency, with the best TCO. It just, that doesn't work. That world doesn't exist. Instead, we're going the opposite direction. We believe in specialized customized model for each individual business task, and we'll offer hundreds of those models under one umbrella where we're going to put a really great router that sits on top of hundreds of them. And eventually to our user, they don't need to understand, oh, which model? Now I have a problem. Which model should I pick to solve my problem? I have a discovery problem, and we want to also solve the discovery problem by simplifying that. So that's the direction where we're going towards with smaller models. With smaller models, actually, we can deliver a much higher precision of solving specific business tasks.

Yeah, it's almost, not to use Alphabet as another example, but it's almost analogous to the early days of the internet where most of the browsers tried to be everything. You had one page that had your news, your email, your shopping, everything on it. And then Google had a totally different approach, right? You have one search box and you can go find the best place to discover news or to shop or to do whatever. And so it's sort of like that in a way where you probably don't need one model for everything. You're probably better off finding smaller models that are great at specific tasks. And I'm curious, as you look at those smaller models, Are most of those open source or are there smaller commercial models as well?

I think most of the small models are actually open source models. The reason is people feel, one is it's easier to train smaller models. It's less costly. So the open source community actually can pool resources to do it. And the quality is getting better and better. Observation, my prediction is at the small model size, uh, 7 billion, 13 billion, it's going to converge. So the small model quality for solving, uh, general purpose problems is going to converge because we are running out of data and no more data is going to help the model because the model with, with that size is kind of have limited capacity of absorbing knowledge. So it doesn't matter whether it's open source or Closed source, the overall quality, the overall logical reasoning, kind of the foundational quality is going to converge. And then open source has much more advantage of customize, right? Because we have a huge community and they can start to tune, customize those open source models to solve specific problems so much better. And I'm betting on the velocity of the community to move the industry forward. And that squarely fits into our vision. We believe eventually it will be hundreds of those business-specific task models. And then we can kind of put a wrapper on top of to help the discovery issue.

Are you seeing companies train their own proprietary models? Or in most cases, are they taking an open-source model and fine-tuning it for a specific use case?

Majority company fine-tune, um, and because that's the industrial shift in the AI, um, is the foundation model, right? So before foundation model, um, all, all companies have to train from scratch, um, because there's no, um, open source is just about model architecture, uh, but there's no base model which has the universal knowledge that you build on top of. So every single company needs to source data and train and do hyperparameters, sweeping all this kind of stuff. Foundation model basically is a new concept that this model is going to be trained with world data, whatever public information you can crawl from internet, and feed into it so it has the base knowledge. And then the disruptive part is it's super easy to customize. You don't need to crawl again from scratch. The problem you're going to solve is alignment, right? One way to think about is all these models are not designed to solve all problems. The models are being designed to have objective function where it's chosen the model the model builder has decided at the beginning of training, when they prepare training data, they are opinionated about what they care about, you know, what they don't care about. So, so in that sense, it's guaranteed that the foundation model is not perfectly aligned with a particular narrow problem solving. So, so then we need to add additional data to align the model better towards the problem you want to solve with a better quality. And that process is very lightweight. So Meta published a paper that with 1,000 samples, they can, they can fine-tune, I think, Llama 2, a 70 billion model, to be on par with GPT-4. For specific task. So that's actually very encouraging, right? So that's significantly lower the bar of barrier, entrance barrier to get your customized model, to build your own mode, and so on. So, but we have also seen few company, large ones, for legal reason they cannot use some of the open source models. For example, for Meta, there's this kind of threshold, you know, if you pass the threshold, you cannot use that, and then they have to train from scratch. Or for legal reason, they just want to know exactly where's the source of data to explain to their customer, and then it's better for them to train. So they're the kind of various different reasons for training from scratch.

It seems like one of the reasons you don't want to train from scratch is because it's hard to do. And if you're like, I mean, if you're an early stage startup or I don't know, like a John Deere, right? Like a legacy enterprise company, like it's pretty hard to recruit the people and pretty expensive to recruit the people that know how to train state-of-the-art models. And so it seems like the talent constraint is also pushing people towards, like everything else aside, it's one of the factors that's pushing people towards fine-tuning open-source models versus training their own proprietary models. Yeah.

And the other thing is what's the ROI, right? So it's not just people investment. Data procurement is always, it's not free. Of course you can crawl from internet, but if you crawl, then what's the difference between your data. And a lot of companies need to actually buy data or manually generate data, which is also cost. So then the question is, what is ROI for you to do that compared with building on top of existing foundation models?

This is like a very elementary question, but what is— unlike traditional open source software where you can have like, you know, 1,000 decentralized contributors working on a project. Training a model is expensive, and it seems like the effort needs to be more coordinated. So who is actually training these open source models, and what is their incentive to do so?

I would say most of those good open source model providers, they kind of are startups. And recently I'm seeing like more— of course, Meta is big on that for strategic reasons. Google started to do also open source. Some of the smaller size of their models. But by and large, most of those companies' open source models are startup. And I, I think that's kind of a way to counter OpenAI, is they are completely closed sourced, and by open sourcing, they can build a much bigger community. And I think that strategy is working, right? Because there is a demand that community will like to get hold of model and have freedom of using the model in whatever way they want to use. And by that, it's actually advanced the model quality. You know, they fine-tune, they open source again, so there's a compounding effect that's happening. So that means for those companies who open source models, they have ecosystem built on top of it. They become much more stickier, but the community also benefits from that. The bigger industry benefits from that. So win-win for everyone. I've seen a few companies play that route, and it's doing well for them. Then the question to them is how they monetize. They cannot open-source everything because then it's kind of giving away, and it's very expensive training cost. How they kind of monetize out of that is then they start to offer better proprietary model and hopefully their users are loyal to them and pay them for proprietary models. So I have seen that business model start to grow.

Yeah, and for big companies like Meta, I think there are other strategic reasons why it makes sense. If we shift to inference cost a little bit, on the training side, we've seen training costs declining at more than 70% a year. More than twice the rate of Moore's Law. And that really comes down to algorithmic efficiency and architectural improvements, but also hardware. GPUs have just gotten better and better, and the relative cost of training has declined. On the inference side, are cost declines really just driven by hardware, or is there something you can do at the software level to reduce inference cost?

I think it's actually across the whole entire stack. It's not just hardware. I think, of course, hardware is becoming better, Hardware is— I just want to go through the layers. Hardware has become denser in terms of price performance. The flops that can be packed within the same power envelope is going to be higher and higher, and the memory bandwidth has become higher and higher. But I think it's not moving faster, fast enough to just significantly drop the price yet. Nevertheless, I think there's always hardware advancement to kind of get us to a better, better place. But at the same time, at the software layer, that's where kind of there's so many things we can optimize for. One layer is purely at the model level. So for specific model architecture, there are— the model has layers, model has operators, and those modules, the heavy-hitter modules, we can customize that extremely towards the hardware capability. So there's still a long way to go. I don't think we have fully maximized that yet. And then there's also orchestration scheduling efficiency. So think about An analogy is in the database world, a database engine that is— you basically do a query towards a database and get a result. But depends on your workload pattern, the layout of data can be different, so you get better latency and so on. So it's called optimizer for databases. For a long time, there's no such optimizer. Optimizer is DBA, a human being, to observe your workload and manually tune it. And it's actually a well-paid job. And a large database company actually offers certificate for people to get qualified to run this job. It's literally create a career for many people. And over time, those big database companies start to build smarter and smarter optimizer in the those jobs are less relevant. I think that is also happening in the large language model inference serving tier. There is model-specific optimization, but there's also application-specific optimization. And that's where what we do is we optimize across the whole entire stack. We have helped many customers onboard and kind of help them tune knobs here and there, to the point we feel it's not sustainable. We, we gotta automate this. So we, we are also building the kind of database equivalent optimizer, equivalent optimizer for large language model applications. So I'm very excited about this product line we're building, and we're gonna announce soon, in about a month. I think many people will benefit from that.

That's great. It makes a ton of sense. If you look at the hardware side, you know, GPUs have been a bottleneck, it seems, for training. Has that supply constraint also impacted companies' ability to do inference at scale?

I think there's kind of interesting dynamics. On one hand, because we are going through a tidal wave of GenAI technology-powered application, new application development. So there is suddenly a rush to get in. I think almost feel like a traffic jam. But I think over time this will be, and we are in the process of rationalize the whole thing. The industry is rationalize the whole thing. From my point of view, we partner very closely with AWS, GCP, OCI. They, I think they did a good job. They did a really good job onboarding the GPUs and allocating them in a most efficient and prioritized way. So we never have any problem getting GPUs supporting our customer. And I think that's the point of using an aggregator is because individual customer, if they need to kind of acquire GPU, they need to go through the whole procurement cycle negotiation and even think about what is the capacity I need. It's a science. It requires a lot of data. But if I, you know, to make their job easier, they can actually offload that completely to us and have we managed that. And I think there are also a lot more players that's coming to the picture to alleviate hardware constraint. I think NVIDIA is definitely leading the pack here, but there's also AMD coming up. There's— there are different kind of hardware accelerators that specialize in solving certain problems really well. So I, I am pretty optimistic about, um, the forward-looking of the industry getting more diverse hardware supplies, and eventually we will we will not run into this problem anymore.

Yeah, you have a lot of startups like Groq that are focused on— to be focused more on inference specifically and trying to build hardware that will drive inference costs down. So it seems like this is a short-term bottleneck that will not persist for the next 3, 5, 7 years, which is a good thing for the industry writ large. If we shift to the application layer for a second, You have an interesting view because you see a lot of different customers building a lot of different types of AI applications. Could you share a couple of cool applications that you've seen where you're like, wow, that's a really cool use case for AI?

Yeah, I'm constantly amazed by the creativity of the founders building new applications on top of us. One recent example is we, we work very close with AnySphere. AnySphere, their product name is called Cursor. The idea is, um, they're in kind of a coding copilot space, but it's not just about generating code. That's traditional way of coding copilot. There's also a lot of challenging challenges reviewing code in with high quality in the timely fashion. So that's why they're a product called Cursorizing. The idea is you put your cursor in the code review doc and they would create suggestions. Very interactive. And the way they build their product is highly innovative and they push us a lot. They push us to give them extremely, extremely low latency to make the product look really, really awesome. But We thoroughly enjoy that partnership because we deliver what they ask for and then they are delivering this awesome product. There are some other products, for example, PowerPoint generation. PowerPoint always takes me a long time to come, hey, give a keynote, give a speech, and to describe things accurately and in an interesting way. Punchy way. And Tome is one application that generates PowerPoints that you just say what you want to generate and it boom generates one and it kind of significantly shortens the time. And there are assistants built on top of us, like assisting lawyers to do case study. So, you know, it's just lawyers are going to be so much more efficient, where AI is going to retrieve the relevant cases that existed before, re-rank them, and drive the, you know, give you a summary of the top 10, and then you can make judgment call, you know, how similar your case is towards those cases. Super efficient. And yeah, so there's a whole slew of applications built on top of, and I think every single one blew my mind.

That makes sense. Yeah. I'm equally excited about all the different applications we're seeing, and especially in industries that are a little bit more stale. I think there are a lot of industries that still haven't been really touched by technology in the last 20, 30 years, which is surprising. Where the incumbents don't know what the cloud is, right? But AI is, AI is this new disruptive catalyst where many of these industries are facing labor shortages or other constraints. And if you can go in and do something 10 times better, cheaper, and faster with AI, like, I think the, the adoption rates are going to be quite high. If we think about moats though for these application companies, if you're not training your own model but you're fine-tuning on top of an open-source model, um, Is there a data advantage to be had just from fine-tuning alone and usage of your model that generates more data?

My answer would be, so it's a chicken-egg. The mode is actually your traffic. Okay. I'll explain what that means. People always say your data is your mode. That's true. But where do you get data? right? It's never a one-time problem. It's not like you source data somehow magically from somewhere. Maybe you use a data labeler, you get the initial label data, and then boom, you tune it perfect for your application. That's it, your job is done, and then you focus on product. It's never the case because I think there's a lot to, you know, your product will never stay still, right? You will evolve your product, your, your traffic, workload pattern is going to shift, shift and change. So the data part of data mode is constant. So what we believe is your traffic is your data. It's going to create a virtuous cycle. Initially, to bootstrap your traffic, you can use your traffic to make your model lower latency, much better quality. It attracts more traffic, make your model better, more traffic. So it creates this virtuous cycle for your, your product to go viral. And that's where we can help. And, uh, um, stay tuned, we're gonna have a product announcement soon.

Makes sense. Yeah, in some ways it's like the— its own version of like economies of scale, right? Where the more scale you get, lower latency, lower cost, better, better experience for customers. Um, it makes a lot of sense. Yeah, I think the, the initial sort of training or fine-tuning run with, with a fixed amount of data is, is like probably over-indexed on by people. Like, like continued usage of your product will, will you'll discover new edge cases that you couldn't have thought of when you first started, um, and continuously improve the model. And like, the analogy I like to use is Tesla and blue stop signs. There are a couple of resorts in Hawaii for— that for whatever reason have blue stop signs. And you know, if you— if a Tesla FSD rolls up, it probably doesn't know that a blue stop sign is a stop sign, right? Like, it probably somehow relates the color to to the meaning of a stop sign. And so if it rolls through the blue stop sign and you hit your brake, that's an intervention. If that happens 2 or 3 times across different cars in the fleet, then, you know, the next, the next fine-tuning run, like, the model will learn that a blue stop sign is also a stop sign. But you wouldn't encounter that unless you had probably hundreds of thousands or millions of cars on the road, uh, where you're likely to encounter these edge cases. And it's obviously most true in full self-driving, but it's true in a lot of other cases as well. So if you're thinking about the legal use case, if you produce an output reviewing a contract for some esoteric clause and a human lawyer looks at it and says, "Wait, no, that's not the right answer," and corrects it, that's an intervention, right? That data can be really valuable as you get to scale. So totally agree that scale certainly has some advantages. If you look back on your experience at Meta, and the first short period of time building this company, and you could go back to day one after everything you've learned, what advice would you give yourself?

I think one consistent philosophy I have, it kind of over time I developed that it works, is to be customer obsessed. And I will say, The way to perfect the product is to expose to customers as soon as possible, as fast as possible, and get iterating velocity there. And don't wait for perfect. Don't kind of be the perfectionist and bake things out of vacuum and make tons of assumptions not validated. So I think that's kind of, that's important. I think probably that's the most important thing throughout my journey.

That's great advice. And if I have an application startup and I want to work with Fireworks, where should I go?

Oh, fireworks.ai. And we have a self-serve tier, very easy to onboard. We also have initial credits. You can use. And we have a Discord channel. We are everywhere. You can ping us, super responsive. I'd love to help you get started.

That's awesome. Well, thank you so much, Lin Qiao.

Thank you.

Thanks for tuning in to the Autopilot Podcast. If you're using AI to create or disrupt a large industry, I would love to learn more about your story. Drop me a line on Twitter or go to our website, apv.vc. If you want to support the show, the best ways are to leave a review wherever you're listening and subscribe. The Autopilot Podcast is part of Turpentine, the podcast network behind Moment of Zen, Turpentine VC, Age of Miracles, and more shows for experts by experts in tech.