Why Compound AI + Open Source will beat Closed AI — with Lin Qiao, CEO of Fireworks AI

Hey everyone, welcome to the Late in Space podcast. This is Alessio, partner and CTO at Decibel Partners, and I'm joined by my co-host Swyx, founder of SmallAI.

Hey, and today we're in a very special studio inside the Fireworks office with Lin Qiao, CEO of Fireworks. Welcome. Oh, you should welcome us. Yeah, welcome.

Both.

Yeah, thanks for having us. It's unusual to be in the home of a startup, but like it's also, I think our relationship is a bit unusual with compared to all our normal guests.

Definitely. Yeah. It's, I'm super excited to, to talk about very interesting topics in that space with both of you.

You just celebrated your 2-year anniversary yesterday. Yeah.

It's quite a crazy journey. We circle around and share all the crazy stories across these 2 years. And, uh, it has been super fun. Yeah. Uh, all the way from we experienced Silicon Valley bank run, to we delete some data that shouldn't be deleted operationally. We went through massive scale where we actually are busy getting capacity to, yeah, we, we learned to kind of work with it as a team with a lot of brilliant people across different places. Join a company, it has really been a fun journey.

When you started, did you think the technical stuff will be harder or the bank run and then the people side? I think there's a lot of amazing researchers that want to do companies and it's like the hardest thing is going to be building the product and then you have all these different founder things. So were you surprised by what has been your experience?

Yeah, to be honest with you, my focus has always been on the product side. And then after product go to market, and I didn't realize the rest has been so complicated, operating a company and so on, but because I don't think about it, I just kind of manage it, so it's done. So I think I just somehow like don't think about it too much and solve whatever problem coming our way and it worked.

So let's, I guess, let's start at the prehistory, like the pre, the sort of initial history of Fireworks. You ran the PyTorch team at Meta for a number of years. And we previously had Sumit Chintala on. And I think we were just all very interested in the history of GenAI. Maybe not that many people know how deeply involved FAIR and Meta were prior to the current GenAI revolution.

My background is a DP in distributed system, database management system. And I joined— Meta from the data side, and I saw this tremendous amount of data growth, which costs a lot of money, and we're analyzing what's going on, and it's clear that AI is driving all this data generation. So it's a very interesting time because when I joined Meta, Meta is going through ramping down mobile-first, finishing the mobile-first transition, and then starting AI-first. And there's a fundamental reason about that sequence because mobile-first gave a full range of user engagement that was, has never existed before. And all this user engagement generate a lot of data and this data power AI. So then the whole entire industry is also going through, following through, uh, the same transition. When I see, oh, okay, this AI powering all this data generation and, and look at, uh, where's our AI stack? There's no software, there's no hardware, there's no people, there's no team. I like, hey, I want to dive it up there and, and help this this movement. So, when I started, it's very interesting industry landscape. There are a lot of AI frameworks. It's a kind of proliferation of AI frameworks, uh, happening at the industry. But all the AI frameworks focus on like production, um, and they use a very certain way of defining the graph, uh, of neural network and, and use that to drive, uh, the model iteration and, uh, productionization. And PyTorch is completely different. So Vic Kuroso sumiu, then he was the user of his product, right? He basically says, researcher face so much pain using existing AI framework. This is really hard to use, and I'm going to build something different for myself. And that's the origin story of PyTorch. PyTorch actually started as the framework for researchers, don't care about production at all. And as it grow in terms of adoption, Uh, so the interesting part of AI is research is the top of funnel for production. There are so many researchers across academic, across industry, and they innovate and they put their results out there in open source, and that power the downstream production innovation. So it's brilliant for Meta to establish PyTorch as a strategy to drive massive adoption in open source because Meta internally is a PyTorch shop. So there's, it create a flying wheel effect. So that's kind of a strategy behind, uh, PyTorch. But when I, uh, took on PyTorch, it's kind of at the cusp of Meta established PyTorch as the, the framework for both research and production. So no one has done that before. And we have to kind of rethink how to architect PyTorch so it can really sustain production workload. The stability, reliability, low latency, all these production concern was never a concern before. Now it's a concern, and we actually have to adjust its design and make it work for both sides. And that took us 5 years, uh, because Meta has so many AI use cases all the way from ranking recommendation that's powering the business top line, or ads ranking, newsfeed, video ranking, to site integrity, detect bad content automatically using AI, to all kind of effects, translation, image classification, object detection, all this. And also across, um, AI running on the server side, on mobile phones, on AR/VR devices. A wide spectrum. So by the time we actually basically managed to support AI across ubiquitous everywhere across Meta, but interestingly, through open source engagement, we work with a lot of companies. It is clear to us like this industry is start to take on AI-first transition. And of course, Meta's hyperscaler always go ahead of industry. And we feel like— it feels like when we start this AI journey at Meta. There's no software, no hardware, no team. For many companies we engage with through PyTorch, we feel the pain. That's the genesis why we feel like, hey, if we create Fireworks and support the industry going through this transition, it will be a huge amount of impact. Of course, the problem that the industry is facing will not be the same as Meta. Meta is so big, right? So it's kind of skewed towards extreme scale and extreme optimization. The industry will look different. But we feel like we have the technical chops and we have seen a lot. We'd love to drive that. So yeah, so that's how we started.

When you and I chatted about the origins of Fireworks, it was originally envisioned more as a PyTorch platform and then later became much more focused on generative AI. Is that fair to say? Right. What was the customer discovery here?

Right. So I would say our initial blueprint is, hey, we should build a PyTorch cloud because PyTorch is a library and there's no SaaS platform to enable AI workloads.

Even in 2022?

I would not say absolutely no, but like cloud providers have some of those, but it's not first-class citizen, right? Because at 2022, there's still like TensorFlow is massively in production. And this is all pre-GenAI. And PyTorch is kind of getting more and more adoption. But there's no PyTorch-first SaaS platform existing. At the same time, we are also a very pragmatic set of people. We really want to make sure from the get-go, we get really, really close to customers. We understand their use case. We understand their pain points. We understand the value we deliver to them. So we want to take a different approach. Instead of building a horizontal PyTorch cloud, we want to build a verticalized platform first. And then we talk with many customer. And interesting, we start a company September 2022, and October, November, then OpenAI announced ChatGPT. And then boom. Then when we talk with many customer, they are like, can you help us working on the GenAI aspect? So of course, there are some open source models. It's not as good at that time, but people already putting a lot of attention there. Then we decide, uh, if we're gonna pick a vertical, we're gonna pick GenAI. The other reason is all GenAI models are PyTorch models. So that's another reason. We believe that because of nature of GenAI, it's gonna generate a lot of human consumable content. It will drive a lot of consumer, customer, developer-facing application and product innovation, guaranteed, right? We're just at the beginning of this. Our prediction is for those kind of application, inference is much more important than training because inference scale is proportional to the upper limit of world population. And training, training scale is proportional to the number of researchers. Of course, each training round could be very expensive. Although PyTorch supports inference and training, we decided to laser focus on inference. So yeah, so that's how we got started. And we launched our public platform August last year. And when we launch, it's a single product. It's a distributed inference engine with simple API, OpenAI-compatible API with many models. We started with LLM, and later on, we add a lot of model. Fast forward to now, we are full platform with multiple product lines. So we love to dive deep into what we offer. But that's a very fun journey in the past 2 years.

What was the transition from you start focus on PyTorch and people want to understand the framework, get it live. And now I would say maybe most people that use you don't even really know much about PyTorch at all. They're just trying to consume a model. From a product perspective, what were some of the decisions early on? Right in October, November, you were just like, "Hey, most people just care about the model, not about the framework. We're going to make it super easy." Or was it more a gradual transition to the model library you have today?

Yeah, so our product decision all based on who is our ICP. And one thing we want to acknowledge here is The GenAI technology is disruptive. It's very different from AI before GenAI. So it's a clear leap forward because before GenAI, the companies that want to invest in AI, they have to train from scratch. There's no other way. There's no foundation model. Doesn't exist. So that means they need to start a team, first hire a team who is capable of crunch data. There's a lot of data to crunch, right? Because training from scratch, you, you have to prepare a lot of data. And then they need to have, they need to have GPUs to train, and then you need to start to manage GPUs. So then it becomes a very complex project. It takes a long time and not many company can afford it actually. And the GenAI is a very different game right now because it is a foundation model. So you don't have to train anymore. That make AI much more accessible as a technology. As an app developer or product manager, even not a developer, they can interact with generative models directly. So, and our goal is to make AI accessible to all app developers and product engineers. That's our goal. So then getting them into the building model doesn't make any sense anymore with this new technology. And then building easy accessible APIs is the most important. Our, uh, early on when we get started, we decided we're gonna be OpenAI compatible. It's just kind of very easy for developers to adopt this new technology and we will manage the underlying complexity of serving all these models.

Yeah, OpenAI has become the standard. Even as we're recording today, Gemini announced that they have OpenAI compatible APIs.

Interesting.

So then we just need Anthropic to fall in line and then we have everyone.

Yeah, that's interesting because we are working very closely with Meta as one of the partners. Meta announced— Meta, of course, is very generous to donate many very, very strong open-source models, expecting more to come. But also, they have announced Llama Stack, which is basically standardized the upper-level stack built on top of Llama models. They don't just want to give out models and you figure out what the upper stack— they instead want to build a community around the stack and build a new standard. I think there's this interesting dynamics in play in the industry right now. One is more like standardized across OpenAI because they are kind of creating the top of funnel or standardized across Llama because this is the like mostly used open source model. So I think really a lot of fun working at this time.

I've been a little bit more doubtful on Llama Stack. I think you've been more positive. Basically, it's just like the meta version of whatever Hugging Face offers, or TensorRT, or BLM, or whatever the open source opportunity is. But to me, it's not clear that just because Meta open sources Llama, that the rest of Llama stack will be adopted. And it's not clear why I should adopt it. So I don't know if you have a—

Yeah. It's very early right now. That's why we'll work very closely with them and give them feedback. The feedback to the Meta team is very important. So then they can— use that to continue to improve the model and also improve the higher-level stack. I think the success of Llama stack heavily depends on the community adoption, and there's no way around it. And I know Meta team would like to kind of work with the broader satellite community, but it's very early.

One thing that after your Series B, so you raised from Benchmark and Sequoia. I remember being close to you for at least your Series B announcement. You started betting heavily on this term of compound AI. It's not a term that we've covered very much in the podcast, but I think it's definitely getting a lot of adoption from like Databricks and the Berkeley people and all that. What's your take on Compound AI? Why is it resonating with people?

Right. So let me give you a little bit of context why we even consider that space. Yeah.

Because like pre-Series B, you were not, there was no message. Yeah. Now it's like on your landing page.

So it's kind of very organic evolution from when we first launched our public platform. We are single product, we are distributed inference engine where we do a lot of innovation on customized CUDA kernels, ROCm kernels running on different kind of hardware, and build distributed disaggregated execution, inference execution, build all kind of caching. And so that is one. So that's kind of one product line is the fast, most cost-efficient inference platform. Because we wrote PyTorch code, we know we basically have a special PyTorch build for that. Together with a custom kernel we wrote. And then we work with many more customer, we realize, oh, the distributed inference engine is our design is one size fits all, right? We want to have this inference endpoint, then everyone come in and they, no matter what kind of form and shape or workload they have, it will just work for them, right? So that's great. But the reality is they, we realize all customer have different kind of use cases. The use case come in all different form and shape, and the end result is the data distribution in their inference workload doesn't align with the data distribution in the training data for the model, right? It's a given actually if you think about this because researchers has to guesstimate what is important, what's not important during like you prepare data for training. So because of that misalignment, then we leave a lot of quality latency cost improvement on the table. So then we're saying, okay, we want to heavily invest in a customization engine, and we actually announced it called Fire Optimizer. So Fire Optimizer basically helps users navigate a three-dimensional optimization space across quality, latency, and cost. So it's a three-dimensional curve, and even for one company, for different use cases, they want to land in different spots. So we automate that process for our customer. It's very simple. You have your inference workload, and you inject into the optimizer along with our objective function. And then we spit out inference deployment config and the model setup. So like it's your customized setup. So that is a completely different product. So that product thinking is one size fits one, different from one size fits all. And now on top of that, we provide a huge variety of state-of-the-art models, hundreds of them. Varying from text, the large state-of-the-art large language models. That's where we started. And then as we talk with many customer, we realize, oh, audio and text are very, very close. Many of our customers start to build assistant, all kind of assistant using text, and they immediately want to add audio, audio in, audio out. So we support transcription, translation, speech synthesis, text audio alignment. All different kind of audio features. It's a big announcement. We're gonna— you should have heard by the time this is out.

Yeah.

And the other areas of vision and the text are very close with each other because a lot of information doesn't live in plain text. A lot of information live in multimedia format, live in images, PDFs, screenshots, and many other different formats. So oftentimes solve a problem, uh, we need to put the vision model first to extract information. and then use language model to process and then send out results. So vision is important. We also support vision model, various different kind of vision model specialized in processing different kind of source, uh, uh, and extraction. And we're also gonna have another announcement of a new API endpoint we'll support for people to upload various different kind of multi, multimedia content and then get the, uh, extract very accurate information out and feed that to you. Into LM. And, and then of course we support embedding because embedding is very important for semantic search, for RAG, and all this. And in addition to that, we also support text-to-image, image generation models, text-to-image, image-to-image, and we're adding text-to-video as well in our portfolio. So it's very comprehensive set of model catalog that build on, run on top of File Optimizer and distributed inference engine. But then we talk with more customer, they saw business use case And then we realized one model is not sufficient to solve their problem. And it's, it's very clear because one is the model hallucinates. And many customers, they, when they onboard this GenAI journey, they thought this is magical. GenAI is gonna solve all my problems magically. But then they realize, oh, this model hallucinates. It hallucinates because it's not deterministic, it's probabilistic. So it's designed to always give an answer, but based on probability. So it hallucinates. And that's actually sometimes a feature for creative writing, for example. Sometimes it's a bug because, hey, you don't want to give misinformation. And different models also have different specialties. To solve a problem, you want to ask different special models to kind of decompose your, your task into multiple small tasks, narrow tasks, and have expert models solve that task really well. And of course, the model doesn't have all the information. It has limited knowledge because the training data is finite, not infinite. So model oftentimes doesn't have real-time information. It doesn't know any proprietary information within enterprise. It's clear, you know, had— in order to really build a compelling application on top of GenAI, we need a compound AI system. Compound AI system basically is gonna have multiple models across modalities along with APIs, whether it's public APIs, internal proprietary APIs, storage systems, database systems, knowledge systems to work together to deliver the best answer.

Are you going to offer a vector database?

We actually heavily partner with several big vector database providers. Which is your favorite? Um, they are all great in different ways, but it's public information, like MongoDB is our investor, and we have been working closely with them for a while.

When you say distributed inference engine, what do you mean exactly? Because when I hear your explanation, it's almost like you're centralizing a lot of the decisions through the Fireworks platform on the quality and whatnot. When you mean distributed, it's like you have GPUs in a lot of different clusters, or you're sharding the inference across—

Say more. Right, right, right. So first of all, we run across multiple GPUs, but the way we distribute across multiple GPUs is is unique. We don't distribute the whole model monolithically across multiple GPUs. We chop them into pieces and scale them completely differently based on what's the bottleneck. We also are distributed across regions. We have been running in North America, EMEA, and Asia. We have regional affinity to applications because latency is extremely important. We are also like doing global load balancing because a lot of applications there they quickly scale to global population. And then at that scale, like different content wake up, wakes up at different time, uh, and you want to kind of load balancing across. So all the way in a week, we also have, we manage various different kind of hardware SKU from different hardware vendors and different hardware design is best for different type of workload, whether it's long context, short content, long generation. So all these different type of workload is best fit it for different kind of hardware scale, and we can even distribute across different hardware for a workload. So yeah, so the distribution actually is, is all around in the full stack.

At some point we'll show on the YouTube the image that Ray, I think, has been working on with like all the different modalities that you offer. Like to me, it's basically you offer the open source version of everything that OpenAI typically offers, right? I don't think there is actually, if you do text-to-video, You will be a superset of what OpenAI offers because they don't have Sora. Is that Mochi, by the way? It's Mochi, right? Yes.

Mochi. And there are a few others. I will say the interesting thing is I think we're betting on the open source community is going to grow, proliferate. This is literally what I'm seeing. And there's amazing video generation companies. Amazing audio companies. Like cross-board, the innovation is off the chart. And we are building on top of that. I think that's the advantage we have compared with a closed-source company.

I think I want to restate the value proposition of Fireworks for people who are comparing you versus like a raw GPU provider like a RunPod or Lambda or anything like those, which is like you create the developer experience layer. And you also make it easily scalable or serverless as an endpoint. And then I think for some models, you have custom kernels, but not all models.

For almost for all models, for all large language models, all your models.

Just write kernels all day long?

And the VRS. Yeah, yeah. Almost for all models we serve, we have.

And so that is called FireRTension?

That's called FireRTension.

I don't remember the speed numbers, but apparently much better than VLLM, especially on concurrency basis.

Right. So Fire Attention is specific for mostly for language model, but for other modalities, we'll also have customized kernel.

Yeah, I think the typical challenge for people is understanding like that has value. And then like there are other people who are also offering open source models, right? Like your moat is your ability to offer like a good experience for all these customers. But if your existence is entirely reliant on people releasing nice open source models, other people can also do the same thing.

Right. Yeah. So I would say we build on top of open source model foundation. So that, that's the kind of foundation we build on top of. But we look at our, the value prop from the lens of application developers and product engineers. So they want to create new UX. So what's happening in the industry right now is people are thinking about completely new way of designing products. And I'm talking to so many founders. Is just mind-blowing. They help me understand existing way of doing PowerPoint, existing way of coding, existing way of managing customer service. It's actually putting a box in our head. For example, PowerPoint, right? So PowerPoint generation is we always need to think about how to fit into my storytelling into this format of slide one after another. And I, I'm gonna juggle through like design, Together with, you know, what story to tell, but the most important thing is what's your story? So telling lies, right? And why don't we create a space that is not limited to any format? And those kind of new product UX design combined with automated content generation through GenAI is the new thing that many founders are doing. What are the challenges they're facing? All right, let's go from there. One is, again, because a lot of products built on top of GenAI, they are consumer, personal, developer-facing, and they require interactive experience. It's just a kind of product experience we, we, we all get used to, and our desire is to actually get faster and faster interaction. Otherwise nobody want to spend time, right? So again, and then that requires low latency. And the other thing is the nature of consumer, personal, developer-facing is the, uh, your audience is very big. You, you want to scale up to product-market fit quickly. but if you lose money at a small scale, you're gonna bankrupt quickly. So it's actually a big contrast is I actually have product market fit, but when I scale, I scale out of my business, right? So that's kind of very, very funny, uh, way to think about it. So then have low latency and low cost is essential for those new application and product to survive and really become a generation company. So that's the design point for our distributed inference engine and the file optimizer. File optimizer, you can think about that as a feedback loop. The more you feed your inference workload to our inference engine, the more we help you improve quality, lower latency further, lower your cost. It basically becomes better. And, and we automate that because we don't want you as an app developer or product engineer to think about how to figure out all these low-level details. It's impossible because you are not trained to do that at all. You should kind of keep your focus on the product innovation. And then the compound AI, we actually feel a lot of pain as the app developers, engineers. There are so many models. Every week there's at least a new model coming out.

Tencent had a giant model this week.

Yeah, yeah, I saw that. I saw that.

It's like 500 billion parameters.

Yeah. Um, so they're like, should I keep chasing this or should I forget about it, right? So, and which model should I pick to solve what kind of subproblem? How do I even decompose my problem into those smaller problems and fit the model into it? I have no idea. And then there are two ways to think about this design, right? I think I talked about that in the past. One is imperative, as in, uh, you tell, you figure out how to do it, right? You give, developer tools to dictate how to do it, or you build a declarative system where developer tells what they want to do, not how. So these are completely two different designs, right? So like the analogy I want to draw is in the data world, the database management system is a declarative system because people use database, use SQL. SQL is a way you say, what do you want to extract out of database? What kind of result you want? but you don't figure out which node is gonna, how many nodes you're gonna run on top of, how you redefine your disk, which index you use, which project. You don't need to worry about any of those. And database management system will figure out, generate a new best plan, uh, and execute on that, right? So database is declarative and it makes it super easy. You just learn SQL, which is learn a semantic meaning of SQL and you can use it. Imperative side is there are a lot of ETL pipelines and people design this DAG system of with triggers, with actions, and, and you dictate exactly what to do and it fails and then how to recover. So that's a declarative system. And we have seen a range of system in the ecosystem like go different ways. I think there are value of both. There are value of both. I, I don't think one is gonna subsume the other, but we are leaning more into the philosophy of through the declarative system. System because from the lens of app developer and product engineer, that would be easiest for them to integrate.

I understand that's also why PyTorch won as well, right? This is one of the—

ease of use. So yeah, focus on ease of use and then let the system take on the hard challenges and complexities. So we follow, we extend that thinking into current system design. So another announcement is we will also announce a our next declarative system is going to appear as a model that has extremely high quality. And this model is inspired by O1 announcement from OpenAI. You should see that by the time we announce this or, you know, soon.

Trained by you?

Yes.

Is this the first model that you trained of like this scale?

It's not the first. We actually have trained a model called Fire Function. As a function calling model, it's our first step into compound AI system because function calling model can dispatch a request into multiple APIs. We have pre-baked set of APIs the model learned, and you can also add additional APIs to, through the configuration to let model dispatch accordingly. So we have a very high quality function calling model that already released. We have actually 3 versions. The latest version is very high quality. But now we take a further step that you don't even need to use function calling model. You use our new model we're going to release, it will solve a lot of problem approaching very high, like, OpenAI quality. So I'm very excited about that.

Do you have any benchmarks yet, or—

We have benchmark. We're going to release it hopefully next week. We just put our model to LM-SYS, and people are guessing Is this the next Gemini model or a MADAS model? People are guessing. That's very interesting. We're like watching the Reddit discussion right now.

I mean, I have to ask more questions about this. When OpenAI released O-1, a lot of people asked about whether or not it's a single model or whether it's like a chain of models. And Noam and basically everyone on the Strawberry team was very insistent that what they did for reinforcement learning chain of thought cannot be replicated by a whole bunch of open source model calls. Do you think that that is— they're wrong? Have you done the same amount of work on RL as they have, or was it a different direction?

I think they take a very specific approach where I do— the caliber of team is very high, right? So I do think they are the domain expert in doing the things they are doing, but I don't think there's only one way. To achieve the same goal. We are on the same direction in the sense that the quality scaling law is shifting from training to inference. We are definitely on this. For that, I fully agree with them. But we are taking a completely different approach to the problem. All of that is because, of course, we didn't train the model from scratch. All of that is because we build on the shoulder of giants, right? So the current model available we have access to is getting better and better. The future trend is the gap between the open source model, closed source model, it's just going to shrink to the point there's not much difference. And then we're on the same level field. That's why kind of, I think our early investment in inference and all the work we do around balancing across quality, latency, and cost pay off because we have accumulated a lot of experience there. And that empowers us to, to build and release this new model that is approaching OpenAI's quality.

I guess the question is what do you think the gap to catch up will be? Because I think everybody agrees that open source models eventually will catch up. And I think with 4, then with Llama 3.1, 4.0, 5B, we closed the gap and then O-1 just reopened the gap so much and it's unclear. Obviously you're saying your model will have—

We're closing that gap.

Yeah, but you think in the future it's going to be months?

So here's the thing that's happened. There's public benchmark. It is what it is. But in reality, open source model in certain direct— dimension already on par or beat closed source model, right? So for example, for in, in the coding space, open source models are really, really good. And, uh, in function calling, like, Fire Function is also really, really good. So it's all a matter of whether you build one model to solve all the problem and you want to be the best of solving all the problems, or In the open source domain, it's gonna specialize, right? All these different model builders specialize in certain narrow area, and it's logical that they can be really, really good in that very narrow area. And that's our prediction, is with specialization, there will be a lot of expert models, really, really good, and even better than, like, one-size-fits-all open source, uh, closed source model.

Yeah, I think this is the, the core debates that I am still not 100% either way on in terms of compound AI versus normal AI, because you're basically fighting the bitter lesson.

Look at the human society, right? We specialize. And you feel really good about someone specializing in doing something really well, right? And that's how our, like, when we evolved from ancient time, we're all generalists, we do everything, the tribe too. Now we having specialized in different domain. So my prediction is in the AI model space, it will happen also.

Except for the bitter lesson. You get short-term gains by having specialists, domain specialists. And then someone just needs to train a 10x bigger model on 10x more inference, 10x more data, 10x more model params, whatever the current scaling law is. And then it supersedes all the individual models because of some generalized intelligence world knowledge. You know, I think that, that is the core insight of the GPTs, the GPT-1, 2, 3.

That was right. But the scaling law again, right? The training scaling law is because you have increasing amount of data to train from and you can do a lot of compute, right? So I think on the data side, we're approaching the limit and the only data to increase that is entirely generated data. And then there's like, what is the secret sauce there, right? Because you have a, if you have a very good large model, you can generate very good synthetic data and then continue to improve quality. So that's why I think even OpenAI, they are shifting from the training scaling law into inference scaling law. And it's the test time and all this. So I definitely believe that's the future direction. And that's where we are really good at, and doing inference.

Couple questions on that. Are you planning to share your reasoning traces?

That's a very good question. We are still debating.

Yeah.

but it's still debating.

I would say if you, for example, it's interesting that like, for example, SWE-Bench, if you want to be considered for ranking, you have to submit your reasoning traces. And that has actually disqualified some of our past guests. Like CoSign was doing well on SWE-Bench, but they didn't want to leak those results. So that's why you don't see O1 preview on SWE-Bench because they don't submit their reasoning traces. And obviously it's IP, but also if you're going to be more open, then that's one way to be more open. So your model is not going to be open source, right? Like it's going to be an endpoint that you provide. Yes. That's right.

Okay, cool.

And then pricing, also the same as OpenAI, just kind of based on—

Yeah, this is— I don't have actually information. Everything is going so fast. We haven't even think about that yet. Yeah, I should be more prepared.

So I mean, this is live. It's nice to just talk about it. As it goes live. Any other things that you want feedback on or you're thinking through? It's kind of nice to just talk about something when it's not decided yet about this new model. I mean, it's going to be exciting. It's going to generate a lot of buzz.

Right. I'm very excited to see how people are going to use this model. So there's already Reddit discussion about it. And people are asking very deep medical questions. And as soon as the model get it right, surprising. And internally we're also asking model to generate what is AGI and it generates a very complicated DAG of thinking process. So it's, we're having a lot of fun testing this internally, but I'm more curious, like how would people use it? How, what kind of application they're gonna try and test on it? And that's where we'll really like to hear feedback from the community and also feedback to us, like what works out well, what doesn't work out well, what works out well by surprising them and what kind of thing they think we should improve on. And those kinds of feedback will be tremendously helpful.

Yeah. I mean, so I've been a production user of Preview and Mini since launch. I would say they're very, very obvious jumps in quality, so much so that they made Claude on it and 4.0 just like they made the previous state of the art look bad. Like it's really that stark, that difference. The number one thing I actually, you know, just feedback or request, feature requests is people want control on the budget because right now in 0.1, it kind of decides its own thinking budget. But sometimes you know how hard the problem is and you want to actually tell the model like, spend 2 minutes on this or spend some dollar amount. Maybe it's time, maybe it's dollars. I don't know what the budget is, but that makes a lot of sense.

So we actually thought about that requirement and it should be at some point we need to support that. Not initially, but that makes a lot of sense.

Okay. So that was a fascinating overview of just like the things that you're working on. First of all, I realized that I don't know if I've ever given you this feedback, but I think you guys are one One of the reasons I agreed to advise you, because I think when you first met me, I was kind of dubious.

I was like, who are you?

There's Replicate, there's Together, there's like a laptop. There's like a whole bunch of other players. You're in very, very competitive fields. Like, why will you win? And the reason I actually changed my mind was I saw you guys shipping. I think your surface area is very big. The team is not that big.

No, we're only 40 people.

Yeah. And now here you are trying to compete with OpenAI and everyone else. What is the secret?

I think the team. The team is the secret.

Oh boy. So there's no thing I can just copy?

No. I think we all come from very aligned on the culture because most of our team came from Meta and many startups. So we really believe in results. One is result, and second is customer. We're very customer obsessed. And we don't want to drive adoption for the sake of adoption. We really want to make sure we understand where to deliver a lot of business values to the customer. And we are, we really value their feedback. So we would wake up middle of night. And deploy some model for them, shuffle some capacity for them. And yeah, over the weekend, no-brainer. So yeah, so that's just how we work as a team. And the caliber of the team is really, really high as well. So like, as Plugin, we're hiring. We're expanding very, very fast. So if we are passionate about working on the most cutting-edge technology in the generative space, come talk with us.

Yeah. Let's talk a little bit about that customer journey. I think one of your more famous customers is Cursor. We were the first podcast to have Cursor on, and then obviously since then they have blown up. Cause and effect are not related. But you guys especially worked on a fast-apply model where you were one of the first people to work on speculative decoding in a production setting. Maybe just talk about What was the behind the scenes of working with Cursor?

I would say Cursor is a very, very unique team. I think unique part is the team has very high technical caliber, right? There's no question about it. But they have decided, although like many companies building coding copilot, they will say, I'm going to build the whole entire stack because I can. And they are unique in the sense they seek partnership, not because they cannot. They're fully capable, but they know where to focus. That to me is amazing. And of course they want to find a best partner. So we spent some time working together. They are pushing us very aggressively because for them to deliver high caliber product experience, they need the latency. They need the interactive but also high quality at the same time. So actually we expanded our product feature quite a lot. As we supporting Cursor, and they are growing so fast and we massively scaled quickly across multiple regions, and we develop pretty high-intense inference stack, almost like similar to what we do for Meta. That, I think that's a very, very, uh, interesting engagement. And through that, there are a lot of trust being built as in they realize, hey, this is a team they can really partner with and they can go big.

With.

That comes back to, hey, we're really customer obsessed and all the engineers working with them, there's just enormous amount of time syncing together with them and discussing. And we're not big on meetings, but we are like Slack channel always on. Yeah, so it almost feels like working as one team. So I think that's really highlighted.

Yeah, for those who don't know, so basically Cursor is a VS Code fork, but most of the time people will be using closed models, like I actually use a lot of Sonnet. So you're not involved there, right? It's not like you host Sonnet or you have any partnership with it though. You're involved where Cursor, Small, or like their house brand models are concerned, right?

I don't know what I can say for the things they haven't said.

I'm just saying like, it's very obviously the dropdown is 4.0, Spot, and then Cursor, right? So like, I assume that the Cursor side is their Firework side, and then the other side is calling out to the other. Just kind of curious. And then, like, do you see any more opportunity on, like, the, you know, I think you made a big splash with, like, 1,000 tokens per second. That was because of speculative decoding. Is there more to push there?

We push a lot. Actually, when I mentioned a file optimizer, right? So, as in, we have a unique automation stack that is one size fits one. We actually deployed to Cursor early on. Basically optimized for their specific workload. And that's a lot of juice to extract out of there. And we see success in, in that product is actually can be widely adopted. So that's why kind of we, we started a separate product line called the File Optimizer. So speculative decoding is just one approach. And speculative decoding here is not static. We actually wrote a blog post about it. There's so many different ways to do speculative decoding. You can pair a small model with a large model in the same model family, or you can have EAGLE. Heads and so on. So there are different trade-offs of which approach to take. It really depends on your workload. And then with your workload, we can align the Eagle heads or Medusa heads or, you know, small-big model pair much better to extract the best latency reduction. So all of that is part of the File Optimizer offering.

I know you mentioned some of the other inference providers. I think the other question that people always have is around benchmarks. So you get different performance on different platforms. How should people think about, people are like, hey, Llama 3.2 is X on MMLU, but maybe using speculative decoding, you go down a different path. Maybe some providers run a quantized model. How should people think about how much they should care about how you're actually running the model? What's the delta between all the magic that you do and what a raw model Okay.

So there are two big development cycles. One is experimentation where they need fast iteration. They don't want to think about quality and they just kind of want to experiment with product experience and, and so on, right? So that's one. And then it looks good and they want to kind of postpartum market scaling and the quality is really important and latency and all the other things are becoming important. During the experimentation phase is just pick a good model. Don't worry about anything else. Make sure even like GenAI is the right solution to your product, and that's the focus. And then post-market product market fit, then that's kind of the three-dimensional optimization curve start to kick in across quality, latency, cost, where you should land. And to me, it's purely a product decision. To many product, if you choose a lower quality but better speed and lower cost, but it doesn't make a difference to the product experience, then you should do it. So that's why I think inference is part of the validation. The validation doesn't stop at offline eval. The validation is kind of— we'll go through A/B testing through inference. And that's why we kind of offer various different configurations for you to test which is the best setting. So this is the traditional product evaluation. So product evaluation should also include your new model versions and different model setup into the consideration.

I want to specifically talk about what happened a few months ago with some of your major competitors. I mean, you know, all of this is public. What is your take on what happened? And maybe you want to set the record straight on how Fireworks does quantization, because I think a lot of people may have outdated perceptions, or they didn't read the clarification post on your approach to quantization?

First of all, it was a surprise to us that without any notice, we got called out.

Specifically by name, which is normally not what—

Yeah, in a public post and have certain interpretation of our quality. So I was really surprised and it's not a good way to compete, right? We want to compete fairly. And oftentimes when one vendor give out result from interpretation of another vendor, it's always extremely biased. So we actually refrain ourselves to do any of those. And we happily partner with third party to do the most fair evaluation. So we are very surprised and we don't think that's a good way to figure out the competition landscape. So then we react. I think when it comes to quantization, the implementation, we wrote out actually a very thorough blog post because again, uh, no one size fits all. We have various different quantization schemes. We can quantize very different parts of the model, uh, from weights to activation to cross-GPU communication to like consi— they can use different quantization scheme or consistent across the board. Again, it's a trade-off. It's trade-off across this three-dimensional quality, latency, and cost. And for our customer, we're We actually let them like find the best optimized point and that's kind of how, and we have very thorough evaluation process to pick that point. But for self-serve, there's only one point to pick. There's no like customization available. So of course we depends on like what we, we talk with many customer, we have to pick one point and I think the end result, like AA published, uh, later on AA published a quality measure and we're actually, we look really good. So I, I don't, I wouldn't, that's why what I mean is I will leave the evaluation of quality or performance to third party and work with them to find the most fair benchmark approach and methodology. But I'm not applaud of approach of calling out specific names and critique other competitors in a very biased way.

Databases happens as well. I think you're the more politically correct one and then Dima is the more— This you on Twitter? He's like the Russian—

We partner. No, actually all these directions we build together. We play different roles.

Another one that I wanted to— on just the last one on the competition side, there's a perception of price wars in hosting open source models. And we talked about the competitiveness in the market. Do you aim to make margin on open source models?

Oh, absolutely, yes. So but I think it really— when we think about pricing, it's really need to coordinate with the value we are delivering. If the value is limited or There are a lot of people delivering same value. There's no differentiation. There's only one way to go is going down, right? So through competition. If I take a big step back, there is pricing from— we are more compared with like closed model providers, APIs, right? The closed model provider, their cost structure is even more interesting because we don't have any— we don't bear any training cost and we focus on inference optimization and that's kind of where we continue to add a lot of product value. So that's how we think about the product. But for the closed-source API provider, model provider, they bear a lot of training cost and they need to amortize the training cost into the inference. So that created very interesting dynamics of, yeah, if we match pricing there, and I think how they are going to make money is, is very, very interesting.

So for listeners, OpenAI is 2024, $4 billion in revenue, $3 billion in compute training, $2 billion in compute inference, $1 billion in research compute amortization, and $700 million in salaries. So that is like, I mean, a lot of R&D.

Yeah. So I think Matter is basically like make it zero. Yeah. So that's a very, very interesting dynamics we're operating within. But coming back to inference, right? So we are, again, as I mentioned, our product is— we are a platform. We're not just a single model as a service provider as many other inference providers, like they're providing single model. We have our optimizer to highly customize towards your inference workload. We have a compound AI system where significantly simplify your interaction to high quality and low latency, low cost. So those are all very different. For other providers.

What do people not know about the work that you do? I guess people are like, okay, Fireworks, you run model very quickly, you have the function model. Is there any underrated part of Fireworks that more people should try?

Yeah, actually one user post on x.com, he mentioned, oh, actually Fireworks can allow me to upload the LoRA adapter to the service model with at the same cost and use it at same cost. Nobody has provided that. That, that's because we have a very special, like, we, we rolled out Multi-LoRA last year actually, and we actually have this function for a long time and many people has been using it, but it's not well known that, oh, if you fine-tune model, you don't need to use on-demand. If you fine-tune model, use LoRA, you can upload your LoRA adapter and we deploy it as if it's a, a new model. And then you use, you get your endpoint and you can use that directly, but at the same cost as the base model. So I'm happy that user is marketing it for us. He discovered that feature, but we have that for last year. So I think we, I think to feedback to me is we have a lot of very, very good feature as, as Sean just mentioned the weights.

I've been 5 years to the company, and I didn't know that you had speculative decoding released.

We have Palm Cache way back last year also. We have many. Yeah. So yeah, so I think that is one of the underrated features. And if they're developers, you are using our self-serve platform, please try it out. Yeah.

Yeah. Yeah. The LoRA thing is interesting because I think you also— the reason people add additional cost to it. It's not because they feel like charging people. Normally, in normal LoRa serving setups, there is a cost to loading those weights and dedicating a machine to that inference. How come you can avoid it?

Yeah, so this is our technique called multi-LoRa. So we basically have many LoRa adapters share the same base model. And basically, we significantly reduce the of memory footprint of serving, and one base model can sustain 100 to 1,000 law adapters. And then basically all these different law adapters can share the same, like, drive the same traffic to the same base model where base model is dominating the cost. So that's how we augmentize that way, and that's why how we can manage the tokens per dollar, million token pricing the same as base model. Awesome.

Is there anything that you think you want to request from the community or you're looking for model-wise or tooling-wise that you think like someone should be working on?

And it's, yeah, so we really want to get a lot of feedback from the application developers who are starting to build on GenAI or on the already adopted or starting about thinking about new use cases and so on. To try out our Fireworks first and let us know works out really well for you and what is your wishlist and what is, what sucks, right? So, uh, what is not working out for you and would you really like to continue to improve? And for our new product launches, typically we want to launch to a small group of people. Usually we launch on our Discord first to have a set of people use that first. So please join our Discord channel. We have a lot of communication going on there. Again, you can also give us feedback. We'll, we'll have started office hour for you to directly talk with our Gabriel and engineers to exchange your love notes.

And you're hiring across the board?

We're hiring across the board. We're hiring frontend engineers, infrastructure, cloud infrastructure engineers, backend system optimization engineers, applied researchers, and like researchers who has done post-training. Who has done a lot of fine-tuning and so on.

Cool. That's it. Thank you.

That's awesome.

Thanks for having us.