In this presentation Lin Qiao, co‑founder and CEO of Fireworks AI, explains how generative AI can move from hype to delivering real business value. He begins by highlighting the team’s background in AI infrastructure at Meta and Google and their deep expertise in PyTorch. Qiao contrasts large foundation models (e.g., OpenAI’s GPT‑4) with smaller open‑source models, emphasizing the trade‑offs: large models offer high out‑of‑the‑box quality but suffer from high latency and cost, while smaller models can be fine‑tuned cheaply and run faster, which is critical for user engagement and profitability. He quantifies latency and cost differences, showing that a 70‑billion‑parameter Llama model on Fireworks can be an order of magnitude faster and up to 40× cheaper than GPT‑4 for a billion‑token daily workload. He then outlines the production challenges of deploying small models—hardware procurement, serving‑stack setup, observability, and lack of hardened open‑source tooling. Fireworks AI addresses these with its proprietary inference stack: hand‑written CUDA kernels (FireAttention), a quality‑preserving quantization scheme, speculative decoding, multi‑layer semantic caching, a custom distributed inference system, and the TAC workload optimizer that auto‑tunes configurations. Benchmark results demonstrate 8‑15× lower latency or higher throughput compared with competitors such as Groq and the vanilla vRM stack. The talk moves to the model catalog, noting support for over 100 top‑quality models across modalities (Llama 3, SD3, Phi‑3 Vision, Whisper, Gemini, Mixtral, etc.) and proprietary models. Fireworks also offers fine‑tuning services; it efficiently serves thousands of LoRA adapters by sharing a single base model, achieving up to 1,000× resource savings. Recognizing that a single model cannot solve all tasks, Qiao introduces the concept of "Compound AI"—orchestrating multiple models and external APIs via a proprietary Fire Function model that decides when to call tools, retrieve real‑time data, or generate images. A live demo showcases a chat that fetches stock prices, builds a chart, and generates an image, illustrating the seamless integration of language, retrieval, and multimodal generation. The compound AI platform is already used by enterprises such as Upwork, Quora, Uber, DoorDash, and startups like Cresta and Sourcegraph. Finally, he outlines product options (serverless inference, custom fine‑tuning, enterprise plans) and shares an optimistic outlook on agentic workflows, where AI agents can self‑organize and coordinate to solve complex tasks, echoing the self‑play advances seen in AlphaGo. Overall, the presentation positions Fireworks AI as a high‑performance, cost‑effective inference and orchestration platform that enables businesses to harness small, fine‑tuned models and compound AI systems for real‑world applications.
Chapters
Hi everyone, I'm co-founder and CEO of Fireworks AI. Today I will talk about how to go from GenAI as a hype to high-value AI. Before I start, I would like to quickly talk about our team. Our founding team came from Meta and Google. We were veterans building AI infrastructure from scratch, supporting hyperscalers' AI needs. We're also backed by both Benchmark and Sequoia. By the time we left, we're serving more than 5 trillion inference per day across the entire globe. I also want to quickly talk about PyTorch. It is the dominating AI framework today. And we have been the team building PyTorch over the past 5 years. In the context of today's talk, we're gonna focus on generative AI technology that is transforming the industry. And almost all generative AI models were built in PyTorch and currently run in PyTorch. So that's our main expertise, and that's why we're focusing there. I'm gonna dive into this interesting question for all of you building disruptive, innovative applications and product on top of this new GenAI technology. Are you going for the big models or going for the small models? Of course, big models come with a lot of benefits. OpenAI is leading the pack here. Its famous GPT-4 model, now GPT-4o, can support doing multiple tasks at the same time. For example, it can do chat, it can generate code, it can build agents, it can do many other things. What comes with it is it's really good quality from get-go, off the shelf, but also because the model is on a much bigger side, its latency is much higher, and cost is also much higher. Speaking of latency and cost, let's dive into it. It's proven and well-established for B2C applications, fast latency directly correlates with better user engagement. Just give you one example. In the GenAI world, this GPT-4 latency compared with the open source model actually coming from Meta, Llama 3.7 billion. Served by Fireworks, for example, the latency difference can be one order of magnitude. With that latency reduction, our application can get drastically different product experience. It could be make or break for the product viability. Next, the cost. There's this fundamental shift of building application pre-gen AI and on-gen AI. Because pre-gen AI applications are building on top of, or running on top of, heavily commoditized CPU. From the business operation cost point of view, it's very small. The CPU overhead is very small. And gen AI is different. It's on the high end, highest end of AI models, and it can only run on GPUs. GPU is very power hungry. Power is very expensive. Power generates heat, heat requires cooling, cooling is very expensive. So all the surrounding infrastructure of operating Gemini is very costly, and that significantly shifts the cost and operating structure of applications built on top of that. And as you can see, comparing big model and small models, again, Llama 3, 70 billion, running on Fireworks, and GPT-4 from OpenAI, the cost difference can be 40 times. And just put that in more concrete terms. Assuming you are processing 1 billion tokens per day, and that means use large over small model, you're gonna every day, daily, spend more than $37,000. That amounts to $13 million a year. So that's a big amount of money, not just for SMBs, but also for big enterprise. And this high cost or low cost is break or make it for business viability. And next, about how— whether— how easy is it to adapt to domain-specific problems? And the goal of a big model is, hey, you don't need to adapt, and these big models can solve all the problems for you. But it's— a lot of time, it doesn't. And it's actually much easier to adapt to small models. With small amount of data, as long as high quality, into specific use case, specific domain, or specific workload. One great example is Hugging Face. There are more than 50,000 open source models that is, has been adapted or specialized to solve specific use case problems. That highlights the need of doing that across the board. And we see many small models has been adopted and used to unlock real business at scale across chat, code generation, Code Copilot, agentic workflows, data processing in general. So this all sounds great. What are the problems? What are challenges of working with small models and why small models are not as popular as big models yet in terms of adoption, usage, traffic, revenue, all this. So there are a bunch of production challenges to use small models for real. It starts with you still need to set it up, right? You just have a model itself, it doesn't run itself. You have to procure hardware from cloud providers, and then you have to bring up the serving stack, which you have never operated, and you have also bring up production-related tools, observability, monitoring, all these nine yards. And next, even if you bring it up, that doesn't mean your small model is the most optimized when it comes to latency or cost to help your business scale. And you still need to do a lot of work to make it more optimized, and that's very deep specialized expertise. And last but not least, the whole entire stack from open source support point of view, it's not very production ready. And takes, takes a while to harden, solidify it. So Fireworks, we are committed to offer a fast and cost-efficient AI inference engine to address all the challenges I just mentioned. To begin with, we offer the fastest serving stack based on proprietary technology called FireAttention. So we handwrote CUDA kernels from scratch. We also implement a special quantization scheme that does not trade off quality with speed. We also have our special speculative decoding implementation. We implement multiple layers of semantic caching to make inference much more efficient. And we also implement our own way of doing distributed inference that's very unique. Not just that, we also bring our TAC, like, to further customize the serving for you. It almost work like a database optimizer where it observe the workload, it can adjust the configuration, the setup, so your inference become even more optimized and customized based on your unique workload pattern. And we have rolled it out to, to production, and we care about the most common use cases. For example, large language model with RAG is very commonly used, and of course there are many people trying to solve similar problems. This is a benchmark result driven by Affirm, called artificial analysis. As you can see, Fireworks is number 1 in delivering low latency here. And note that Groq has been very vocal in marketing their fast inference engine built on top of ASIC, and in this use case, we're 8 times faster. When it comes to image generation, the same firm benchmarked us against other providers. We're also are extremely fast, 9 times faster compared with the ones here, and we're very proud of where we are, and we'll continue to push the boundaries of inference speed. We also compare with open source serving tier. The most popular one is VRM. As you can see from this chart, we are 8 to 15 times higher throughput. That is the efficiency we can drive on top of the vanilla VRM. So let's switch gears, talk about the models, the small model itself, right? The small models in the industry comes with various different modality, various different flavors, and we support all the best ones. For example, Llama 3 from Meta, SD3 from Stability, Phi-3 Vision from Microsoft, Whisper from OpenAI, Gemini from Google, Mixtral from Mistral, On top of that, there are many community-tuned small models. In collection, we offer more than 100 top-quality models for our inference engine with our fast— the fastest, most cost-efficient inference behind it. And you do not just leverage existing models we provide. On top of that, we also have our own proprietary models that specialize in certain areas. I will touch on that soon. But many developers or enterprise come to us with their own proprietary data. You can use our fine-tuning service to generate your customized model, and that customized model will significantly create differentiation for your applications because it's unique, and nobody will have any advantage over your own unique customized model. So that becomes your moat. After you have your customized model, we help you further optimize the inference part. As you can see from this diagram, usually people use PaFT or LoRA adapters to fine-tune models, and the naive way to serve LoRA adapters is to serve that together with the base model and independently. That is extremely wasteful in terms of using the resource. and we implement an extremely efficient way by having hundreds of thousands of LoRa adapters sharing the base model. That way you can achieve 1,000 times more efficiency when, when you kind of want to serve this many LoRa adapters for your customers. And this mechanism is, is extremely effective. And we also heard that Apple announced their Apple Intelligence last week. They are also using similar technology. We talk a lot about small models, single small models, but oftentimes to solve a business task, we have to go beyond a single model. The fundamental reason is single model is very limited. It's limited because take large language model, for example, it is probabilistic, and every single token it generates is based on the probability of prediction. And that's very different from app development before GenAI because it's more— it's a lot more deterministic. And now app developers have to figure out and learn new ways to control hallucination. And second, individual model, they specialize in certain area, but they cannot cover a much broader ground. As you can see here, we have different modalities from large language model to audio, TTS, speech synthesis, or embedding for semantic search, vector search, or image generation, or multimodal to extract information from images. Within each category, there are also many models with its own specialty. For example, there could be models really good at long context. There could be models good at chat. There could be models good at classification, so on and so forth. Oftentimes when you want to have a combination of strands, then you have to leverage multiple models, and single model is not enough. Last but not least, each generative model has limited knowledge because its training data is finite. It's not infinite. With finite training data, you can only fusing finite amount of knowledge. Oftentimes there are a lot of additional knowledge sit outside of the training data, sit outside of those generative models. For example, real-time information behind public API, or within enterprise, there are hundreds of private APIs having enterprise internal knowledge where it's, it, it's not there in the GNN model, in the foundation models. There are a lot of information in the storage tier, in databases, file system, and so on and so forth. So we have to broaden the single model serving into a lot more complexity in order to solve a business problem. So that, that leads us into so-called compound AI systems. Compound AI system is to have multiple models to work with each other, or model works— work with additional tools or APIs to deliver better AI results. And the Fireworks AI we are building out a compound inference engine. So here's what it looks like. On the left-hand side, you can see we have our building block of Fireworks inference engine where it serves the hundreds-plus GenAI models from stock model to tuned models along with your customized model with your data. And sitting on top of that, we offer our proprietary Fire Function model. That model is capable of calling out into different APIs or different models. So imagine the Fire Function model itself is a large language model. It's capable of doing converting, you know, having chat with you, having conversation, but also it will recognize, oh, this is going out of my knowledge. I need to call into additional tools, as you can see from right-hand side. It will understand for certain case, need to call into certain kind of API to retrieve information. So the Fire Function layer provide the orchestration to assemble different models, different modalities, different tools and APIs all into one compound AI inference engine provided by Fireworks. Here I want to quickly show you a demo, and this demo is built by us. As you can see, you can start to have a conversation. It's a chat model, and on the upper right corner, it shows you what are the APIs or models we have preregistered. Here we have search, we have stock access APIs, we have image generation APIs, and so on, so forth. That is being preregistered, and, and now you can start to ask more questions. Just for the demo, we want to ask, what is the stock price of top 3 cloud providers, right? And under the COVID this model is gonna issue multiple function calls, multiple API calls. First API call into retrieve the top 3, Cloud provider name, and then 3 parallel function calls, API calls to get the stock price, and the next function calls to create this chart. So boom, you get this chart quickly. And then we ask, hey, what is Oracle's? Add Oracle into it. It carries the context. Oh, you were asking the stock price. So it, see here, it repeat the same process and add Oracle to the chart. So it's very smart. And for the fun of this demo, we want to generate image of cat sitting in a data center. It understand, oh, you are now, you need an image generation capability. So here it generates image by calling into our image generation API by Fireworks. So this is a quick demo of the, how powerful a compound AI system can be, can build on top of Fireworks. We have open source this demo code. You can access it, go to the link here, play with it, tweak it, enhance it, build your own application on top of Fireworks Compound AI system. And this Compound AI system has been used by many companies already, from enterprise like Upwork, Quora, Uber, DoorDash, to emerging startup like Cresta, Cursor, Sourcegraph for Coding Copilot, a ton of other assistants like medical assistants, educational assistants, productivity tools, and so on and so forth. And I would like to invite you to try out Fireworks. It's extremely easy. Go to fireworks.ai/models, and you can get the testnet going. We also offer various different options for you to try, all the way from serverless inference. You can also start your own customized model or fine-tuning service on demand, or you can talk with us to start an enterprise plan. And those 3 options give you the maximum amount of flexibility from pay-as-you-go to fastest and most cost-efficient deployment. To wrap up my talk, looking forward, I'm super excited about the development of agentic workflow. Instead of producing generative content only for consumers or developers, there's active development in building agents to talk with each other. As we see from AlphaGo, it can learn for itself and significantly improve the quality of its skills by playing Go with itself. In the future, those agents will be able to self-organize and self-coordinate to solve a complex task. I'm super excited what can come in the future.
