Lin Qiao, Fireworks AI: Wave of Open Source AI Adoption in 2025

Thank you, Philip. So next up, we have Lin Qiao of Fireworks AI. So I've always wanted to talk about AI trends, and they have some of the best data to give us the real trends. So please welcome Lin Qiao.

Good morning, everyone. I'm Lin. I'm CEO and co-founder of Fireworks AI. It's good to have you here. So today, I'm going to talk about, as a lot of you are going into building various kind of agents for your company, for your new products, what are the challenges you're going to run into and how Fireworks helps you go through that development cycle. So there's no question that this year is the year of agents. Fireworks, we have onboarded so many first different kind of agent applications. All the way from coding agents that is redefining how we develop software engineer— software across the board, and the document agents where it assists you to write proposals, take actions, and track your projects, and sales marketing agents where it can automate the outbound emails and messaging target list based on your target ICP, and hiring agents who can automatically do interviews and select the best candidates based on your job profiles. Customer service agents, which can make your human agents very high quality, productive, and even automate some of the tasks. And this list goes on and on across various different areas from retail, insurance, finance, education, healthcare, manufacturing, and so on. So let's just take a few examples. There are many companies running on top of Fireworks today already. Please raise your hand if you are Cursor user. So we have a huge Cursor fan here. Cursor runs on top of Fireworks. It is the fastest growing AI-native startups. Its flagship TabTab product has been leading the industry, and it grows from a very small cohorting to extremely large developer base. and the fireworks help them fulfill their full potential. Uber is one of the biggest mobility and delivery global company worldwide. They have also been on the frontier of building all sorts of agents, from the agents that help the driver onboarding within a few hours, not in a few weeks. They also build personalized customer service agents. as well as data analytics agents for internal productivity use. Uber use Fireworks. UiPath has been historically an RPA company, and recently they shift massively and rebrand the company to become an agent-first company. And their goal is to partner with Fireworks and deliver a hybrid experience with robotics, AI agents, and humans working seamlessly together. If we take a huge step back and look at the companies that is taking the lead on redefining the user experience, there's a common pattern across them. First, their way to think about the product design principle is different. Principle number 1, They do not think about model just as a commodity, as a utility they call into, as their product wrapper built on top of this utility. They're actually thinking about how to extract the maximal amount of value out of the model and design a product hugging around that model. So they are doing active product-model co-design. And that's very deep work. Not everyone is thinking that way. We have seen many companies who are very deep in extracting the best value out of the model being extremely successful. And there's a profound reason behind that. So first of all, the product which is developed by those app companies and the model which is developed by Frontier Labs, those are by design misaligned. The fundamental reason is the data distribution used to train the model in the Frontier Labs cannot be fully aligned with the data distribution of the new user experience you are pushing. The app company is going to push forward and experiment with their consumer or developer basis. And because of this misalignment, you're going to leave a lot of quality, speed, and the cost efficiency optimization on the table. So model-product co-design becomes a new thing. And second, because of this model product co-design, and there's a trend of people tune their model and model becomes their proprietary IP. And they're creating a healthy data flywheel from the data trace from their product to make the model better. Better model made a better product, driving more user engagement, more user engagement drives more data, and therefore this data flying wheel is virtuous. So at Fireworks, we have seen the lifecycle of the development of agents in roughly these 3 stages. First is build, where you experiment with various different new ideas of your product. And second, once you hit the early stage of product market fit, you start to customize. And start to create this data flywheel. And third is significant scaling phase post product market fit into billions of consumers or millions of developers. Let's first take a look at the build stage. So first of all, as you are developers and practitioners in the space, you must be aware there's gazillion number of models that emerging almost every week. And Fireworks is— we are always on the frontier of enabling the state of art, especially open models. And this is a funny tweet from Artificial Analysis that we provide an open-weight model faster than the model provider. In this case, it's Google. It's kind of bizarre, but that's who we are, and we enable day-zero model access for our platform. Over the past 2 years, We have enabled a slew of models across multiple modalities, from large language model to image generation models to embedding models for semantic search to models who can— vision language model, which can understand images, PDFs, documents, to audio models, which can enable more human-like interaction. And video generation models. So even better news, we have significant pop-up the number of models we support our platform from 200 to more than 1,000. With that, there's no limitation of your imagination. You can pick and choose the best of the best for all different modalities from our platform. This summer is a crazy summer for Open Models. And here on this table, we just listed a subset of the models we have enabled over just this past month and a half. And as you can see from this table, the models all coming with various different sizes, number of parameters, number of— different number of active parameters, different requirements on KV cache, and the different number of layers So enable them is a big headache. At Fireworks, we take over that burden, and we bring the best models with the best performance and the best quality to all of your developers. So you can tap into all the state-of-the-art models without any other burden to consider. We have launched experiment platform. Where you have day-zero access to the best models, best open models in the world, and you can easily scale up and down your deployment for you to experiment, to build new innovative product on top of those models and engage with your specific user base to experiment and test your ideas easily. We have also offered a build SDK where you do not need to go through our UI, and you can use code to control the scale up and down of your deployment easily. We designed this for enterprise from day zero because we offer private and secure access to our platform. One of the biggest challenges I've heard is How do I even know? It's kind of exhausting to chase all these different models. How do I even know whether I should care about this model or should forget about it? So evaluation is a real pain. The way for you to think about evaluation is it is the unit test of our software development, actually. So during the early stage of model selection phase, pick and choose the best model for your use case. To continuous integration testing and regression testing, whether a certain model, if you want to introduce, is going to change the quality in your production, to finally want to customize and using very cutting-edge techniques like reinforcement learning. Everything underpinning the whole entire lifecycle of this development is evaluation. So to support developers to do evaluation much more efficiently, we have open sourced a product called eval protocol. And here, please go to evalprotocol.io. We have a lot of pre-baked examples to help you bootstrap your development process of writing easy evals for your production and experimentation. And we love to have your contribution back. To this open source project. Many companies built with us. We have onboarded more than 10,000 companies on our public platform. This is more than— I think this data is old— is more than 20x growth within a year. And all these companies' feedback is using our experimentation platform, they are 100 times faster in their iteration. The second phase of development is customization. Now you have your product market fit, you start to curate a data flywheel. And here's what you're going to do. Fundamentally, the picture here is you're going to trace data from your product, from your applications. And this is nothing new. Because for you to do product analytics, understand which product feature is good or bad for your product metrics is a well-established practice. So you have already been doing that. You already have those product traces. And then what matters is how to feed those traces into your model to make your model more smarter, more catered towards your use case. So the process here is to create this data frame more easily. We have invested in various different technology in that space for a long time. First is pretty established technology called supervised fine-tuning. And there, what we heard is our user, they really care about powerful big models, and tuning those models are extremely hard. And they really care about long context because those long context, context engineering, is very important for the quality of the products. So we have significant investment in those areas to enable very large MoE models and very long context fine-tuning. More importantly, recently, we have enabled reinforcement fine-tuning. So reinforcement learning is a new technology. We are making it extremely accessible to our developers, many of you here. That you should give it a try. And here, you have access to be able to use reinforcement fine-tuning to tune the best models in the world and make it your proprietary models and build your mode around that. So here, I want to dive a little bit more into the reinforcement fine-tuning process. The process is actually a process we are very familiar with. For example, this is a selection process of— imagine we're all farmers to kind of plant the best tomato that tastes like the best in the world. In the selection process, you plant the seed, you harvest it, you taste it, and only select the seed from the tomato you like and then plant it and then continue to do do this tasting experiment until you like it. So reinforcement learning or reinforcement fine-tuning work exactly like that. So here is the workflow. And first of all, the tasting part of this planting tomato exercise is called create evaluator. As you curate a reward function, and telling the model what is a good taste, what is a bad taste. And then based on that reward, the model is going to adjust itself and plant in the next seed for you to taste again. So this is the most creative part of this process. And it will require you to kind of figure out what is your tasting criteria. And after that, it's pretty conventional. After that, you just select a model you really want to tune and select a dataset you want to tune against, and then start running your tuning job. And finally, look at the result and see whether you want to continue or you want to change your setup. Here, let me show you a quick demo of the process. If it plays—

I don't think—

let's see if we can make this demo work. OK. Anyway, so if you're interested in demo, please come to me after this talk, and I will show you. The process is very simple. Very simple, and it basically breaks down into two, two phases. One is you test your reward. You basically test your tasty buds, how you want to encourage model to go in which direction. And we have IDE for you to develop that and then do local testing. Once you're happy with that, then you go to the training process, and there's a dropdown menu for you to select any model you want and your dataset you have uploaded. And then there are a bunch of configurations you can pick and choose from, or we have preset configuration you don't need to worry about. And then you kick off the job and go. After job is done, we give you back a lot of telemetry about quality you can look at and take action based on that. So the process is actually very straightforward. And you can go to fireworks.ai to start to play with it and build your own reward function and your own reinforcement learning jobs. We have seen extraordinary results based on this new technology. So for example, for an agent that application built for doing candidate hiring, candidate selection, recruiting purpose, We have tuned a model that is 9 times faster than a state-of-the-art closed model and 20% more accurate. So 20% more accurate is a big deal. Here's another example. This is a code editing example where we drive code rewrite for a very large file. And here on the left-hand side is a proprietary model, 70% accuracy. In the middle is an off-the-shelf open model with only 57% accuracy. After our reinforcement training, we, we reached 85% accuracy, beating the closed model. And the result is it's just not better quality, it's also 8 times faster and 50 times fewer errors. This is not the end. Actually, it's very, it's very good to combine supervised fine-tuning and reinforcement fine-tuning together to drive even better quality results. And here's another example. It's a much harder task. This task is about reasoning in a specific domain, vertical domain. And with the combined supervised fine-tuning and reinforcement fine-tuning, we can achieve almost 20% higher accuracy compared with the CoAsE model. As you are all building agents, function calling is extremely important to get it right, as in to call into which kind of MCP server. So this task is very important for the agentic quality. And we started with an open model in the middle orange bar that is extremely low quality in terms of function calling accuracy. And we do this combined supervised fine-tuning and reinforcement fine-tuning to bring the accuracy on par with the closed model. Therefore, with lower cost, With lower cost, we actually get into on-par quality. And that's really good for your application. Now I want to talk about how you scale your application to a large cohort of your end users. Typically, our users ask us, when they scale, which hardware should they use? Where is the capacity they should get from? And there are a lot of system failures cutting across a thick system stack. How do they deal with it? And how do they optimize for production metrics, including reliability, speed, and other things? So this is really daunting for them to really take care of all different aspects. And we take care of that. Today, we already have a very a very large fleet of Fireworks virtual cloud that is running on top of 8 different cloud providers across 18 regions. You can access to any of the underlying cloud or hardware, heterogeneous hardware we provide through our virtual cloud platform. We have very advanced disaggregated engine to maximize the efficiency and pass the efficiency of the execution to you. It's very high reliability, and we allow global scheduling to really pack the execution very, very well. This is also designed for enterprise from day zero with fully baked-in privacy and compliance. So one proprietary technology we have is called 3D Optimizer. This 3D Optimizer basically optimizes across quality, speed, and cost all at the same time. Because this is always the top of mind that as an application developer, you do not want to sacrifice any one dimension. You want to get the best out of all three dimensions for you to launch your product and scale fastest to the broader, broadest set of users and users you want to reach out to. The bad news is this three-dimensional optimization is very complicated. There's so many different options that will compound with each other and create a huge search space of more than 100,000 different options you can pick and choose from. At Fireworks, we help you find the best point in this 3-dimensional space that is catered towards your application needs. So we do not believe in one size fits all, and we believe in one size fits one. And we have this 3D Optimizer that is designed to cater the inference stack towards your specific application workload pattern. By numbers, today we are running more than 10 trillion tokens per day. And this is actually more, way more than the reported number from one of the largest CSPs of their open-weight model traffic. We're also processing more than 180,000 requests per second. Just putting perspective, this is in the same ballpark of Google search traffic volume. Many enterprise customers scale very fast on top of us. We have onboarded, for example, FuCheng company. They are scaling from using Fireworks in 1 store to 100 stores within 3 days, and now scaling to 1,000 stores within 1 month. We have a digital native company, they deploy Fireworks to enable their AI native feature and grow their customer customer base by 30 times within 1 month also. And there's another software productivity companies quickly scale their AI features from tens of millions of their end users to hundreds of millions of their end users. We would love to invite you to build with us together. So please go to fireworks.ai to start your building journey. And we have various different tools to help you go through the build, customize, and the scaling phase of your end-to-end agent development lifecycle. So please go to our website and take a look. Or you can LinkedIn me, Lin Qiao, last name Q-I-A-O. Happy to have a conversation with you.