Lin Qiao from Fireworks argues that scaling AI inference 100x requires aligning models with your actual workload, not just using off-the-shelf models. He describes a three-dimensional optimization problem across quality, speed, and cost, with over 100,000 possible configurations. Fireworks' platform helped a food-chain client expand from one shop to 1,000 in three months.
I'm so excited to bring up Lin Qiao. If you want reliability, scale, performance, you couldn't find a better person than Lin to deliver that. As Lin sets up, I'll tell you a little bit about Fireworks, which is one of the leading, most reliable, most performant inference providers in the world. And I can't wait to turn it over to Lin to share a little bit about how they do it.
Cool. We can get going. Yeah. So today, I'm going to talk about what's the future state of inference? What does it look like? I guess many of you are very interested in that topic. So I want to start talking about inference through the eyes of app developers and innovative product developers. You first start from build. An ideation into scale after your product market fit. And I want to actually look at that, those phase through a very simple lens of view. Is eventually you're all doing alignment towards your audience. It's massive alignment process you're doing from the beginning to the end. So this alignment actually had multiple phases. Here, you actually want to optimize for the user behavior you want to drive. Then from there, your product design need to in tune towards that. This has been an area that has been heavily studied in the past many, many years. A lot of tools has been enabled for you to do product analytics on logging and so on. Now in Gen 9 phase, as model becomes intrinsic part of your product stack, and this part of alignment is completely new. And I will say it's very weak in this phase because app developers just use off-the-shelf model most of time. And how to infuse your product knowledge into your model is a new area that most people doesn't know how to do. And the best way to steer those models through prompt engineering. So, but if you look at a lot of successful applications and products, As we talked about earlier, a lot of data flying wheel is the mode you're going to incorporate into a product and stand out from that point. So this is a very interesting area that there's not much work that has happened, but we have seen— we have worked with many high fast-scaling startups, and we have seen a lot of success in the areas who can mask that deeply. Again, coming to this phase is the product design to user behavior alignment is in application. The future inference to product design alignment is off application space and how to make it easy, accessible, and high quality is the next generation question. More specifically, more specifically, this alignment is very simple. The idea is very simple, is to align data distribution in your application workload into the data distribution in your training data of the model, essentially. But that alignment is very hard to do because the researchers has to make assumptions who is going to use this model to solve what kind of problem. And oftentimes, those assumptions may not fully connect with real-life application use cases. It's almost always guaranteed there was always gap, as well as the inference system design, it is not one size fits all. It also need to use your production data to drive maximum amount of optimization. So I will say we talk a lot about scaling law in training, scaling law in inference. I think the future of scaling law is actually across three dimensions in inference, across quality, speed, and user concurrency, which is cost. When we think about that, we would talk with many startups or enterprise coming to us, and I'm like, the question is, what do you want? What ideally, what is the ideal state you want? The business said they want open eyes quality, light speed, and high concurrency as if they are running fraud detection. So this is, they want all of that. So basically, it becomes a multidimensional optimization problem for inference, but specific for a particular application. That's the future, is the heavy customization of scaling inference and optimizing inference towards a specific application. That's the future scaling law. And the approach to do that is actually not look at inference in isolation. The approach to do that is to combine post-training and inference to do co-optimization. That's the area to drive massive amount of innovation and acceleration. And ultimately, the goal here is to drive down extremely high inference cost, which is today's state, to 10 times or 100 times lower. Imagine this is a— we are seeing an iceberg situation where the waterline of inference cost is very high right now. And when the cost is driven down to 10 times, 100 times lower, And all the applications which fit into product-market fit can scale into a sustainable business will be much, much bigger. So that is goal of the future scaling law. The bad news is it is very hard. To solve this problem, it is very hard. Because to do this three-dimensional optimization across quality, speed, and cost, there are a lot of elements to put together. Those elements cover through, hey, I don't want to predict one token at a time. I want to predict 10 tokens at a time and how to do that. How do I align numerics, the positions towards your data distribution in your application? How do I align the selection of hardware? Each hardware skill has its own unique advantage and benefits. They're all different towards operation data distribution. How do we shard the model towards your application? Distribution? How do we do cross-host distributed inference towards your application distribution? How do we select the most optimized kernel towards your application distribution? And how do we have variety of tuning mechanism for quality towards the application? So in combination, it can lead into more than 100,000 combinations to pick and choose from. So it's a hard problem. The good news is we are set to solve that problem. And we have been focused on investing R&D to deliver and solve complex— this combinatorial explosion problem. And this is a little bit too detailed. But I don't have a good way to capture what we have built at Fireworks. At the bottom layer, we have a virtual cloud infrastructure where we take over the consideration of managing all the complexity across where do I find GPU, which is guaranteed to have high quality, high reliability for me to scale my application. And we also build on top of multiple different hardware from different hardware vendors, from different SKUs, where some hardware has much higher flops. Some hardware has much more memory bandwidth that is good for different form and shape of your application specifically. Two, a variety of models you can pick and choose from open model libraries where you can easily steer towards your application needs to various different ways to tune. We tune for speed. We tune for quality and so on. Various different mechanisms. I'm very excited. We're going to talk about how to incorporate your production data to reinforcement tuning towards your application needs. And on top of that, we have a lot— bunch of developer tools to make it super easy and accessible. So we have launched our developer-facing platform self-serve for a long time. This is in the past year and a half. So a lot of— we have seen like the most exciting thing for us is to see through this lens of alignment product, a lot of applications has able to onboard and do experimentation on our platform and scale. So very interestingly, we have seen that a food chain company, which is one of my favorite. They use our service and scale from running one shop of AI feature, very interesting AI feature, into 1,000 shops in 3 months. We also seen a software development company roll out their AI feature from 100,000 developers to 25 million developers within 3 months. So the fast scaling is fascinating here. And we want to make sure as they scale, they get to the most optimal sweet spot across quality, cost, and speed. With that said, it's very easy to get started. And love to hear from you. All the people here build application. What are the next generation problems you are facing for inference? And love to chat more.
