Lin Qiao breaks down when enterprises need fast versus slow AI models. Real-time applications like fraud detection need sub-second responses, while offline work like legal research can use slower models that reason through complex problems. She recommends a small-to-big-to-small workflow: validate with a small model, fine-tune a large one on clean data, then distill it down for production speed.
Chapters
You might have a, uh, an interesting and informative perspective on reasoning models or slow thinking models. So that's been a big thing last year. OpenAI, Anthropic, Google— there's a big fuss around the slow thinking models that they released, and particularly their performance on complex tasks like Math Olympiads and writing academic papers and these kinds of things that require, you know, more processing before just outputting some kind of train of thought. Uh, I noticed on the Fireworks website that you mentioned very fast latency, like sub-2-second or even sub-500-millisecond latency in a lot of real-world deployments. Do you think that The like slower reasoning models that take time before they output something. Do you think that there's a lot of enterprise need for that?
So I would classify again, extremely simplified way of looking at application. So one type of application is real-time response. So it's either human-facing interactive response latency, or it's kind of fast transaction-facing, for example, fraud detection, right? So in aggregate way is kind of very fast going. The other type of application is offline. So for example, I have a case study, legal case study. I need my legal assistant to go off and analyze similar cases and come back in 3 days, give me a report. Now an agent, legal agent can go off and do this study for a couple hours and come back, right? So those are two very big categories of agents or agent applications that has different latency requirements. So typically for the real-time, extremely low latency requirement, you cannot afford to think. You have to basically react. And usually the customization, the go back goes back to customization, right? Usually the customization process is very interesting also, is you go small, big, and then small for real-time use cases. What does that mean is when you customize, a lot of time it's about your data. Is your data high quality? Keep in mind this kind of garbage in, garbage out, same apply here. If your data is not high quality, then your end model is not high quality. So, For you to test data quality and fix quality issue, you start with small model to see if the model is going the right trajectory. And then if it is going, not going, you go back to fix our data and then, you know, small model help you iterate fast. But that small model is not important. The trajectory of the model quality change is important. Once you like the trajectory and then you move to the biggest model and usually those are biggest MoE model, really hard to to get it running. So obviously that's where we will definitely help you. And use the clean data to tune the model, get the best specialized model to solve your real-time problem. But usually those largest model is not as fast, right? Because they're very big. They're trillion, trillions of parameters. Very big, but very good in terms of quality. And then you go small again, but you still find the biggest model you have tuned into small model because you need fast response. And then you launch the smallest model, you can get to high quality from that process and can go from there. So small, big, small, right? Now we move to offline asynchronous agents. And usually those agents are doing some deep research in certain kind of domain, whether it's about legal, whether it's about finance, whether it's about software engineering, or whether it's about something else, right? So, but usually it takes time to think like us human being, we're gonna, hey, do some homework, go hide in a cave, figure it out. So similarly, and those model requires the highest level of intelligence, highest level. So then the kind of slow thinking mode becomes extremely important. But not just that, those big models for offline research also requires a lot more context, a lot more context to make the right thinking process, like get the the right design of the flow. And, but at the same time, a lot of the time those models also need to get specialized in solving certain kind of problem really well. The legal process, the finance process, the process of working, building PowerPoint, the process of building an executive overview a pitch to investors, the process of doing customer service, those are all very, very different. So we have also seen a lot of Application Agentic developers, they start to customize offline agents where that just means they need to figure out how to tune with the thinking tokens. So, so that's kind of a very different classes of equation. I oversimplify here.
