Ep 50: Fireworks CEO Lin Qiao on Why There Won’t be a Single Model, Will Hyperscalers Win Inference & AI Use-cases with PMF

Lin Qiao is the co-founder and CEO of Fireworks, a company building fast and efficient inference for compound AI systems. She was also one of the creators of PyTorch over at Meta and has raised over $75 million from investors like Sequoia and Benchmark. On unsupervised learning, we had a fascinating, wide-ranging conversation. We hit on Lin's thoughts on the evolving open source ecosystem, what infrastructure is required to make agents work in 2025, and we also hit on the challenges of building an AI infrastructure company when so many changes are happening at the model layer. I think folks are really going to enjoy this conversation. Without further ado, Here's Lin. Well, Lin, thanks so much for coming on the podcast. Really appreciate it.

Thank you for having me.

Yeah, I've been looking forward to this one for a while, and I feel like it's probably a really interesting time to get to chat with you. Obviously, you guys have built just an incredible platform for developers and folks that are building in the AI space. And there's always— it's never a dull moment in the AI world, and I feel like there's always things changing. And, you know, one that I'd be really curious to get your take on is obviously I think everyone seems to be talking about test-time compute right now and O-1 and, you know, many models that seem that they will shortly follow. And I'm wondering, like, how does that change your kind of strategy or product that you offer at Fireworks, like these types of models?

Yeah, I think from the inception of Fireworks, we always had that in mind. As in the— well, like, what is Fireworks? Fireworks is a GenAI platform with laser focus on inference. And here the top-level goal is deliver the best quality, lowest latency, and lowest cost, right? So in the inference stack, and we can— I would love to dive into details there because inference is not a single model-as-a-service inference. It's not that simple. As you allude to test time inference, test time quality scaling law, the inference system we envision in the future is a complex inference system with logical reasoning, with access to a lot of hundreds of small expert models and so on. The level of problems we're trying to solve is not, hey, this is just API call, in, out, done. It's not really simple.

It sounds like you envision a world where where the user puts in a query and at Fireworks, you're doing the routing and figuring out what is the perfect, the best performance model for whatever that query is as a response.

Right. So I think let's take a big step back of what the problem we're trying to solve here. So there are a lot of limitations of models and its limitation comes from the models are not deterministic, it's probabilistic by nature. It's not desirable when you want to deliver factual results, always truthful result to your end user. Controlling that hallucination is extremely important. The second is a complex business problem. We have many customers using us. It has to assemble across multiple models and across multiple modalities to solve those problems. As we're talking right now, a lot of applications are to consumer, to consumer developers. We're not communicating texting each other. I'm processing audio information and visual information to deliver a good interactive experience, a conversation we're having right now. Similar to a lot of GenAI-based native applications, consumer, customer-facing, they also need to process across multiple modalities. And even within same modality, let's take LLMs, there are many different expert LLM models specializing in doing classification, summarization, Budget and chats, tool calling, and they're all slightly different from each other. So that's kind of single model is very limited if you want to solve a real-world problem. And then last but not least, single model is very limited in knowledge. It's knowledge limited by training data. Training data is finite, not infinite. So a lot of real-world information lives behind APIs. They are public APIs or even proprietary private APIs within enterprise that there's no way you can access to it without like working directly with enterprise. So across the board, we envision that the next barrier is how to go beyond single model as a service, right? So the world needs a notion called compound AI system, as in multiple models across different modalities along with various different APIs holding knowledges including databases, storage systems, knowledge base, need to work together to deliver the best AI results.

As you think about building these more complex compound AI systems, what are the tools you think developers will need to be able to effectively build these?

Maybe we can first start talking with like a polar opposite design points, right? So one design is called imperative where you have full control of what is the workflow and what are the inputs, outputs, everything you want to make it deterministic, right? So that's kind of one level of design. The other design So you basically design the house. Dictate that. The other design is called declarative. You define the what, the what problem this system should solve for you, but then you let the system figure out the house. So in general, in the industry, it's not unique to AI. In general, we have these two school of thoughts in approaching the system design. For example, in the database world, it's kind of full of those kind of examples. SQL is an example of declarative. Where your data engineers or analysts, they define very exactly what we want to retrieve out of the database. And database management system figure out the execution plan, the best, most efficient execution plan. And then the ETL process is declarative. And you basically define, oh, here's the source, and what are the processing logic, how you trigger the next step, and how do you do backfill, very imperative. And those are nothing wrong with either of those approaches. Those are different, just different. Approaches. And the file was based on our PyTorch experience. We want to deliver— ultimate, our design principle is we want to deliver the simplest user experience and hide all the nitty-gritty details and complexity in the backend as much as possible without hurting the speed of iteration. And it's a very fine balance to strike there. So we are leaning more towards a more declarative system with full debuggability and maintainability.

What are What are some of the examples maybe of where you're on the line and you're like, we could do it this way that would be more declarative or a different way? What are some of those trade-offs you have to, or you think about making?

Yeah, so for example, when we started, we started with the lowest level abstraction, which is single model as a service. Where today we provide hundreds of models across large language model, across audio models, transcription, translations, speech synthesis, and alignment across vision models where the source can be PDF images, screenshots, shots and so on. And embedding models, image generation models, we're adding video models. So those are kind of the lowest level of building blocks. And developers can just assemble what they want on top of that. But there's a lot to assemble, a lot of pieces to assemble. And the quality control is difficult because every week there are multiple models get released. And they constantly asking us, we want to try this model. What's your suggestion? Should we even try? Or then what is our production stability, right? And how to do version control and all this becomes a problem for them. But we start there, so we know how we can understand how the industry is evolving. But then we quickly realize there's a huge gap of usability, especially when it comes to enterprise, and there's a huge gap there and we want to fill in the gap.

When you're talking to folks, why aren't there more GenAI applications out there?

There are a lot of barriers, actually. One is there's no one model fits all. That's our observation. It's kind of the nature of the training process. The training process is a very opinionated process that you have to pick which subset of problem across thousands of problems in the world you care the most about. And you devote the biggest amount of resource and money to acquire data and make sure your data quality and the variety of data in that space is the best. And which solves a set of problems you don't care, and we're in the middle, right? So because of that, the end result of the model will be really, really, really good at doing things and then really bad at doing certain things, right? So that's just the nature of model training. We believe the future are the hundreds of small expert models. Because when you shrink the problem down to a narrow space, it's actually much easier for small model to thrive in pushing the quality. And as a matter of fact, in the open source world, this is really great for the open source community because with open source base models, it gives a lot of control for you to customize your model. There are a lot of model providers, they focus on post-training or fine-tuning and deliver specialized model giving back to open source community that's really, really good at solving certain problems very well. So we believe the future are hundreds of small models. I think that's kind of where we see the enterprise world is moving towards. They want to have more control. They want to have steerability.

In this world of like hundreds of small expert models, I could imagine one world in which there's so many people taking open source models and tweaking them in some ways. And so Fireworks just has a big library of them. And I'm an enterprise and I have a problem. And you're like, hey, here's a dozen really great models that we'll orchestrate together. I could also imagine there's probably some cases where you tell an enterprise, hey, you should really fine-tune for this use case. Or you might actually, I mean, maybe in some cases even pre-train a model to some extent. How do you think about where that plays long term? Like, do you, how do you think in 5 years, like, will enterprises be fine-tuning a lot of models or pre-training, or how does that evolve?

We deeply believe in customization, again, as I mentioned, but the process is, it's kind of, it's not, I would say it's not straightforward. Like, it's not like, like, oh, like 1 minute done, right? So it's longer than that. So we see a fine balance between fine-tuning and prompt engineering. There's a strong trade-off there, and that's where we are working on. Because we believe in customization, we're working on how to make customization extremely easy. So the balance is falling, right? So it's much immediate. You can see the result immediately. It's much more responsive, interactive using prompt engineering. This is just kind of— that's the nature of it. And we a lot of enterprise developers start with prompt engineering. So hey, is this model steerable towards the direction I want to do? And I can test it out quickly. But then we see thousands of lines of system prompts. At some point, if you want to continue steering it, you got stuck because you don't know where to change and it's going to all get lost. So solving a very complex system prompt management is a problem by So actually, that eventually is the key to how to make customization easy, is to solve that problem. And we are going to launch a product to solve that problem. Interesting. Yeah. So, and then we see, then at that phase, leave that aside, at that phase you have thousands of line system problems, then what do you do? And usually that's the time, it's the prime time for you to do use fine-tuning to absorb that system prompt into the model itself. Because by that time, you should already proven the model is steerable to solve your problem. You are able to let the model follow your instruction. Your instruction is pretty much the data you provide to the model. And then it would be a good candidate to do fine-tuning. And that already means you're probably evolving from pre-product-market fit into post-product-market fit product scaling. And once you absorb the long, lengthy system prompt into the model, the model can run faster also and cheaper and also with high quality. So we see that movement. It's a very organic movement, but still getting to the stage of managing, like even kind of before you absorb into fine-tuning, managing these long prompts, system prompts is still a very, very challenging thing. So yeah.

Do you think pre-training ever makes sense for enterprises?

Pre-training is kind of the whole entire industry is going to, hey, pre-training is being consolidated, right? So into hyperscaler, that pretty much That's just the reality. We do feel like there are— we do see enterprise, some enterprise do pre-train models because they have— this is kind of core to their business or for other reasons. But yeah, so, but the differentiation, the question we see a lot of enterprise asking themselves is, or even startup asking, what's the differentiation? Because pre-training is very expensive and the ROI has to justify you threw a lot of money and the human resource into doing this and if you can just do post-training on top of very strong base model, then ROI is much stronger and you are much more agile to test different ideas.

You sit in such an interesting vantage point because you see so many people building on top of Fireworks. What use cases for GenAI do you feel like have product market fit today?

I would say there are— I'll classify those use cases probably into different buckets. In summary, I think most of use case we've seen successful getting adoption is kind of human-in-the-loop automation, not human-out-of-loop automation yet. So my hypothesis is a GenAI system has to be, it has to be human-debuggable, understandable, maintainable, operable. If a human cannot evaluate it, cannot maintain it or put influence over it and operate that in production, it will be really, really hard to get adoption. So because of that nature, we see a lot of vibrant product offering from all different kind of assistant to human in different categories. We have seen people building assistant to doctors to make their scribing much easier. We've seen assistance to teachers and students and people want to learn foreign languages, educational. We have seen assistance to coding. Of course, that's a highly competitive space, but we have worked very closely, for example, with Cursor or Sourcegraph and many other great companies. We have seen assistance to medical assistance. Apparently, there's shortage of more than a million nurses in North America. There are a lot of patient need being paying attention to. There's a whole slew of consumer, prosumer, developer space. So that is one. The other is kind of more B2B, as in there are automation, like call center automation. But more like call center automation is a great example. You can build assistant to human agent to make them more productive and answer the questions better or replace human. We have seen a lot of success of building this assistant to help human agents, right? So that's another example. A lot of process of optimizing business logic, business workflow efficiency, and so on.

And then what about on the model side? Like what have you noticed in terms of the models that are actually getting used by companies?

We see a lot of convergence of variation of Llama models. I think that's a testimony to the quality of the model and it's a very strong base model, very, very good instruction following, and very good for tuning. So, and of course, Meta backing it is very important and see a lot of enterprise adoption there.

Obviously, I'm sure something your enterprise customers think a lot about, or, you know, is just like evals. And I'm sure, you know, you obviously have thought a lot about this, like, what's kind of the current state of evals? And, you know, how do you think about the right building blocks to give enterprises that might have very different use cases and as a result, very different ways of evaluating success.

Many enterprises do at the stage of eval is vibe-based, which is not surprising, right? So yeah, so they are, but they quickly realize, I mean, for the early stage, product development is kind of more vibe-based. Hey, I just want to get a sense of how this product is is feel like with different models and so on. And they quickly evolved to the stage that they are consciously building eval and they know this is kind of investment area. But for them to stay on top of the state of art, they have to be able to evaluate and also they have to be able to evaluate once they get into kind of more deeper prompt engineering or fine-tuning, like how to just evaluate quality is important. They cannot always go to A/B testing. A/B testing is the ultimate process to determine the product impact, but it's a longer cycle. Totally.

It is funny that we talked to some, I think Sourcegraph's customer of yours said they don't even bother with evals. They just release it out, A/B test it, and they'll know pretty quickly from their developers which model is better.

Yeah. But the evals are very important because, and from our customer's point of view, they realize investing in generating good eval dataset and adding more and more to it can give them a very clear view what matters, what doesn't matter. Because in this process of catching up with the constant moving train of those small expert models, the model is not just every week there's a better model like beating leaderboard. The model is also getting more and more specialized. And they will also— many of our customers, they start from their product is open-ended. They don't have a strong opinion what kind of question they should expect from their user. Today they start to get more and more clear opinion. And they harden their product design into open-ended product to specialized product in different product features. And then from those product features, they want to have specialized model to solve those problem. So it's a natural evolution of product development.

And I guess like, you know, maybe just talking about some of the stuff you guys have built, obviously, you know, you're taking the great models that exist in the open source world and then you also build your own models, right? You built F1. Maybe talk a little bit about why, you know, why you decided to do that.

So we released F1 under API and that's a model API. Basically how you use F1 is as if you are using a new model. But F1 is really a complex logical reasoning inference system. So which underneath, we haven't talked too much about what's underneath F1. It's kind of, here is a sneak peek of what's underneath. So underneath there are actually multiple models and also there is logical reasoning steps that we implement in our system. So then actually building a system is very complex. It's not like just regular single model as a service inference. And also we have to solve a lot of quality-related problems because now you let the model talk with itself, let the model talk with each other, and what kind of information they communicate to drive quality and they are sensitive to that, right? And how to do quality control. In this complex system. I think the complexity is actually higher than building a database management system in that sense. And yeah, and then of course, because there are multiple steps involved and then your overall inference latency and cost becomes more interesting. So that's our bread and butter because we're a team really specializing in optimizing for that. And then we view our When we start a company, we build a lower level of single model as a service. That's a building block towards this complex logic reasoning stack.

Interesting. And then obviously I think a big part of it is function calling as well, right? And what are the barriers, I guess, to getting function calling even better today? It feels like we're still kind of in the nascent stages of that in many ways.

So function calling, many people recognize that actually we see a lot of— after we launched F1, we have We have a waiting list for people to get in and we talk with many of them. Apparently their use case, the majority of use case is actually building agents and they need function calling. So what is function calling? Function calling basically is an extension point for this model to call into other tools to enhance the quality of the answer. But function calling is not just about calling into one tool, it's actually very complex if you think about it. Usually people use those function calling model in the multi-turn chat context, right? The model, it needs to be able to hold a long context of what the customer's— and then use that context to influence which is the best tool to call. And oftentimes they also need to call into multiple tools, up to like, could be up to hundreds of tools, tool selections. Also, they need to— it's not just call one tool at a time. Oftentimes they also need to call multiple tools in parallel and in sequential. It's a complex coordination plan in one shot. Our function calling is able to do parallel sequential complex planning and then orchestrate and execute on that plan. Of course, then And the position is important. Like if I ask the model to like do, like generate chart of a stock price of top 3 cloud providers, underneath it will return the answer quickly and then show you a chart. But underneath there's a search to find out top 3, there's 3 parallel function calls to get the stock price and then one call into the chart to get the chart back. So this is a very simple example, right? But it demonstrates the complexity of orchestration and the model need to have very strong capability of understanding if you plug in your tool, when to call this tool and how to drive that position is very important. So that makes the tuning process very complicated. But we have been investing in this space for now about a year. So yeah, so we start to see when we start to release the first function calling model, I think we're ahead of the adoption curve. The use case hasn't fully taken off.

And how do you think about why decide to build that versus, I'm sure if you waited at some point there'd be a good open source model with some of those capabilities. How do you figure out when to build things in-house on the model side?

Yeah, so we heavily build on top of open source community. That's absolutely is the case. And we are betting on that because we believe the hundreds of small expert models will come from the open source community. So that is directly aligned with our vision. But at the same time, we're also heavily investing in the Compound AI system where it's all about composing those hundreds of small expert models to solve a complex business task in a very easy way. And the critical part of this composability lies in this this layer of orchestration, right? So the model who is intelligent enough to be able to call into different tools. And here you can think about the general way of thinking about tool is each individual small expert model is also a tool in addition to this particular large language model. So then it becomes a critical ingredients of tying everything together. And we will invest in these strategic strategically in critical area that we cannot just wait to see what happens.

Where do like the reasoning models end up fitting in? I mean, obviously at least the ones we've seen today are, you know, I think of them as bulky and generally good at reasoning of all things. Do we end up with like small expert, you know, like test-time compute-intensive models as well? Or like, do you imagine you will only kind of route to those in very specific instances? Or like, as you think about these compound systems, I'm just curious, like, where these O-1-like models will fit into that?

I think even for reasoning, there are different paths to solve reasoning problems, and there will be different models specializing in different paths. One path is to have a very strong base model itself to do self-inspection, and it looks like a lot of techniques already being discussed with chain of thought, tree of thoughts, and backtracking and all these different technologies. So that's just one. And there will be a new set of models. They can do logical reasoning, not in the prompt space. And this is something I'm very excited about, but in latent space. Because you can think about when we think, we don't have to talk in words, like our thinking process. We can, but we don't have to. So I'm pretty sure when we process a problem in our head, we're probably thinking in a different space. Right. So similar to the model as well. So there are a lot of active research happening to how to make the thinking process much more efficient and much more native to the process. So I'm very excited about that kind of research. There will be other flavor of logical reasoning. So we are not— we don't want to be very opinionated about which one is going to win. Instead, we're going to integrate all these different flavors into our logical reasoning system.

I love that. I guess outside of what you're doing at Fireworks, what do you feel like are some of the other big unsolved problems in AI infrastructure today?

I think we have seen a lot of movement into building agentic workflows.

Yeah.

And again, I think we're still at the early stage. Stage of figuring out what is the right user experience. What is the abstraction? The whole industry, right? What is the right abstraction? What we should hide behind the system? What we should expose to developer? We're still kind of in very early stage and there it's, I think it's very experimental right now. But where the abstraction lies will determine the complexity of the infrastructure hiding behind the abstraction. But I think we now start to form opinion around that. In a month or two, we're going to GA F1. So, okay, I was thinking processes building F1 is one is F1 is our own exercise to understand the system abstraction and the complexity of building logical reasoning engine. From there, when we GA, we want to expose developer-facing plugins and how developers can build their own F1s. So, yeah. So, I'm very excited about that direction.

That's super cool because obviously you learn the tools and the abstractions that are necessary by building it yourself. And then, you know, you basically allow developers to recreate that. I love that as an approach. I guess, you know, a lot of it feels like on the hardware side, I mean, obviously everyone's using NVIDIA, but there's a lot of competitors have been popping up. And I'm curious, like, what you make of that space and the viability of some of those efforts. I think you guys support AMD chips for inference on the platform. Like, you know, when does that make sense for developers and how do you see that kind of evolving?

In general, we see a kind of scarcity in the developer space that they understand hardware, low-level hardware optimization. And we also see a very big change in the kind of hardware development cadence. Before, it's kind of every 3 years there's new hardware SKU. Now every 1 year there's a new hardware SKU from each hardware vendors. And then the whole hardware space is moving very fast and how to kind of access to the best hardware. And by best, I really mean there's, again, there's no one size fits all. So it really depends on your workload pattern. Even for accessing the same model, there's no best hardware for one model even because your workload distribution is going to determine where is your bottleneck, biggest bottleneck, and different hardware SKU is going to be the best for solving kind of removing certain bottleneck the best. So we even at the Fireworks layer, we absorb the burden of integrating and determining which hardware is best for what kind of workload. Even for one workload, when you have mixed access pattern, we can route to different hardware. So yeah, so we want to really kind of alleviate the concerns and burden of developers, they should focus on building product and we will, we will kind of take the complexity of managing and optimizing for the hardware.

Yeah, it seems like you're obviously super focused on these compound systems. You know, there's obviously always going to be some set of folks that like, you know, just want to call a Llama model as it is and run it. In that world, you know, it feels like for a while there was really a lot of competition on the inference side. Like a new model, an open source model would come out and everyone would be offering their prices and it would get lower Or in that game, do you think that eventually the hyperscalers are best suited to win just kind of straight inference on an open source model? Or how does that kind of develop?

That's a very complex question to answer. I think all hyperscalers, they want to be the Apple, right? So they want to build the iPhone. They want to kind of build a vertically integrated stack. Because they can. But again, the direction we're heading towards is we feel like the future lies in hundreds of small expert models and we want to harvest that energy and build a company, a system that leverages the best models to solve a complex problem. So in my view, I feel like the hyperscalers, what's their biggest benefit? Their biggest benefit is to solve the problem requires a lot of resources in terms of money and human. Those examples are like building data centers, acquiring powers, and install tons and tons of machines, deploy them, light them up, and turn up large-scale storage and compute. Cloud provider please solve those problems. Massive problem at scale. And we specialize in solving problems that requires a lot of the combination of engineering craftsmanship and deep research. And then we can deploy at scale. So I think inference is like that. I work at Meta. It's not like Meta has a 1,000-people inference team, right? Because once you have a system that can be horizontally scalable with high production quality, then the rest is just let it scale. That's the beauty of designing a highly scalable system. So in that sense, I think this is squarely in our strength and expertise. And also, again, Inference system is not simple. It is actually going to— we're evolving to getting to this compound logical reasoning inference system where there's a lot of complexity in building that. And that complexity is not by throwing people and money to solve. So yeah.

Yeah. I guess with these smaller expert models, one thing I wonder is, as these models get smaller, it feels like there's a big trend to running them locally too. And I'm curious how you see that playing out over the next few years and and if that becomes part of the Fireworks platform as well.

Yeah. I've seen a lot of argument for running models locally for two reasons. And also we should talk about what does local even mean, right? One reason is cost saving because yeah, you need to pay GPU on cloud and running on desktop, you don't need to pay. And yeah, so the other is privacy, right? So I have different thoughts on that. I think that makes sense to offload compute from cloud to desktop. I think many applications like Zoom, for example, that save a lot of cost for them. But offload compute from cloud to mobile is a different story because mobile phone has a very— it's getting beefier and beefier, but it has very limited power. Right. So at Meta, for the application, a lot of application metrics are closely monitored, including cold start time, power consumption, all these affect the application rating and user adoption experiences. So because of that, the model that can practically deploy to mobile is tiny, like one 1B, 10B, and very, very small. And those models have limited capability. So I think between mobile and cloud, the difference is extremely big. And offload to desktop, I think for many consumer-facing applications makes sense. But also there's another argument of, hey, it's more private. And privacy is interesting. Interesting, right? Because we have a lot of personal data is already on the cloud. Most of our personal data is not on local disk anymore. So then kind of, hey, does that even make sense for privacy? Yeah. So I think that's a separate concern.

Yeah. Super interesting. I guess on the open source side, obviously you used to work at Meta. It feels like obviously they've been providing a huge service to the ecosystem by training these larger and larger models. How far do you think they go on continuing to scale spend on open source models before at some point— I mean, I guess everyone's asking this for all the players, not just the open source players, but how do you think about the extent to which it makes sense to keep pushing on pre-training spend?

I think that's a good question to Zach. No, I mean, jokes aside, we work very closely with the Meta team. Of course, we came from Meta. And we actually co-design. They're not just open source Llama models, they're also building a standard called Llama Stack. And Llama Stack, the intention is kind of standardized the tools stack around Llama models. And we co-design with them because we have a lot of information coming from our customer that can help them. Make design choices. I think that's ambition from Meta is to build this Android world where everything is nicely standardized and you plug into different components and make it very easy to adopt. From my vintage point of view, my understanding is Llama 4 is coming soon. There would be continuous investment from Meta that train. I haven't seen that stopped. Of course, if it's a general question, when would model providers stop or invest less on pre-training is when there's not much ROI. So why there's not much ROI is because we're hitting a wall of data. We're running out of data. Everyone crawl the same internet data. We have exhausted synthetic generated data. We have exhausted a combination of multimedia and text data, then kind of there's less return. Before we hit that wall, I think there will be continuous investment.

When do you think we'll hit that wall?

I think we're— it's definitely getting longer and longer, like from like OpenAI's release point of view, right? So I, and I definitely see kind of a lot of investment start to shift from pre-training to post-training, from post-training to inference. So I think we're already hitting a soft wall, not a hard wall. So yeah, so I think in general, as an industry, kind of where the ROI is kind of transitioning from pre-training to post-training to inference.

As you think about all the sets of tools that are required to make an agent successful, I mean, there's one world where as Fireworks, you could be a one-stop shop and build all of them. There's others where you could partner and you're like, look, we're not going to build this part of the thing. How do you think about that?

Yeah, so we always are very compatible with the imperative agentic tools, right? I think most notably is LangChain.

Yeah.

And we have been a very strong partner with them because we are not going to Lin Qiao is doing great work in what they are building. Huge community following and adoption. Really awesome. And we just want to simplify the next level above single model as a service by composing wherever it makes sense, multiple models to solve the problem much better. And we are fully kind of, we will we will not change our position, we'll plug into LangChain. For example, as one example, there's also of course many other examples in the community.

Yeah, makes a ton of sense. As you think about kind of like the competitive landscape today, I mean, I imagine when people think of Fireworks, they also probably lump you in with like Together AI. And, you know, I know Databricks talks a lot about these like compound AI systems too. Like, how do you think about what like differentiates Fireworks today? And then, you know, 5 years from now, what will ultimately determine who captures this really interesting space?

Yeah, we are probably thinking— so compound AI actually is the term— we didn't coin that term, right? So it's kind of coined by Berkeley. And we know that Databricks is also looking into that space. So I think it's great. It's a new category that we're defining, right? So definitely multiple players will be in the same space. And we are very happy that Databricks also think about that problem. So I think that space is very meaningful because it's a very complex space for application developer, product developer, engineers to get into. And there will be a rich set of tools that emerge in that space to make development much more efficient. And today we're not there yet. So we are determined to kind of be a key player in that space. Other than that, we are not in the space of being a GPU cloud, for example, and provide access to cheap GPUs. That's not our play at all. So we actually, we build on top of GPU cloud to offer a complex inference stack.

I guess one thing that's really interesting about the company is I guess you started Fireworks like what, a few months before ChatGPT? And so I'm curious, like, you know, what would— like, as you think about the original vision for the company and then, you know, where you've come through today with this like GenAI craziness, like how much of it was the same, you know, sitting there in September '22? How much of it have you had to like change as the world has just changed so much in the last few Yeah, it's actually, yeah, this is a very interesting question.

When I started Fireworks, there was actually active debate. Is AI here or not? Because it's pre-awareness of foundation models and it's kind of, oh, there's so many fragmented AI application and the data is not there. AI is not there. But we clearly see that AI is coming. So usually hyperscalers are 3 to 5 years ahead of the whole entire industry and Meta is massively AI powered. And clearly the industry is through PyTorch adoption is clearly see that the wave is coming. So that's the timing about firewall. Works. And the GenAI did skew, ChatGPT heavily skewed the adoption curve of AI. And it's a special kind of AI. So it's very interesting. GenAI is a different beast, not because it's GenAI, it's magical, because it's fundamentally changed the accessibility. So pre-GenAI, when it's like traditional machine learning or early deep learning, Any company investing in AI particularly have to first hire a machine learning team because they have to train model from scratch. There's no other alternative. There's no other model that's pre-trained you can build on top of. And that means this team has to curate, spend a lot of time curating data. And then the training process, 'cause training from scratch, it kind of take a long time. So they have to have the resource to hire those very scarce talent and put money into training.

And was the original Fireworks product built for those people? Or like, because that was originally what it felt like was going to be needed to get adoption.

Yes. Yes. That's what I— when we think about when we started, we think about that cohort. And then with GenAI, it makes it much— it completely changed the landscape, right? You don't have to hire a big team to carry data. Because it has the beauty of GenAI is it creates a foundation model that absorbs a majority of the knowledge and then you just build on top of it as is or you tune, right? You use thousands of samples, you tune. So then you either don't need to have any machine learning team, you just need to have that application product team to build on top of that directly or you have a small machine learning team, right? So this is a massive, massive unblocking of accessibility of that technology. And that's why we see the adoption curve take off like crazy because of that accessibility is fundamentally different before and after. So because of that, we laser focus on it because the pull is much stronger in that direction. And the other side effect is all these models, all GenAI models are PyTorch models. And we wrote PyTorch code for years and we operate PyTorch, large complex PyTorch model in production with high volume traffic for years. And that's what we are really good at doing.

Yeah, very well situated for that. You mentioned when you're at a place like Meta, you get kind of a window into the future before it's obvious. Was there something you saw when you were at Meta where you're like, God, this is just gonna be clearly so big shortly thereafter?

I think it's more the kind of AI is gonna be very big wave. That's clear. So I worked at Meta for 7 years. It's an interesting time. When I joined Meta, they were going through the mobile-first transition in the kind of tail end. And the mobile-first, and then the industry is also going through mobile-first, right? So from desktop to mobile. And mobile-first enabled consumers to be able to access application everywhere. And that drove our adoption and engagement. And because of that, it produced a lot of data. And a lot of data is powering AI, right? So that's— then the whole entire industry is also following the mobile-first, drive a lot of data, and data power AI. So that trend is very clear.

What AI research do you pay attention to, maybe even outside of like the core day-to-day of Fireworks?

I usually pay attention to two kinds of research. One is is model-system co-design kind of research. Because there is an interesting organizational division, as in there's researchers, they focus on quality only. There's a system builder like us, we focus on deliver the best quality with low latency and low cost, right? So we solve system problems. But oftentimes, And the best ROI is think about that together. And that's how we operate at Meta is kind of the research team and kind of infra team sit very closely to discuss trade-offs and the co-design. So there are a lot of research in the co-design space. I think it's going to be very appealing to make— to kind of find the best design point across quality, latency, and cost as a three-dimensional optimization. The other kind of research is kind of fundamentally different, disruptive kind of research, as in Transformer as a technology is overdue for disruption. Yeah. Right? So where are the next generation of Transformers that's going to change how we train model, change how we do inference, and so on? So that's very interesting. And also in the agentic world, how different agents is gonna communicate with each other. As I mentioned, thinking the latent space, that kind of very new kind of thinking is very, very interesting.

One challenge that I imagine in building an AI infrastructure company is that obviously the pace of change is so fast, both like the improvement in models plus just like how enterprises are thinking about and actually using these things. And I can imagine one school of thought would be, it's changing so fast, we'll just quickly update what our core tools are because of how fast it's changing. And at some point it will settle down and then we can build the standard set of tooling. How do you think about that tension of building for however folks are doing things today without knowing exactly, maybe in 2, 3 years, models have all these other different capabilities and we're designing systems in all these other different ways?

Yeah, that's a really Good question. That's all we think about all the time, right? So we don't want to keep chasing what's coming because chasing is exhausting. We always want to stay on top of the curve. So I think the model capabilities are definitely evolving, has evolved from 2 years ago, especially if we just look at OpenAI's model capability, from GPT-3 to now today, there are differences. But fundamentally, a few trends that doesn't change. One is, again, anchor back to our vision, is we believe the direction is specialization and customization. That doesn't change how the model core capability evolves. Because we just don't believe that one size fits all different workloads with proprietary data and so on. We believe there's a much better solution if you can customize, if you can steer, if you have control, the destiny here. That's why we build our stack to heavily enable that in an easy way. Specifically, we offer on top of our inference engine, which is one size fits all, we have optimizer, file optimizer, that's one size fits one. We take inference workload as input and also your customization objective as input and spit out inference deployment configuration along with potentially a model, adjusted model for you to you have the control whether you want to deploy that or not. So this is just a process of we kind of close the loop and make it super easy to customize. So I don't think that's going to change in the future.

Totally. Just what you're customizing may change, but like still the way you will go about it, I think that makes a tremendous amount of sense. Well, we always like to end our conversations with a quickfire round where we get your take on a standard set of questions. And so maybe to start, art would love, just one thing you think is overhyped in the AI world right now and one thing you think is underhyped.

I think overhyped is the perception GenAI is magical. It's a recipe of all problems. We should just ask it any questions, it's kind of, it will have the right answer. I think, I think it's going through the correction time right now. And again, that's where kind of we believe, like there's no magical one model that solves all the problems the way or in the correct way.

What's one thing you've changed your mind on in the AI world in the last year?

I think my hypothesis in building this company, how this GenAI technology can adapt, adopted in our go-to-market strategy, in my mind is always in a sequential way. Startups are GenAI native, so they will be on the frontier of adoption and And then we will work with digital native. They have very strong engineering resource and always kind of be more tech forward. And then traditional enterprise because they have more other kind of problem to solve. But now we're working with all of them at the same time. And to me, it's a little bit crazy. Of course, the use case are not all the same, but the adoption curve is happening all at the same time. It's very different from what I was imagining.

Totally. I mean, on the application side, it feels like the normal rules of how long it takes to land one big enterprise and then land a few after, it's all out the window, right? Some of these application companies are scaling so fast in the large enterprise. I think there's just a tremendous appetite for it.

That's right. Yeah. That's right. So that's why I feel like we're in this revolution way, right? So a lot of things will be done differently, not just kind of the application we build will be different, technology adoption curve will be different, but also how to think about go-to-market will be different. The sales cycle is much shorter, and people are open to think the procurement process differently. It's a very interesting ripple effect from this massive transformation.

Do you feel like— is there any difference in what you have to build for enterprise versus the startup folks, or is it actually like the needs pretty similar?

My observation is startups, they are typically like they want to get access to, and when we talk about abstraction, they want to get access to the low-level abstraction because they want to think a lot. So, assemble a lot of things. For traditional enterprise, I'm not saying this in an exhaustive way, but typically, they want to have a higher-level abstraction. If they don't need to solve for low-level details, you don't need to pay attention to, rather not to pay attention to. So, even at Meta, we have those different abstraction layers for different kinds of teams because they want to pick and choose. So, usually two-level abstraction will suffice. They were happy with at least have two choices. We see a little bit of those.

Oh, must be interesting to build both at the same time as you're rapidly scaling.

Right. We need a low-level abstraction for ourselves anyway. To us, it's not additional overhead, but we see the adoption will happen at different layers.

Yeah. I know it's probably— I won't ask you to pick a favorite, but what is one favorite application that's built with Lin Qiao that works today?

This is a tricky question. I would say Cursor, which— but we see a lot of similar company in productivity space, not just Cursor, but also Sourcegraph, Zed, Cognition, Factory. They're all very forward-thinking company. I think in that space is still early, but that space we're very excited about the whole entire space.

Do you think someone will spend $100 billion on training a model in the next few years?

Ooh, maybe. I think it only makes sense if they are training using a fundamentally different model architecture. Yeah, it's so disruptive, it's worth that investment. Yeah.

You kind of already mentioned the coding space and some of those startups, but any other like AI startup or space that you're really excited about outside of Fireworks right Again, I think the agentic world hasn't been fully figured out yet. Have you seen any applications do that well?

We have seen early applications. For example, the digital SDRs, the digital marketing agents. And yeah, so we have seen kind of good startups getting great adoption there. But I do think in the future there will be a lot more complexity in that space. And we're still early.

Well, this has been a fascinating conversation. I'm sure folks will want to pull on all sorts of threads. So I want to leave the last word to you. Where can folks go to learn more about you, about what you're building at Fireworks? The floor is yours.

Yeah, so yeah, definitely. So we offer for the benefit of our developer community, we offer a self-serve platform. Very simple. Just go to fireworks.ai and then you can get access to our playground and hundreds of model capabilities we have. Yeah. And feel free to reach out to me also. Connect me on LinkedIn and love to hear your use case, your challenge, and pain points.

Amazing. Well, thanks so much. Seriously, this was a fascinating conversation.