Why There Won’t be One Model, Will Hyperscalers Win Inference & AI Use-cases with PMF

Lin Qiao is the co-founder and CEO of Fireworks, a company building fast and efficient inference for compound AI systems. She was also one of the creators of PyTorch over at Meta and has raised over $75 million from investors like Sequoia and Benchmark. On unsupervised learning, we had a fascinating, wide-ranging conversation. We hit on Lin's thoughts on the evolving open source ecosystem, what infrastructure is required to make agents work in 2025, and we also hit on the challenges of building an AI infrastructure company when so many changes are happening at the model layer. I think folks are really going to enjoy this conversation. Without further ado, Here's Lin. Well, Lin, thanks so much for coming on the podcast. Really appreciate it.

Thank you for having me.

Yeah, I've been looking forward to this one for a while. I feel like it's probably a really interesting time to get to chat with you. Obviously, I feel like you guys have built just an incredible platform for developers and folks that are building in the AI space. And there's always— it's never a dull moment in the AI world. And I feel like there's always things changing. And, you know, one that I'd be really curious to get your take on is obviously I think everyone seems to be talking about test-time compute right now and O-1 and, you know, many models that seem that they will shortly follow. And I'm wondering, like, how does that change your kind of strategy or product that you offer at Fireworks, like these types of models?

Yeah, I think from the inception of Fireworks, we always had that in mind. I think the— well, like, what is Fireworks? Fireworks is a GenAI platform with laser focus on inference. And here the top-level goal is deliver the best quality, lowest latency, lowest cost, right? So in the inference stack, and we can— I'd love to dive into details there because inference is not a single model-as-a-service inference. It's not that simple. As I just kind of allude to test time inference, test time quality scaling law, the inference system we envision in the future is a complex inference system with logical reasoning, with access to a lot of hundreds of smart expert models and so on. So the level of problems we're trying to solve is not, hey, this is just API call, in, out, done. It's not really simple.

And so it sounds like you kind of envision a world where where the user puts in a query and at Fireworks, you're doing the routing of figuring out what is the perfect, the best performance model for whatever that query is as a response.

Right. So I think let's take a big step back of what the problem we're trying to solve here. So there are a lot of limitations of models and its limitation comes from the models are not deterministic, it's probabilistic by nature. It's not desirable when you want to deliver factual results, always truthful result to your end user. Controlling that hallucination is extremely important. The second is a complex business problem we have many customers using us. It has to assemble across multiple models, across multiple modalities to solve those problems. As we're talking right now, a lot of applications are two-consumer, two-person developers. We're not communicating texting each other. I'm processing audio information and visual information to to deliver a good interactive experience, a conversation we're having right now, similar to a lot of GenAI-based native applications, consumer, customer-facing, they also need to process across multiple modalities. And even within same modality, let's take LLMs, there are many different expert LLM models specializing in doing classification, summarization, multi-turn chats, tool calling, and they're all slightly different from each other. Yeah. That's kind of single model is very limited if you want to solve a real-world problem. And then last but not least, single model is very limited in knowledge. Its knowledge is limited by its training data. Training data is finite, not infinite. A lot of real information, real-world information lives behind APIs. They are public APIs or even proprietary private APIs within enterprise that there's no way you can access to it without working directly with enterprise. Across the board, we envision that the next The next barrier is how to go beyond single model as a service, right? So the world needs a notion called compound AI system, as in multiple models across different modalities along with various different APIs holding knowledges including databases, storage systems, knowledge base need to work together to deliver the best AI results. Yeah.

When you think about building these more complex compound AI systems, what are the tools you think developers will need to be able to effectively build these?

Maybe we can first dive talking with like a polar opposite design points. One design is called imperative where you have full control of what is the workflow and what are the inputs, outputs, everything you want to make it deterministic. That's one level of design. The other design, so you basically design the house. Dictate that. The other design is called declarative. You define the what. What problem this system should solve for you, but then you let the system figure out the hows. In general, in the industry, it's not unique to AI. In general, we have these two schools of thought in approaching the system design. For example, in the database world, they're full of those kind of examples. SQL is an example of declarative where your data engineers or analysts, they define exactly what we want to retrieve out of the database and database This managed system figure out the execution plan, the best, most efficient execution plan. And then the ETL process is declarative. And you basically define, oh, here's the source and what are the processing logic, how you trigger the next step, and how do you do backfill, very imperative. And those are nothing wrong with either of those approaches. Those are different, just different approaches. And the file was based on our— PyTorch experience, we want to deliver ultimate— our design principles, we want to deliver the simplest user experience and hide all the nitty-gritty details and complexity in the backend as much as possible without hurting the speed of iteration. It is a very fine balance to strike there. We are leaning more towards a more declarative system with full debuggability and maintainability.

What are some examples maybe of where you're on the line and you're like, we could do 'Do it this way that would be more declarative or a different way.' What are some of those trade-offs you have to, or you think about making?

Yeah, so for example, when we started, we started with the lowest level abstraction, which is single model as a service. Yeah. Where today we provide hundreds of models across large language model, across audio models, transcription, translations, speech synthesis, and alignment, across vision models where the source can be PDF images, screenshots, and so on, and embedding models, image generation models, We're adding video models. Those are the lowest level of building blocks. Developers can just assemble what they want on top of that, but there's a lot to assemble. A lot of pieces to assemble and the quality control is difficult because every week there are multiple models get released. They constantly ask us, we want to try this model, what's your suggestion? Should we even try? Then what is the production stability? How to do version control? All this becomes a problem for them. But we start there, so we know how we can understand how the industry is evolving. But then we quickly realize there's a huge gap of usability, especially when it comes to enterprise, and there's a huge gap there and we want to fill in the gap.

When you're talking to folks, why aren't there more GenAI applications out there?

I think there are a lot of barriers actually. One is there's no one model. Fits all. That's our observation. It's kind of the nature of the training process. The training process is a very opinionated process that you have to pick which subset of problem across thousands of problems in the world you care the most about, and you devote the biggest amount of resource and money to acquire data and make sure your data quality and the variety of data in that space is the best. And which subset of problem you don't care, and we're in the middle, right? So because of that, the end result of the model will be really, really, really good at doing things and then really bad at doing certain things. So that's just the nature of model training. We believe the future are the hundreds of small expert models. Because when you shrink the problem down to a narrower space, it's inevitably much easier for small models to thrive in pushing the quality. And as a matter of fact, in the open source world, this is really great for the open source community because with open source base models, it gives a lot of control for you to customize your model. There are a lot of model providers, they focus on post-training or fine-tuning and deliver specialized model, giving back to open source community that's really, really good at solving certain problems very well. So we believe the future are hundreds of small models. I think that's kind of where we see the enterprise world is moving towards. They want to have more control. They want to have steerability.

In this world of like hundreds of small expert models, I could imagine one world in which, you know, there's so many people taking open source models and tweaking them in some ways. And so Fireworks just has a big library of them and I'm an enterprise and I have a problem and you know, you're like, hey, here's a dozen really great models that we'll orchestrate together. I could also imagine there's probably some cases where you tell an enterprise, hey, you should really fine-tune for this use case. Or you might actually, I mean, maybe in some cases even pre-trained a model to some extent How do you think about where that plays long term? Like, do you— how do you think in 5 years, like, will enterprises be fine-tuning a lot of models or pre-training, or how does that evolve?

We deeply believe in customization, again, as I mentioned, but the process is— it's kind of— it's not— I would say it's not straightforward. Like, it's not like, oh, like, 1 minute, done, right? So it's longer than that. So we see a fine balance between fine-tuning and prompt engineering. There's a strong trade-off there, and that's where we are working on. Because we believe in customization, we're working on how to make customization extremely easy. The balance is falling, right? It's much immediate. You can see the result immediately. It's much more responsive, interactive using prompt engineering. This is kind of— that's the nature of it. And we see a lot of enterprise developers start with prompt engineering, right? So, hey, is this model steerable towards the direction I want to I just do and I can test it out quickly. But then we see thousands of lines of system prompts. At some point, if you want to continue to steer it, you got stuck because you don't know where to change and it's going to all get lost. Solving a very complex system prompt management is a problem by itself. Actually, that eventually is the key to how to make customization easy. Is to solve that problem. And we are going to launch a product to solve that problem.

Interesting.

Yeah. So, and then we see, then at that phase, leave that aside, at that phase you have thousands of lines of system prompt, then what do you do? And usually that's the time, it's the prime time for you to use fine-tuning to absorb that system prompt into the model itself. Because by that time you should already proven the model is durable. To solve your problem, you are able to let the model follow your instruction. Your instruction is pretty much the data you provide to the model. And then, you know, it will be a good candidate to do fine-tuning. And that already means you're probably evolving from pre-product-market fit into post-product-market fit product scaling. And if once you absorb the long, lengthy system prompt into the model, the model can run faster also and cheaper and also with high quality. So we see that movement. It's a very organic movement, but still Getting to stage of managing, even before you absorb into fine-tuning, managing these large prompt systems is still a very challenging thing.

Do you think pre-training ever makes sense for enterprises?

Pre-training is kind of the whole entire industry is going to, hey, pre-training is being consolidated into hyperscaler. That's pretty much the reality. We do feel like We do see enterprise, some enterprise do pre-training models because they have, this is kind of core to their business or for other reasons. But yeah, so, but the differentiation, the question we see a lot of enterprise asking themselves is, or even startup asking, what's the differentiation? Because pre-training is very expensive and the ROI has to justify you threw a lot of money and the human resource into doing this. And if you can just do post-training on top of that, very strong base model, then ROI is much stronger and you are much more agile to test different ideas.

You sit in such an interesting vantage point because you see so many people building on top of Fireworks. What use cases for GenAI do you feel like have product market fit today?

I would say there are many. I'll classify those use cases probably into different buckets. But in summary, I think most of use cases we've seen successful getting adoption is kind of human in the loop. Automation, not human out-of-loop automation yet. So my hypothesis is a GenAI system has to be, it has to be human debuggable, understandable, maintainable, operable. If a human cannot evaluate it, cannot maintain it or put influence over it and operate that in production, it will be really, really hard to get adoption. So because of that nature, we see a lot of vibrant product offering from all different kind of assistant to human in different categories. We have seen people building assistant to doctors to make their scribing much easier. I've seen assistant to teachers and students and people want to learn foreign languages, educational. We have seen assistant to coding. Of course, that's a highly competitive space, but we have worked very closely For example, with Cursor or Sourcegraph and many other great companies, we have seen assistant to medical assistant. Apparently there's shortage of more than a million nurses in North America and there are a lot of patient need being paying attention to. That is a whole slew of to consumer, prosumer developer space. So that is one. The other is kind of more B2B, as in there are automation like call center automation, but more like call center automation is a great example. It has, you can build assistant to human agent to make them more productive and answer the questions better or replace human. We have seen a lot of success of building this assistant to help human agents, right? So that's another example. A lot of process of optimizing business logic, business workflow efficiency and so on.

And then what about on the model side? Like what have you noticed in terms of the models that are actually getting used by companies?

We see a lot of convergence of variation of Llama models. I think that's a testimony to the quality of the model and it's a very strong base model, very, very good instruction following and very good for tuning. So, and of course Meta backing it is very important and see a lot of enterprise adoption there.

Obviously, I'm sure something your enterprise customers think a lot about, or, you know, is just like evals. And I'm sure, you know, you obviously have thought a lot about this, like, what's kind of the current state of evals? And, you know, how do you think about the right building blocks to give, you know, enterprises that might have very different use cases, and as a result, very different ways of evaluating success?

Many enterprise do at the stage of eval is vibe-based. Which is not surprising, right? So yeah, so they are, but they quickly realize, I mean, for the early stage, product development is kind of more by basically, I just want to get a sense of how this product is, like, feel like with different models and so on, right? And they quickly evolved to the stage that they are consciously building Yvonne and they know this is kind of investment area. But for them to stay on top of the state of art, they have to be able to evaluate. And also they have to be able to evaluate once they get into more deeper prompt engineering or fine-tuning, how to just evaluate quality is important. They cannot always go to A/B testing. A/B testing is ultimate process to determine the product impact, but it's longer cycle. Totally.

It is funny. We talked to some, I think, Sourcegraph's customer of yours said, They don't even bother with evals. They just release it out, A/B test it, and they'll know pretty quickly from their developers which model is better.

Yeah. So, but the evals are very important because, and they, from our customers' point of view, they realize investing in generating good eval dataset and adding more and more to it can give them a very clear view what matters, what doesn't matter. Because in this process of catching up with the constant moving train of those small expert models, the model is not just every week there's a better model beating leaderboard. The model is also getting more and more specialized. And they will also— many of our customers, they start from their product is open-ended. They don't have a strong opinion what kind of question they should expect from their user. Today they start to get more and more clear opinion and they're hard on their product design into open-ended product to like specialize the product in different product features. And then from those product features, they want to have specialized models to solve those problems. So it's a natural evolution, evolution of product development.

And I guess like, you know, maybe just talking about some of the stuff you guys have built, obviously, you know, you're taking the great models that exist in the open source world and then you also build your own models, right? You built F1. Maybe talk a little bit about why, you know, why you decided to do So we release F1 under API and that's a model API.

Basically how you use F1 is as if you are using a new model, but F1 is really a complex logical reasoning inference system. So which underneath, we haven't talked too much about what's underneath F1, but it's kind of, here is a sneak peek of what's underneath. So underneath there are actually multiple models. Models, and also there is logical reasoning steps that we implement in our system. So then actually building a system is very complex. It's not like just regular single model as a service inference. And also we have to solve a lot of quality-related problems because now you let the model talk with itself, let the model talk with each other, and what kind of information they communicate to drive quality and they are sensitive to that. And how to do quality control in this complex system. I think the complexity is actually higher than building a database management system in that sense. And yeah, and then of course, because there are multiple steps involved and then your overall inference latency and cost becomes more interesting. Interesting. So that's our bread and butter because we're a team really specializing in optimizing for that. And then we view the— when we start a company, we build a lower level of single model as a service. That's a building block towards this complex logic reasoning stack.

Interesting. And then obviously, I think a big part of it is function calling as well, right? And what are the barriers, I guess, to getting function calling even better today? It feels like we're still kind of in the nascent stages of that in many ways.

Function calling, many people recognize that actually we see a lot of, after we launch F1, we have a waiting list for people to get in and we talk with many of them. Apparently their use case, the majority of use case is actually building agents and they need function calling. What is function calling? Function calling basically is an extension point for this model to call into other tools to enhance the quality of the answer. But function calling is not just about calling into one tool. It's actually very complex if you think about it. Usually people use those function calling model in the multi-turn chat context, right? The model, it needs to be able to hold a long context of what the conversation is, and then use that context to influence which is the best tool to call. And oftentimes they also need to call into multiple tools up to hundreds of tool selections. Also, it's not just call one tool at a time. Oftentimes they also need to call multiple tools in parallel and in sequential. It's a complex coordination plan in one shot. Our function calling is able to do parallel sequential complex planning and then orchestrate and execute on that plan. Of course, then the position is important. Like if I ask the model to like do, like generate chart of a stock price of top 3 cloud providers, underneath it will return the answer quickly and then show you a chart. But underneath there's a search to find out top 3, there's 3 parallel function calls to get the stock price, and then one call to the chart to get the chart back. This is a very simple example, but it demonstrates the complexity of orchestration. The model needs to have very strong capability of understanding if you plug in your tool, when to call this tool and how to drive that position is very important. That makes the tuning process very complicated, but we have been investing in this space for now about a year. We start to see when we start to release the first functionally model, I think we're ahead of the adoption curve. The use case hasn't fully taken off.

How do you think about why decide to build that versus, I'm sure if you waited at some point there'd be a good open source model with some of those capabilities. How do you figure out when to build things in-house on the model side?

We heavily build on top of open source community. That's absolutely is the case and we are betting on that because We believe the hundreds of small expert models will come from the open source community. Yeah. So that is directly aligned with our vision. But at the same time, we're also heavily investing in the Compound AI system where it's all about composing those hundreds of small expert models to solve a complex business task in a very easy way. And the critical part of this composability lies in this layer of orchestration. So the model who is intelligent enough to be able to call into different tools. And here you can think about the general way of thinking about tool is each individual small expert model is also a tool in addition to this particular large language model. So then it becomes a critical ingredient of tying everything together and we will invest these strategically critical areas that we cannot just wait to see what happens.

Where do, like, the reasoning models end up fitting in? I mean, obviously, at least the ones we've seen today are, you know, I think of them as bulky and, you know, generally good at reasoning of all things. Do we end up with, like, small expert, you know, like, test-time compute-intensive models as well? Or, like, do you imagine you will only kind of route to those in very specific instances? Or, like, as you think about these compound systems. I'm just curious where these O-1-like models will fit into that.

I think even for reasoning, there are different paths to solve reasoning problems, and there will be different models specializing in different paths. One path is to have a very strong base model itself to do self-inspection. And it looks like a lot of techniques already being discussed with chain of thought, tree of thoughts, and backtracking and all these different technologies. So that is one. And there will be a new set of models. They can do logical reasoning, not in the prompt space. And this is something I'm very excited about, but in latent space. Because you can think about when we think, right, we don't have to talk in words, like our thinking process. We can, but We don't have to write. So I'm pretty sure when we process a problem in our head, we're probably thinking in a different space.

Right.

So similar to the model as well. So there are a lot of active research happening to how to make the thinking process much more efficient and much more native to the process. So I'm very excited about that kind of research. There will be other flavor of logical reasoning. So we are not But we don't want to be very opinionated about which one is going to win. Instead, we're going to integrate all these different flavors into our logical reasoning system.

I love that. I guess outside of what you're doing at Fireworks, what do you feel like are some of the other big unsolved problems in AI infrastructure today?

I think we have seen a lot of movement into building agentic workflows.

Yeah.

Again, I think we're still at the early stage of figuring out what is the right user experience. What is the abstraction? The whole industry, right? What is the right abstraction? What we should hide behind the system, we should expose to developer. We're still in very early stage and I think it's very experimental right now, but where the abstraction abstraction lies will determine the complexity of the infrastructure hiding behind the abstraction. But I think we now start to form opinion around that. In a month or two, we're going to GA F1. So, okay, I was thinking processes building F1 is one is kind of F1 is our own exercise to understand the system abstraction and the complexity of building logical reasoning engine. From there, when we GA, we want to expose developer-facing plugins and how developers can build their own F1s. So yeah, so I'm very excited about that direction.

That's super cool because obviously you learn the tools and the abstractions that are necessary by building it yourself, and then, you know, you basically allow developers to recreate that. I love that as an approach. I guess, you know, a lot of it feels like on the hardware side, I mean, obviously everyone's using NVIDIA, but there's a lot of competitors have been popping up. And I'm curious, like, what you make of that space and the viability of some of those efforts. I think you guys support AMD chips for inference on the platform. Like, you know, when does that make sense for developers and how do you see that kind of evolving?

In general, we see a kind of scarcity in the developer space that they understand hardware, low-level hardware optimization. And we also see a very big change in the kind of hardware development cadence. Before, it's kind of every 3 years there's new hardware SKU. Now every 1 year there's a new hardware SKU from each hardware vendors. And then the whole hardware space is moving very very fast and how to kind of access to the best hardware. And by best, I really mean there's, again, there's no one size fits all. So it really depends on your workload pattern. Even for accessing the same model, there's no best hardware for one model even because your workload distribution is gonna determine where is your bottleneck, biggest bottleneck, and different hardware SKU is gonna be the best for solving kind of removing certain bottleneck the best. So we even at the Fireworks layer, we absorb the burden of integrating and determining which hardware is best for what kind of workload. Even for one workload, when you have mixed access pattern, we can route to different hardware. So yeah, so we want to really kind of alleviate the concerns and burden of developers developer, they should focus on building product and we will, we will kind of take the complexity of managing and optimizing for the hardware.

Yeah, it seems like you're obviously super focused on these compound systems. You know, there's obviously always going to be some set of folks that like, you know, just want to call a Llama model as is and run it. In that world, you know, it feels like for a while there was really a lot of competition on the inference side. Like a new model, open source model would come out and everyone would be offering their prices and it would get lower Or, you know, in that game, do you think that like eventually that's the hyperscalers are best suited to win just like kind of straight inference on an open source model? Or like how does that kind of develop?

That's a very complex question to answer. I think all hyperscalers, they want to be Apple, right? So they want to build iPhone. They want to kind of build a vertically integrated stack because they can. But again, the direction we're heading towards is we feel like the future lies in hundreds of small expert models and we want to harvest that energy and build a compound AI system that leverages the best models to solve a complex problem. So in my view, I feel like the hyperscalers, what's their biggest benefit? Their biggest benefit is to solve the problem requires a lot of resources in terms of money and human. Those examples are like building data centers, acquiring powers, and install tons and tons of machines, deploy them, light them up, and turn up large-scale storage and compute. Cloud provider please solve solve those problems, right? So this kind of massive problem at scale. And we specialize in solving problems that requires a lot of the combination of engineering craftsmanship and deep research. And then we can deploy at scale. So I think inference is like that. I work at Meta. It's not like Meta has a 1,000-people inference team, right? Because once you have a system that can be horizontally scalable with high production quality, then the rest is just let it scale. That's the beauty of designing a highly scalable system. So in that sense, I think this is squarely in our strength and expertise. And also again, the inference system is not simple. It is actually going to— we're evolving to getting to this compound logical reasoning inference system where there's a lot of complexity in building that. And that complexity is not by throwing people and money to solve. So yeah.

Yeah. I guess with these smaller expert models, one thing I wonder is, as these models get smaller, it feels like there's a big trend to running them locally too. And I'm curious how you see that playing out over the next few years and then if that becomes part of the Fireworks platform as well.

Yeah, I've seen a lot of argument for running models locally for two reasons. And also we should talk about what does local even mean, right? One reason is cost saving because yeah, you need to pay GPU on cloud and running on desktop, you don't need to pay. The other is privacy. I have different thoughts on that. I think that makes sense to offload compute from cloud to desktop. I think many applications like Zoom, for example, that save a lot of cost for them. But offload compute from cloud to mobile is a different story because mobile phones Phone has a very, it's getting beefier and beefier, but it has a very limited power. So at Meta, for the application, a lot of application metrics are closely monitored, including cold start time, power consumption. All these affect application rating and user adoption experiences. So because of that, the model that can practically deploy to mobile is tiny, like 1B, 10B, and very, very small. And those models have limited capability. So I think between mobile and cloud, the difference is extremely big. And offload to desktop, I think for many consumer-facing applications makes sense. But also there's another argument of, hey, it's more private. And privacy is interesting because we have a lot of personal data is already on the cloud. Most of our personal data is not on local disk anymore. So then, hey, does that even make sense for privacy? Yeah. So I think that's a separate concern.

Yeah. Super interesting. I guess on the open source side, obviously, used to work at Meta. It feels like obviously they've been providing a huge service to the ecosystem by training these larger and larger models. How far do you think they go on continuing to scale spend on open source models before at some point— I mean, I guess everyone's asking this for all the players, not just the open source players, but how do you think about the extent to which it makes sense to keep pushing on pre-training spend?

I think that's a good question to Zach. No, I mean, jokes aside, we work very closely with Meta team. Of course, we came from Meta and we actually co-design. They're not just open source Llama models, they're also building a standard called Llama Stack. And Llama Stack, the intention is kind of standardize the tools stack around Llama models. And we co-design with them because we are kind we have a lot of information coming from our customer that can help them make design choices. I think that's ambition from Matter is to build this Android world where everything is nicely standardized and you plug into different components and make it very easy to adopt. From my vantage point, My understanding is Llama 4 is coming soon, right? So there will be continuous investment from Meta that train. I haven't seen that stopped. Of course, if it's a general question, when would model providers stop or invest less on pre-training is when there's not much ROI. So why there's not much ROI? One is because we're hitting a wall of data. We're running out of data. Everyone crawl the same internet data. There's just, we have exhausted synthetic generated data. We have exhausted a combination of multimedia and text data. Then kind of there's less return. Before we hit that wall, I think there will be continuous investment.

When do you think we'll hit that wall?

I think we're, it's definitely getting longer and longer. Like from OpenAI's release point of view. Right. And I definitely see kind of a lot of investments start to shift from pre-training to post-training, from post-training to inference. Right. So I think we're already hitting a soft wall, not a hard wall. So yeah, so I think in general as an industry, kind of where the ROI is kind of transitioning from pre-training to post-training to inference.

As you think about all the sets of tools that are required to make an agent successful, I mean, there's one world where as Fireworks, you could be a one-stop shop and build all of them. There's others where you could partner and you're like, look, we're not going to build this part of the thing. How do you think about that?

Yeah, so we always are very compatible with the imperative agentic tools, right? I think most notably is LangChain. And we have been a very strong partner with them because we are not going to— like, Lin Qiao is doing great work in what they are building. Huge community following and adoption. Really awesome. And we just want to simplify the next level above single model as a service by composing wherever it makes sense, multiple models to solve the problem much better. We are fully kind of, we will not change our position, we'll plug into LangChain. For example, as one example, there's also of course many other examples in the community.

Yeah, makes a ton of sense. As you think about kind of like the competitive landscape today, I mean, I imagine when people think of Fireworks, they also probably lump you in with like Together AI and I know Databricks Alex talks a lot about these compound AI systems too. How do you think about what differentiates Fireworks today and then 5 years from now, what will ultimately determine who captures this really interesting space?

Yeah, we are probably thinking— so compound AI actually is the term we didn't coin that term, right? So it's kind of coined by Berkeley and we know that Databricks is also looking into that space. So I think I think it's great. It's a new category that we're defining. Definitely multiple players will be in the same space and we are very happy that Databricks is also thinking about that problem. I think that space is very meaningful because it's a very complex space for application developer, product developer, engineers to get into. A rich set of tools that emerge in that space to make development much more efficient. And today we're not there yet. So we are determined to be a key player in that space. Other than that, we are not in the space of being a GPU cloud, for example, and provide access to cheap GPUs. That's not our at all. So we actually, we build on top of GPU cloud to offer a complex inference stack.

I guess one thing that's, I think, really interesting about the company is I guess you started Fireworks like, what, a few months before ChatGPT? And so I'm curious, like, you know, what would, like, as you think about the original vision for the company and then, you know, where you've come through today with this, like, GenAI craziness, like, how much of it was the same, you know, sitting there in September '22? How much of it had you had change as the world has just changed so much in the last few years?

Yeah, it's actually, yeah, this is a very interesting question. When I started Fireworks, there was actually active debate, is AI here or not? Because it's pre-awareness of foundation models and it's kind of, oh, there's so many fragmented AI application And the data is not there, AI is not there. But we clearly see that AI is coming. So usually hyperscalers are like 3 to 5 years ahead of the whole entire industry. And Meta is massive AI powered. And clearly the industry is kind of through PyTorch adoption is clear. They see the wave is coming. So that's kind of the timing about fireworks. And the GenAI did skew, ChatGPT heavily skewed the adoption curve of AI. And there's a special kind of AI. So it's very interesting. GenAI is a different beast, not because it's GenAI, it's magical, because it's fundamentally changed the accessibility. So pre-GenAI, when it's like traditional machine learning or what we call early deep learning models. Any company investing AI particularly have to first hire a machine learning team because they have to train model from scratch. There's no other alternative. There's no other model that's pre-trained you can build on top of. And that means this team has to curate, spend a lot of time curate data, and then the training process, because training from scratch it kind of take a long time. So they have to have the resource to hire those very scarce talent and put money into training.

And was the original Fireworks product built for those people? Or like, because that was originally what it felt like was going to be needed to get adoption.

Yes. Yes. That's what when we think about when we started, we think about that cohort. And then with GenAI, it makes it much— it's completely changed the landscape, right? You don't have to hire big team to carry data. Because it has the beauty of GenAI is it creates a foundation model that absorbs a majority of the knowledge and then you just build on top of it as is or you tune, right? You use thousands of samples, you tune. So then you either don't need to have any machine learning team, you just need to have the application product team to build on top of that directly or you have a small machine learning team, right? So this is a massive, massive unblocking of accessibility of that technology. And that's why we see the adoption curve take off like crazy because of that accessibility is fundamentally different before and after. So because of that, we laser focus on it because the pull is much stronger in that direction. And the other side effect is all these models, all generative models are PyTorch models. And we wrote PyTorch code for years and we operate PyTorch, large complex PyTorch model in production with high volume traffic for years. And that's what we are really good at doing.

Yeah, very well situated for that. You mentioned when you're at a place like Meta, you get kind of a window into the future before it's obvious. Was there something you saw when you were at Meta where you're like, God, this is just going to be clearly so big shortly thereafter?

I think it's more the kind of AI is going to be very big wave, that's clear. I worked at Meta for 7 years. It's interesting time. When I joined Meta, they were going through the mobile-first transition in the tail end. Then the industry is also going through mobile-first, right? So from desktop to mobile. Mobile-first enable consumers to be able to access application everywhere, and that drove adoption and engagement. And because of that, it produced a lot of data. And a lot of data is powering AI, right? So that's— then the whole entire industry is also following the mobile-first, drive a lot of data, and data power AI. So that trend is very clear.

What AI research do you pay attention to, maybe even outside of like the core day-to-day of Fireworks?

I usually pay attention to two kinds of research. One is is model-system co-design kind of research because there is an interesting organizational division, as in there's researchers, they focus on quality only. There's a system builder like us, we focus on deliver the best quality with low latency and low cost, right? So solving system problems. But oftentimes, to find the best ROI is think about that together. And that's how we operate at Meta is kind of the research team and kind of infra team sit very closely to discuss trade-offs and the co-design. So there are a lot of research in the co-design space. I think it's going to be very appealing to make, to kind of find the best design point across quality, latency, and cost as a three-dimensional optimization. The other kind of research is kind of fundamentally different, disruptive kind of research, as in Transformer as a technology is overdue for disruption. So where are the next generation of Transformers that's going to change how we train model, change how we do inference, and so on? So that's very interesting. And also in the agentic world, how different agents are going to communicate with each other. As I mentioned, I think in the latest space, that kind of very new kind of thinking is very, very interesting.

One challenge that I imagine in building an AI infrastructure company is that obviously the pace of change is so fast, both like the improvement in models plus just like how enterprises are thinking about and actually using these things. And I can imagine like one school of thought would be it's changing so fast, like, you know, we'll just kind of quickly update what we're, you know, what our kind of core tools are because of how fast it's changing. And at some point it will settle down and then we can kind of build the standard set of tooling. How do you think about that tension of like, you know, building for however folks are doing things today with an, you know, without knowing exactly, maybe in 2, 3 years, like models have all these other different capabilities and we're designing systems in all these other different ways.

Yeah, that's a really Good question. That's all we think about all the time. We don't want to keep chasing what's coming because chasing is exhausting. We always want to stay on top of the curve. I think the model capabilities are definitely evolving, has evolved from 2 years ago, especially if we just look at OpenAI's model capability from GPT-3 to now today, there are night and day differences. But fundamentally, a few trends that doesn't change. One is, again, anchor back to our vision, is we believe the direction is specialization and customization. That doesn't change how the model core capability evolves. Because we just don't believe that one size fits all different workloads with proprietary data and so on. We believe there's a much better solution if you can customize, if you can steer, if you have control, the destiny here. That's why we built our stack to heavily enable that in an easy way. Specifically, we offer on top of our inference engine, which is one size fits all, we have optimizer, file optimizer, that's one size fits one. We take inference workload as input and also your customization objective as input and spit out inference deployment configuration along with potentially a model, adjusted model for you to have the control Lin Qiao whether you want to deploy that or not. So this is just a process of we kind of close the loop and make it super easy to customize. So I don't think that's going to change in the future.

Totally. Just what you're customizing may change, but like still the way you will go about it, I think that makes a tremendous amount of sense. Well, we always like to end our conversations with a quickfire round where we get your take on a standard set of questions. And so maybe to start, would love just just one thing you think is overhyped in the AI world right now and one thing you think is underhyped?

I think overhyped is the perception GenAI is magical. It's a recipe of all problems. We should just ask it any questions. It's kind of, it will have the right answer, I think. I think it's going through the correction time right now. And again, that's where kind of we believe like there's no magical one model that solves all the problems the best way. Or in the correct way.

What's one thing you've like changed your mind on in the AI world in the last year?

I think my hypothesis in building this company, how this generative technology can adapt, adopted in our go-to-market strategy, in my mind is always like in a sequential way. Startups are like generative, so they will be on the frontier of adoption and then we'll work with digital native. They have very strong engineering. Resource and always be more tech-forward, and then traditional enterprise because they have more other problems to solve. But now we're working with all of them at the same time. To me, it's a little bit crazy. Of course, the use cases are not all the same, but the adoption curve is happening all at the same time. It's very different from than I was imagining.

Totally. I mean, on the application side, it feels like the normal rules of how long it takes to land one big enterprise and then land a few after, it's all out the window, right? Some of these application companies are scaling so fast in the large enterprise. I think there's just a tremendous appetite for it.

That's right. Yeah. That's right. So that's why this, I feel like this is where in this revolution way, right? So a lot of things will be done differently, not just kind of the application we build will be different, technology adoption, Adoption curve will be different, but also how to think about go-to-market will be different. The sales cycle is much shorter and people are open to think the procurement process differently. It's a very interesting ripple effect from this massive transformation.

Do you feel like, is there any difference in what you have to build for enterprise versus the startup folks, or is actually the needs pretty similar?

I think my observation is startups, they are typically, they want to get access to, and when we talk about abstraction, they want to get access to the low-level abstraction because they want to tinker a lot. So, assemble a lot of things. For traditional enterprise, I'm not saying this in an exhaustive way, but typically, they want to hire higher-level abstraction. If they don't need to solve for low-level details, you don't need to pay attention to, rather not to pay attention to. So even at Meta, we have those different abstraction layers for different kind of team because they want to pick and choose. So usually two-level abstraction will suffice. They were happy with at least have two choices. And then we see a little bit of those kind of—

Oh, it must be interesting to build both at the same time as you're rapidly scaling.

Right, right. So we need a low-level abstraction for ourselves anyway. So kind of to us, it's not additional overhead, but we see kind of the adoption will happen at different layers.

Yeah. I know it's probably, I won't ask you to pick a favorite, but what is like one favorite application that's built with Flux?

This is a tricky question. I would say Cursor. But we see a lot of similar company in productivity space, not just Cursor, but also Sourcegraph, Zed, Cognition Factory. They're all very forward-thinking company. And I think in that space it's still early, but that space we're very excited about the whole entire space.

Do you think someone will spend $100 billion on training a model in the next few 2 years?

Ooh, maybe. I think it only makes sense if they are training using a fundamentally different model architecture. Yeah. It's so disruptive, it's worth that investment.

Yeah. You kind of already mentioned the coding space and some of those startups, but any other AI startup or space that you're really excited about outside of Fireworks right now?

Again, I think the agentic The agentic world hasn't been fully figured out yet.

Have you seen any applications do that well?

We have seen early applications, for example, the digital SDRs, the digital marketing. And yeah, so we have seen kind of good startups. Getting great adoption there. But I do, I do think in the future there will be a lot more complexity in that space. And we're still early.

Well, this has been a fascinating conversation. I'm sure folks will want to pull on all sorts of threads. So I want to leave the last word to you. Where can folks go to learn more about you, about what you're building at Fireworks? The floor is yours.

Yeah. So yeah, definitely. So we offer For the benefit of our developer community, we offer a self-serve platform. Very simple, just go to firewalls.ai and you can access to our playground and hundreds of model capabilities we have. Yeah, and feel free to reach out to me also. Connect me on LinkedIn and love to hear your use case, your challenge and pain points.

Amazing. Well, thanks so much, Teresa. This was a fascinating conversation.