When Will Inference Feel Like Electricity? Lin Qiao, co-founder & CEO of Fireworks AI

What would it take for inference to feel like electricity?

Inference scales fundamentally differently. So we actually don't believe in one-size-fits-all. So does anyone have absolute secret sauce? Probably not.

Hello everyone and welcome back to Inference, the interview series on Turing Post. And today I'm thrilled to talk to Lin Qiao. She is the co-founder and CEO of Fireworks AI and the former head of PyTorch at Meta, where she led the rebuild of Meta's entire AI infrastructure stack. Welcome, Lin.

Thanks for having me.

Let's start with a big question. When will inference become a solved problem? Like, what would it take for inference to feel like electricity— reliable, cheap, and visible? And what still stands in the way?

That's an interesting question. I think we have— we are just at the starting point. Of optimizing inference, and there are many different dimensions to look at. I want to probably look at from the eyes of app developers. Those are the cohorts we care the most about. There's no question that GenAI is a revolutionary technology. It's able to generate content closest or beating humans' interaction with the real world, right? So that's kind of the biggest value. Because of that, I think it's inevitable. It's all pretty safe to predict. There will be many generational companies be able to define new user experience that never existed before, building on top of AI. They are going to disrupt the industry and how we interact with software, with application day to day. Through that lens, I think we already see kind of the lot of innovation that is happening in the field, but one interesting phenomenon that's happening is before we talk about product market fit, right? So if you hit the product market fit, then you just scale, right? So that's kind of a well-understood runbook. If you hit product market fit, then you're going to have a viable business. In GenAI, for application built on top of GenAI, hitting product market fit and having a viable business are the two separate problems. It's very interesting because you can build a new user experience having a lot of value to the end users. Whether they're consumers or developers. But it doesn't mean you can quickly scale to a viable business because your operation cost is going to be so much higher. Everything around the GPU infrastructure, about GPU, is orders of magnitude higher than traditional application built on top of CPU. So we often hear stories that, you know, for the companies talking to us, they are like, hey, we are very confident about our product, a lot of great signals, we beta launch it, we have a waiting list of millions of users waiting. We couldn't open up the floodgate because we know if we open up, we're gonna run out of money. In another word, having a product-market fit is the beginning of going to bankruptcy. Isn't that interesting, right?

That's very interesting. Yeah, it's a new thing.

It's a new thing, and that's fundamental because you can visualize this, right? The new application is like the iceberg, right? It's a very big iceberg. That is being built right now, being created right now. But right now, most of the iceberg is submerged under the waterline because infrastructure cost is very high. And if you imagine the cost start to kind of shrink down to 10 times lower, then the amount of application that will emerge above the waterline is going to be enormous, right? That's where the future is going to be. But when it comes to how would the infrastructure cost shrink down, and there are so many different approaches, and we would love to kind of talk about our opinion, our observation. Especially working with the leading application providers.

What is your approach to making this iceberg smaller?

So fundamentally, it boils down to the fundamental misalignment or disparity of two sets of data. One set of data are the data being used to train a foundation model in a research lab. It doesn't matter whether it's closed model, open model, doesn't matter. For the research lab training foundation model, they will have a team defining the objective of what are the most important areas that model will be very good at, and based on that, curate all the data related to the problem statement. So that's the data distribution they will control and manage and feeding the model, and the outcome is what they want to see. So that is one, on one side. On the other side, there are so many application developers. Their goal is to align their product design to user engagement. Right? Their goal is to maximize user engagement from their product design. So they're constantly experimenting the product features and design to drive those product metrics based on the product data they see. I would say the kind of product data, data coming from the product, is another data distribution.

Mm-hmm.

With these two data distribution, as you can see, it's built for different purposes by different companies, by different group of people. By design, those are not aligned. And now you're putting them together. The app developers, when they build product, they're gonna use those models and then these models will power the product. And this misalignment is the fundamental root cause of accuracy gap, of latency gap, of cost gap. We have seen many great companies, they, despite they're new, they're young, they figure out how to create this alignment themselves. They are able to control their destiny to have a better model for themselves, faster model, and also cost-efficient model for them to scale viable business. After product-market fit. So this is the kind of fundamental observation we have across the industry. That is the missing part that a few small set of people are able to figure it out now, but most people are staying at the space where they think about model API as utility. They just send their request to the model, even though there's a big misalignment that exists. That's kind of an interesting dynamic space we observe right now, and we're trying to help our application developers to be able to close the alignment gap.

So you So you're working with enterprises to align these two data columns, right? Can you just tell me a little more about your journey to that? Because you founded Fireworks in October 2022 before ChatGPT. So why did you start it then and how your vision changed and your approaches changed when generative AI really boomed?

Yeah, so I will say the founding team, we have a very big founding team, 7 of us. We have been working at Meta between 7 to 10 years, pretty much bootstrap the AI infrastructure across both training and inference at Meta. When we started, we have option to do both because PyTorch has been used on both sides, or pick training or pick inference. And at that time, this is September 2022, most of people on training, they, they're either training a model themselves or selling GPU for training or building some kind of training infrastructure. We have decided to go all in on inference. We only like to do inference. We have experience on both sides because We believe inference scales fundamentally differently. It scales with the number of consumers or developers where world population is the upper bound. Training scales with a very small talent pool of researchers. So we feel like the inference TAM is way bigger. It also has like high production requirement and complexity goes with it. And those are the problems we love to add value to. So that's kind of the initial focus and strategic decision we made. If you look back, feels like we were against the trend. People at that time will say, who will care about inference? There's people are still early and so on. So, but I think thanks to that decision, we made the investment early and we have built a very sophisticated toolchain for us to be the best provider on the inference side. The idea goes back to data alignment problem I mentioned. So our inference stack is not one size fits all. You don't use our inference stack as, hey, this is just another API. No matter what is the workload shape and pattern is the same. So we actually don't believe in one size fits all. We believe in one size fits one, and every application is different. In some sense, this idea is not new. It exists in many other compute, um, namespace. For example, databases, right? A database will take a query and then optimize it underneath through a query optimizer and execute the most efficient plan, right? So a database is not one size fits all. Every query is treated differently. So similar idea here is we treat every application workload differently, but the optimization happen automatically. We call it 3-dimensional optimization, 3D optimizer. So 3 dimensions are quality, speed, and cost. So we optimize across all 3 dimensions at the same time. So in that sense, it's a much more complicated problem than a database query optimizer. And the bad news is the search space is big because there are a lot of underlying components. We need to optimize for. Each component has lots of options to pick and choose from, and those options stack on top of each other, leading to more than 100,000 combinations. So we are picking one needle in this 100,000 haystack. So that makes the problem very complicated. But the good news is we are very good at solving those kind of problems. Now we're kind of— almost all customers onboarded to Fireworks are using our 3D Optimizer. Today.

It's probably very hard to explain that to enterprises on such a multi-layered level. Yeah.

So this is more like what's happening under the hood. But when we talk with enterprise, we will map it to their business values, right? And they use— right now we have seen a wide variety of very interesting use cases, especially 2025 is the year of agents. It's amazing how fast this industry moves in space is across startups and enterprises. They're all building some form and shape of agents, right? All the way from coding agents, significant various different kind of coding agents, vibe coding or non-vibe coding to significantly increase the productivity. There are agents, there are hiring agents where giving a job profile automatically source talents and even conduct interviews. For you and assess performance. There are SRE agents, those production engineers that can help you debug triage issues, you know, during the incidents. There are customer service agents, very popular one because all large enterprise has customer service. Some enterprise even have more than 20,000 human agents in the customer service space. To make them productive is a huge cost saving for those enterprise. There are marketing agents, so giving your marketing campaign automatic generate outbound where the very specific target audience. And we also see a vibrant adoption from the medical space, from retail space, from education space, from finance space, across the board. So usually we will, when we talk with those enterprise, we will talk about case studies and impact from this video optimizer. And that's kind of land much better than talk about technical terms.

How do you talk to them about models? What is your approach? I know it's no, uh, fit at all, but is it a general model? Is it model that's much more narrow?

Yeah, obviously we believe in open developing in open. So our business model is mainly focused on open models because it provides transparency and full control enterprise can have, and they do care about that a lot. But at the same time, we also want to look at the problem from our users' eyes, right? So their goal is not to make open model successful. That's not their goal. Their goal is to solve the day-to-day problem and land big business impact. And they will use whatever tool it takes to make, make sure they'll be successful. So during enterprise engagement, we have cookbook for them to build their AI gateway where they will connect with whatever model provider they would like to, and we will be one of the provider, and we help them standardize that stack. We also have a set of evaluation, our private evaluation benchmarks for them to assess different use cases, which model is the best. And there we are very objective. If a closed provider then we will just give them the report and then they can pick and choose from. And for the cases that they want to tune open model and to drive the best quality, we also give them tools to do that. So we often come to where, meet them in where their needs are and go from there. That's kind of our fundamental operating principle is customers first. We do not want them to bend themselves to think about how to use a vendor. We want to go to to meet where they are.

Well, talking about open models, Chinese models are completely crushing it lately. And why do you think Deepseek R1 and latest QIME K2 and Qwen 3 and GLM from Zed AI, why do they set new precedents for all model providers, open and closed? What is so special about them?

Yeah, I would say this is very interesting, right? So because they are more resource constrained, they don't have the powerful GPUs as we have here, but that doesn't prevent them, doesn't stop them from innovating. And this is a bunch of things, right? Overall, I think there's a sign that the models are converging overall. It doesn't matter if it's closed or open model because at the end there are two things, right? One is training techniques. You have secret sauce that goes back to talent. I think talent is kind of flattening, as in people move around and there are a lot of information sharing. So does anyone have absolute secret sauce, probably not. And then the second is data, right? So as I mentioned, the data and data distribution determines the model quality at the end. And the amount of data is converging from publicly available data you can crawl or from labeling companies where they can provide additional label data to increase quality. My understanding is all the, you know, all the labs are using more or less similar kind of labeling companies. That part is going to flatten out. And then it's about how you generate more data, right? People generate synthetic data. People train the largest model in order to go small. Largest model can generate synthetic data to train a smaller model, right? So then the synthetic data generation is going to be capital heavy. There's also specific kind of tools or approaches to generate good high-quality synthetic data. I feel that probably no one has unique advantage in the long run. Those are kind of various different factors, but at the same time, people are also being becoming more and more creative in when they train. They don't just about training time quality. They are also thinking about inference time together at the same time. So cross-training and inference alignment becomes the focus. And that's where various different model providers, especially recent models, are— we see a lot of innovation, creativity because they are doing experiments in order to kind of make the inference time go fast. What are the new model component tweaks they can implement? And they did implement that. It shows up as very effective. I would expect continue to see a lot of tweaks, model architecture at the small level, not kind of major level.

Mm-hmm.

Make the model quality and the speed and cost much more optimized continuously from various different providers. But overall, my prediction is the overall space of the base model, is converging if we only look at one modality, right? So one modality is text, text-based. Another dimension is, hey, what if about multimodal? I think multimodal, the closed model is still ahead of open model by a good margin. That's the investment in data and the focus. I think the open model providers are more laser-focused on the text-based, the logical reasoning model, the coding capability, tool call capability for agent development and so on, but less on voice. Vision, video, there are some effort there, but less a focus compared with LLM. So we'll see the convergence on LLM to go much faster and other modality will catch up.

Yeah, I'm pretty sure it will catch up. It's probably just much more expensive and there is more commercial use in multimodal. So for the closed companies, it makes much more sense to protect their invention. But I agree with you, it's going to— they're going to catch up pretty soon. So we were talking about these Chinese models and they've been super innovative and they are open. It feels like they benefit from that. Zhipu AI, which is now Zed AI, they've been obsessed about AGI. So for them, it's apparently their way to achieve AGI through open source. If we talk about the US companies, like your ex-employee Meta, they recently published this statement about superintelligence where Mark Zuckerberg says they might not open source everything. They will, you know, think more carefully. Why such difference here? How do you think about it?

I think on the Matter side, we are yet to be seen, right? So I don't think that's the final, final call, but I can understand there's continuous internal discussion, which is healthy, about where to focus, right? So if pushing the product first to drive the business top line or pushing the ecosystem to drive consolidation on Llama models, right? So those kind of of just business decisions. I would expect continuous discussion within Meta. At the same time, I think Google has been driving open models, and I think I heard other providers are also driving open models from, from US. So I would say very excited about, hey, they do care about the open model community and want to contribute. The interesting part of contributing to the open model community is the bar is not low. You contribute not because it's because today whoever is launching a new open model, they have to show benchmarks across the board compared with the closed model as well. And the latest open model, which is already a very high bar, it will push the open model to become better and better because no one is willing to lower the bar. And if you lower the bar, no one cares anyways. It's a waste of time. So I remain very excited to see what's yet to come. It's kind of a demonstration of showing the depth of the research and showing the talent density and output from a particular company, it's good for their reputation. It's also good for the broader community.

If we talk more about superintelligence and AGI, what's your take on it? Is it like solving intelligence or is it more about building better tools or you just do not think about it?

We have a strong opinion where Fireworks provides value. Obviously different companies will have different positioning. Where we want to provide value is we want to make application developers shine. We want to provide the best tools, toolchains, infrastructure for them to build on top of where they can easily create the data pipeline well from their product data and drive the last mile data alignment with the model. So the model becomes better tuned towards their application. Their product becomes better because better model and drive better user engagement. More user engagement drives more data. More data drives more revenue. Better model. And this is a virtual cycle we want them to create. And that's where we are very passionate about building and delivering to the hands of application developers. So that's our strong opinion is we add value as tools, as infrastructure, and we want to enable them to create the best social value that will show up in our day-to-day, that my family, my mom will be, will be using it and say, oh, this is really cool. And I will be able to say, hey, actually they use Fireworks. And actually I can already say that to many of the newer applications. And I want to kind of be able to enable way more.

When do you think, what's your prospects when AI infrastructure will be much easier, much lighter?

I think we have already demonstrated that capability that through our 3D optimizer, we can significantly accelerate speed and cost, reduce cost. And sometimes, you know, without sacrificing quality or even improving quality by the order of magnitude of improvement can be between 4 times to 10, more than 10 times better. So that's the power we have demonstrated. We plan to continue to maximize that and continue down this journey. But I think the industry should see 100 times more efficient infrastructure. Yeah, that future is definitely coming. Obviously we are adding value, a lot of value in that layer, but I also believe the fundamental infrastructure, the fundamental hardware, underlying hardware, and everything will be much more efficient over time. That's where we see the CPU journey. The CPU itself from single core to dual core, quad core, and with newer, newer generation, the price performance becomes significantly better. And with massive adoption, the manufacturer costs get much better and everything around that becomes much more efficient operationally.

CPU, not GPU.

We have seen that at the CPU time. So right now I think And we should see that in current space of the AI hardware on GPU, other hardware accelerators, ASICs across the board.

Your approach is very pragmatic and you're building this AI world. What excites you and what concerns you the most about the world you're building?

What excites us, being very consistent at the beginning of our journey, is this is extremely fast-moving and dynamic market. And we firmly believe this, we are providing a generational new technology that that will shift the industry. And we are in this huge wave, a huge tidal wave of industrial shift, probably even much bigger than the previous wave of cloud-first, of mobile-first.

Most likely.

That's inside us day to day. Like we woke up, we go to sleep and woke up, immediately think about how to solve problems. What is the strategy that's kind of very energetic and peak our intellectual curiosity and creativity. Activity. So the whole entire company is very energetic around that. What keeps me up at night is how to strike a balance because we tend to go really fast. Our company, like, operating principle, one is customer first. I talk about that. And the second is high velocity of innovation. So these two are the fundamental operating principle and everything else derives from that. On the move fast side, there's a flip side of move fast, tend to break things and how to manage that because we do provide high-quality infrastructure and we proud of ourselves for doing that and maintain a good balance between, you know, continue to move really fast, but infrastructure is highly stable and reliable for customers. We're pushing hard to kind of strike a good balance. Those are the interesting, interesting challenges we're seeing. And we want to also build infrastructure so we can scale very quickly, scale our fleet very quickly, scale our product features very quickly, scale various different customization very quickly. So there are different level of velocity of speed we are adding. Compounding on top of the base layers all the time. So those are the kind of interesting problems we like to solve. It's a very complex infrastructure problem, but we are very good at solving those, and that also keeps us up at night all the time.

Thank you so much. My last question will be, what is a book or idea that shaped the way you think about your leadership or the future? But basically, what is the book that shaped you?

I probably cannot pinpoint one book. It's more like the life experience that shaped me, the interaction with many people that I have learned from and the journey, the internal journey I work with myself to think differently, that have shaped who I am today. You know, 17 years ago, I'm a completely different person. I have a lot of passion to do things, but it's more like myself telling myself, hey, maybe other people is better at doing those things, not you. And it took me multiple years to take that inner voice out. That's a very interesting journey to kind of work with because it's not like I put myself in a challenging position to grow, it's other people push me and force me to embrace myself that I can accomplish and give me a different level of imagination. This is a very unique journey I've experienced. I would say I really highly appreciate. It also gave me a lot of firsthand insight to share with other people whenever I see that their inner voice is actually the limitation, uh, not anything else. That's kind of the interesting part of my, my personal journey.

So your book is the people around you?

Definitely. And find the people who really challenge you and make you uncomfortable is a blessing. And go through the pain of embrace those challenge with the possibility you may fail, and don't be afraid of it. Once you get to the other end of this tunnel, you will be a different person.

That's very inspirational. Thank you so much for this interview.

Thank you for having me. I'm having a lot of fun.