Why This Ex-Meta Leader is Rethinking AI Infrastructure | Lin Qiao, CEO, Fireworks AI

Welcome back to the MAD Podcast. I'm your host, Matt Turk from FirstMark. Today my guest is Lin Qiao, the CEO of Fireworks AI, an LLM inference platform that enables companies like Cursor, Uber, and DoorDash to build AI product experiences on top of hundreds of open source models like Deepseek, Qwen, Llama, and Mistral. Prior to co-founding Fireworks, Lin had an illustrious career at places like IBM Research, LinkedIn, and Meta. We talked about how her experiences with PyTorch at Meta led to the creation of Fireworks.

One of the biggest success we saw from the PyTorch experience is simplicity scales. People don't want to spend time figuring out how to make things work. They just want it to work.

And also covered a bunch of other topics from the continued drop in inference prices.

I believe this overall AI infrastructure would go down by order of magnitude in terms of cost, and they should.

To Lin's thoughts on the future of open-source AI.

Despite Deepseek model extremely hard to tune and optimize, more than 500 variants published on Hugging Face optimizing for local device, optimizing for cloud infrastructures.

Please enjoy this great conversation with a CEO who's very much at the forefront of generative AI infrastructure today. Helen, welcome.

Hi, thanks for having me here.

We are going to talk about all things Fireworks AI today, but as a level set, what is the elevator pitch for Fireworks AI?

Fireworks AI provides a developer platform for application developers to build on top of GenAI technology. They need to hypothesize what kind of product is interesting, and they need to use certain models to integrate and build these great innovative user experience. Oftentimes they are— they have to solve multiple problems. One is quality. Is the model delivering towards the quality they want? And also interactive real-time speed, because many of those products are consumer, person-facing. And then when they hit the product market fit, when they scale the business, they have to have a sustainable way to scale. So cost efficiency is very important. So oftentimes we see application developers heavily focused on three-dimension optimization, quality, speed, and cost. And we want to take away all this complexity from them. So they focus on thinking about what is the best user experience, what is best product idea.

So infrastructure as a service, you abstract away the whole complexity of what needs to happen behind the scenes to deploy those models.

Exactly. Okay, great, great.

Maybe a few words on your story, your background. In particular, you were one of the key people on PyTorch. At Meta or Facebook at the time. Yeah, Facebook at the time, I guess. What is PyTorch for anybody that doesn't follow the things in great detail? And then how did it all come about? What was your role there?

Yeah, that was an amazing journey. When PyTorch happened, what was the kind of industry context here? So I joined Meta in 2015. At that time, Meta it was called Facebook at the time, was going through the transition of finishing mobile-first to starting AI-first. Like, the fundamental reason why it's sequenced that way, because after Meta transition its application from desktop to mobile-first and it's widely accessible everywhere using your phone, you can connect with people and that drive a lot of user engagement and therefore a lot of data being created. And then we all know data fuel AI. So quickly after mobile-first transition, we started AI-first. At that time, there's no software, no hardware, there's no GPU, no people building all this infrastructure. So we bootstrap the whole thing from ground up. And it was interesting time. Feels like right now there's so many company building their own AI framework. Gazillion number of AI frameworks. Even within Matter, there are 3 different flavors, one for mobile, one for research, one for production. So it's very confusing. And we decided to unify all of that and have one framework aiming to be the best tool for researchers to create innovative model architecture, but also one framework to deploy and deliver those innovative models into production to power all the product needs. So these two actually has huge tension across because for researchers, you want the flexibility, you want ease of use, you want them to just think about what's possible, right? And for production, it's a constraint problem solving, as in you have latency budget, you have cost budget, you want to scale, you want to be reliable. So it's kind of, hey, you have all these constraints, let's kind of optimize. It's an optimization problem. So how to create one framework to solve both problems with big tension in between is a very, very hard challenge. I would say we have a lot of fun thinking about that. It was deemed as Mission Impossible at the beginning, and many people feel this is so experimental, probably about to fail, but it's worth a try. And I would say our initial attempt was not successful.

When was that, 2016?

From 2015 to 2016. And because the idea is great, Idea was simple because PyTorch has great frontend, very simple Pythonic, and we have another framework internally. It has high performance kernels backend, and you would think we can zip them together so we get the benefit of both.

How hard can it be?

How hard can that be, right? So let's just zip them together. It's actually internally, it's actually called Zipper. And it turns out it's really, really hard to bring two frameworks completely designed with different goals goals and different interface and internal APIs to kind of work seamlessly together. It's like building a bridge with both ends moving. It's going to be an extremely fragile bridge. So then we decided, hey, making both sides happy is not the goal. We need to build an extremely strong product. And we decided we were going to rebuild the whole entire backend from this beautiful frontend. And bring it to production. It took us 5 years, took us 5 years to get to the stage supporting almost all internal needs using deep learning and massive, massive scale. So yeah, so it also has been deployed to not just data centers. So Meta owns its own data center, but also mobile phones. If you use any of Meta's apps, that means our tiny infrastructure is running on your mobile phone and also AR/VR devices. It's kind of ubiquitous. It's everywhere. At that time, PyTorch externally, it's an open source project. It has grown from a baby GitHub project into more and more massive adoption. So through that, we got exposed to many other companies using PyTorch. And it's fascinating to see the growth of all different use case adoption from image recognition to ranking recommendation to robotics, self-driving cars. And GenAI is nascent, but it's literally everywhere.

And it's used by all the research labs, right? Like all the—

it started from research labs. And the fun fact is OpenAI switched to use PyTorch fully from what from TensorFlow. Right. So TensorFlow was dominating at that time.

TensorFlow being the Google framework.

Google framework. Yes. Google also put a lot of effort behind open sourcing this effort, growing the community. But simplicity wins. PyTorch is so simple and the user interface, the debuggability, and people can easily change the model architecture as they want. It's dynamic and flexible.

To drive it home, the simplicity point. So it does, because you had the sort of low-level and high-level stuff. So it does anything from whatever the research scientists need to do in terms of, I don't know, like backprop, whatever, on one hand, but on the other hand, it's going to do GPU usage optimization. Is that fair? So it spans all the things that you need to do to develop faster and simpler.

Right. So I think the API kind of adopts Python as the programming language. That's why it's called PyTorch. The idea is from Torch, Lua Torch time. And Python is a very simple, easy-to-use programming language for many researchers. And they think about deep learning neural network in code. right? You need to think about that as graphs, as nodes and edges and how to tie them together. When this graph becomes like hundreds, thousands of nodes, then it's how you even kind of manage that thinking process. But represent that in code, in Python code, is very easy to manage. And then you become software engineer to think about a neural network. And it's very easy to debug because all Python tooling the toolchains can all work. So yeah, so that actually unblocks a lot of progress made in the modern innovation. And it also, of course, it can run on GPU, it can run on different hardware SKUs. We have many different hardware vendors plug in to a lower level to support PyTorch and then that including TPUs. So yeah, so And because Google has a compiler that can directly connect with PyTorch and lower the program into kind of the GPU runtime. So that enables significant advancement from a modeling research side. And the interesting part of modeling research is a lot of research has been published. As open source project. And Hugging Face is one example of adopting and becoming the ripple, centralized ripple for new research ideas. And it's very easy for people to take this model architecture. So there's a pre-GenAI and post-GenAI era. Pre-GenAI is the model code is open source, but there's no weights, right? Everyone has to curate their own data and train from scratch. That just means any company want to invest in deep learning, they have to first hire people who can, who understand how to curate data first and then who understand how to train, how to manage GPU fleet and so on. So that takes a lot of time to hire those people, which is come from very, very small pool and takes a lot of capital investment to get this going. So that's why pre-GenAI, not all the company can access this technology even though it's great. And usually concentrate in hyperscalers or whoever has big resource to be able to do it. And then post-GNI, GNI is basically built on top of foundation models which can learn from the world knowledge, internet knowledge. And these models are good by itself. So application can develop directly run on top of those models. So you don't need a machine learning team to begin with to start in imagining what are the new user experience. Or you can have a small machine learning team who just do fine-tuning or reinforcement learning, which requires much smaller samples to curate. And the post-training process is much simpler. So I would say that shift is fundamental. It's monumental to unblock a lot of new innovations in the application space.

Yeah, just to play it back, it's something that's super obvious to people like you at the forefront of the field. World, but that a lot of people don't quite realize this fundamental difference. And just to play back exactly what you said between traditional machine learning and generative AI is not just a different of technologies. It's exactly what you said. You had to build your models from scratch before, more or less. And now you have this concept of foundation models that you can build on top of. Therefore, the whole complexity has moved to how you deploy the models, which is what's going to lead us into fireworks. But just to play it back because it's such an important concept that people may or may not have internalized how that changes everything.

That's exactly right. So that's why we see an explosion of adoption of this technology from the inception of our company. And there are AI-native startups. There's no box around what's possible, and they create completely new user experience that we will never imagine. But also the enterprises, whether it's digital native or traditional enterprise, they are all joining this transformation and rethink their existing product experience or rethink how to power their internal productivity or customer service, many different surface areas. So I would say across the board, what's most surprising to me from the beginning of when I started this company, I had this sequential go-to-market motion in my head. Is to go to native startups first because they are most technical advanced and then digital native because they are more like they're embracing new technology in open-minded way and then traditional enterprise because they want to prove, has seen a proving point before enter. Now it's happening across the board everywhere. That's, that partly I will attribute to like GenAI models are so easy to use and so accessible.

And just to close on PyTorch, and then we'll go into Fireworks in detail as a passing thought. And that sort of echoes a conversation I was just having most recently with Dao from Contextual AI on this podcast, who was probably at FAIR around the same time you were. And we were just like talking about how incredible an impact FAIR, Facebook, Meta have had on the AI ecosystem. So PyTorch being another fantastic example of just open sourcing all of this fundamental framework that everybody uses and now is a fully independent project, right? I think the governance was transitioned completely out of Meta at the end of 2022 or something. It's just incredible, the impact that all of you had on this ecosystem.

Yeah, it transitioned to PyTorch Foundation. With expanded to industry governance. But I believe Mira still have a very representative presence there to continue foster PyTorch growth in the future.

So transition, but not completely transition.

Yeah, I think it's working. Yeah, it is a progressive transition.

How did your experience at Meta and then PyTorch influence the idea for Fireworks in the first place?

I think one of the biggest success we saw from the PyTorch experience is simplicity scales. And this simplicity is user experience simplicity. Because we have a very strong engineering team, we can solve any infrastructure challenges and complexities. But when it comes to adoption, people don't want to spend time figuring out how to make things work. They just want it to work. And that's kind of where PyTorch shines. And that's why in the heavy competition or very noisy market of every company is building their— was building their own AI framework, I think there are like tens of those AI frameworks at that time, but none of those were focusing on the simplicity part. Everyone was focused on production and all the nitty-gritty details to get into production. But to the point, it is very hard for researchers to adopt. And a kind of interesting funnel we saw from the PyTorch journey is the researcher adoption is the top of the funnel because they create new model architecture which get open source and they get adopted by industry practitioners, for experimentation. Maybe 1 out of 10 of those experiments will be successful, and then it goes down to the next phase of further full production. And when it goes to production, people start to transition from PyTorch into other production, more production-focused, optimized framework. But that transition is too hectic. They're going to rewrite the whole model in a completely different message, and then you lose precision. It's too much hassle, and it's much easier to just use PyTorch into production and let them optimize PyTorch. So we saw this journey. Now it's retrospectively, we saw this, how is this evolved. And at time it was not clear who is going to win and how it's going to win. But now it's very clear is to leverage simplicity to get to adoption because that's what end user cares about. And then we have, you know, we focus on optimize behind the scene and that's the way. To drive impact. So we carry the same mindset when we think about Fireworks is we want to keep the Fireworks user interaction really, really, really simple. And we take the heavy lifting of production optimization. And production optimization is so complicated. And I would like to elaborate on that. And so that's kind of our The founding principle of designing Fireworks product is simplicity first.

Because presumably at Meta to manage PyTorch, you had what, hundreds of engineers?

We have hundreds of engineers building PyTorch and the infrastructure around PyTorch. But at the same time, I believe PyTorch within Meta probably has thousands of users. Outside of Meta, it will be hundreds of thousands of developers.

But in terms of the infrastructure required to just manage something like this, you would need to be able to— if you were one of your customers, like with that Firework, you would need to hire dozens of folks or hundreds of folks maybe to just maintain and manage models, right? Is that fair in terms of what you abstract away?

First of all, hiring infrastructure like GenAI infrastructure engineers is a very small pool and it's also very deep. Optimization we do, you probably wouldn't be able to get anybody with our depth. But also we manage huge fleets. The scalability is very fast. We saw, for example, AI-native startups, they can ramp up extremely quickly without even being able to predict what's going to come next. We have seen digital native companies, they transition their online traffic to run on top of GenAI models also extremely quickly.

And famously, one of your customers is Cursor, right? Which is like the poster child example of super fast growth.

Exactly. And from what I know, everyone I talk to, they use Cursor. It's very interesting. So today I talked to one of our customers. They even use Cursor to schedule meetings. Very, very creative. So Another example is that we work very closely with DoorDash and Uber and Samsung. So they are incorporating GenAI into their main products because they are, you know, marketplace and the search relevance extremely important in order for them to accurately target and give recommendation to their end user, their consumers, they have to have high quality data and high quality supplier information, product information. All of those was manually corrected and GenAI actually can have model which can see, extract information and enrich through LM. So we are renovating their whole entire internal setup to have high quality end user experience.

Let's get into the weeds of the product itself. So starting with models, so you sort of offer power hundreds of models. What is the latest number of models that you offer?

Yeah, so our product has multiple layers. At the lowest layer, we have been working on distributed inference engine from the beginning, and that's basically a customized PyTorch. Inference infrastructure we set up for JNS specifically. And that distributed inference infrastructure was designed very composable and building block based. The fundamental reason is we want to launch any state-of-the-art model very quickly. As we just discussed, we have the track record of new model get released in the same day we launch it. Right. So that's because our— we kind of took the PyTorch idea and built our inference infrastructure in a composable way. And then we can pick and choose what is the best options and setup for that particular model. And but the search space is very big. So we think about First of all, our design principle is we don't believe in one size fits all. We don't believe in one size fits all for quality, as in we don't believe one model will solve all the problems in GenAI in the best way. That setup doesn't exist. We also don't believe there's one inference configuration that's best for, for all different use cases. Instead, we believe in one-size-fits-one. We believe in a platform which can automatically do customization across 3 dimensions. As I mentioned, application developer, they usually are concerned about these 3 dimensions. One's quality, one is speed, one is cost. And it's a 3-dimensional space, a 3-dimensional curve, and different product, different use case, they want to land in different spots and they want to they want to make different trade-offs. It's all use case specific. So what we want to provide to our end user is give them the option, find the best option based on their requirements, specific target at their workload and give them the optimal point.

And do you do that through, I don't know, consulting? You do that automatically? How does that work?

So we have a product called Fire Optimizer. It's a platform, it's a SaaS platform, and it took the inference workload for a particular use case and the objective function of across quality, speed, and cost where you want to land. It will do its own magic and spit out a new model model plus a new inference deployment configuration and go into the rest of the CI/CD infrastructure.

A new model, or it would pick one of the hundreds of models that you have on the platform?

So right now it's given a model and there are different ways to improve quality or make quality speed cost trade-off. For example, we have 5 different ways to quantize. We can quantize different parts of model.

Quantize means making models smaller.

Making model run in lower position where it's much more computational efficient or memory bandwidth efficient or communication efficient. So, but you can run instead of 16-bit floating point arithmetic, you can do 8-bit or even 4-bit. But you can apply this to different parts of model. You can apply to weights, you can apply to activation, you can apply to KV cache, you can apply to the communication part, you can apply to different parts and it will have different impact to the end, the sensitivity of the quality change. At the end, sensitivity of quality change, who is the judge? Product is the judge, right? So the end user is the judge. So we would like to kind of fit the best into what end user wants. So that's kind of, that's one example of, and we can do runtime quantization as in you still train, pre-train or post-train to the full position and then we can quantize on the fly during runtime or we give you tools to do quantization-aware training where you decide the position during the pre-training or post-training time and then you can generate much higher quality ability that way. So there are different levers we give our customer to tune. But also, for example, there are— we have different, 4 different, 3 to 4 different ways to do speculative execution. The idea here is to conceptually, you think about speculation as you have a way to predict not one token at a time, you have a way to predict 3 or 4 tokens ahead of time in one shot, and that's more efficient than predict one at a time, right? So because you can predict 4 at a time very accurately, then it's 4 times more efficient. So in the physical mechanism of doing that, there are so many different ways. We support many different approaches. One approach is pair a small model with a large model. Small model, right? And ask small model the question first and have it do the speculation. And if it's correct, then it's great. Then you get the small model speed. If it's wrong, then you ask the— you always ask the large model to judge. If it's wrong, then you ask large model to give you the answer, right? So basically the end result is you get large model quality and small model speed. In combination. So that's a very good example to, uh, to talk about File Optimizer, because in order for this to work, the small model, large model need to align towards your use case and workload pattern. If it doesn't align, then the small model predicts something, it always off. Let's say extreme case, it's 100% off, then you get the worst. Situation where you always fall back to a large model. You have the overhead of running both models, but you have the speed of the large model, quality of a large model. There's no benefit. So this alignment process is basically to make small model always predict higher and higher with higher and higher probability. If you can, for example, if you— we can— we have seen cases improving the prediction hit from 30% to 90%, and that's huge speed saving. So, so yeah, those, those are kind of another examples of options.

And do people on the customer side need to have the level of sophistication to understand those various options?

The idea here— so the number of options is big, right? So I just talk about two, two different areas, but we also have 5 different ways to partition the model and 6, 7 different ways to allocate different parts of model to different hosts in a distributed way.

Let's cover some of this quickly. This is fascinating. So what does that mean? Different ways of partitioning the model, you said?

Right, right. Because we look at the model in— let's take large language model, for example. Some parts of model is doing prompt, like processing the prompt. And some part of the model is predicting what is the possibility of next token. And during runtime, they actually bottleneck by different things. I'm talking about that extremely simplified way is prompt processing is bottlenecked by computation and generating the next, predicting the next token is bottlenecked bottleneck by memory bandwidth, right? So if you think about the model in one chunk and scale them all together, and then you always get stuck in one bottleneck versus the other. So the way we have seen similar kind of infrastructure problem in other domain like databases, and the way to kind of solve that problem is is pull those parts that bottleneck by different system resource to scale them completely differently. So if it's compute bound, then we should just add more GPUs to it, right? If it's memory bottleneck bound, then there's different way to scale it. If it's computation bound, then we make computation more efficient. So by scaling those different parts of model execution independently, then we remove all possible bottlenecks. And that's the most efficient way to run those models. So, so that part, I think we are on pretty cutting edge. I think recently we start to see open source, like people pay attention to those kind of setup. But at the end, it's a complex system problem. There are libraries that get open source to cover parts of the execution, but putting this all together and solving a lot of real system problem is the most difficult. I'll give you one example. Before I give you example, actually, I want to close the complexity on the optimization space. And all these different options add up together, it can lead into more than 80,000 possible, possible way to optimize. So, so then how do anyone even come up with the optimal point for a particular workload? And that's a search problem. Basically, we want to find the one out of these 80,000 possible options that is the best for a particular use case. So it's a search problem. And this is not new. This problem exists in areas like compilers, right? Compiler is solving a search problem. This exists in areas of database query optimizer. And speaking of databases, very interesting. In the early days, databases are designed to be very hard to optimize. And even there is a— it created a new career called DBA. It's a very well-paid career. It's a very well-paid job. People get certificate to qualify to be a DBA and enter the industry. But over time, database vendors start to build automation. They build query optimizer where it takes the query workload and spit out, rewrite the SQL and spit out the most efficient new way of running the SQL. But it's one-dimensional optimization. It only optimizes for speed. And here I'm talking about three-dimensional optimization across quality, speed, and cost. So it's a much more complicated problem, a much more complicated search problem we're solving through File Optimizer. And that's kind of the value add we want to— we want to free up app developers to not be able to need to understand any of this complexity and get the best quality, speed, and latency and cost. For their new user experience.

That search problem, you solve it on a per-task basis, so sort of a priori before for the specific use case, or are you going toward, or maybe you do it today towards real-time optimization, right?

So we have two modes similar to agentic development. I think agent is a very hot topic these days.

Yeah, I'm hearing.

Yes, everybody's talking about agents. And there are basically two extremes of think about agent development, right? So one extreme is autonomous, right? The goal here is to replace human and generate the same quality or even better results than what human can do. And they are tapping into human resource budgets of different professions and so on. So that's one extreme. The other extreme is human assist and to make the profession, the professional become much more productive. Right. So similarly, when we think about this optimization, we also have two modes. One is human in the loop. The second is more automated. So human in the loop that we have, take DeepSeq for example. DeepSeq is a very, very complicated model architecture because it has more than 250 experts. And DeepSeq, actually, the company itself was running and still running this model over more than 300 GPUs. So think about this deployment. One replica is 300 GPUs, and there are so many different, so many more replicas. It's a very big distributed system problem to solve. So that's why the model itself is very hard to tune. So we enabled supervised fine-tuning for DeepSeq where people be in the loop to label, give labeled data and feed into supervised fine-tuning. We have seen excellent results tuning DeepSeq models to customize to a specific domain. So that's human in the loop.

And not to go into too many different rabbit holes, but that's the reinforcement learning with rewards.

So reinforcement learning reward, we are working on the automated flow because they are— so we actually verified reinforcement learning with verifiable reward is very simple version to do reinforcement learning. And there are many domains getting the verifiable reward is simple, is easy. For example, coding space. It's very easy to do grammar check. And then it's very easy to run the code and see if it's runnable. Does it produce right results or not? We have also seen other domain in more like vibe base. It's not like absolutely correct or math is absolutely correct or not. Vibe base, people start to develop a good practice of providing some kind of API where there are, for example, designers sitting behind the scenes and to rate certain kind of design, whether it's good or not good. And that can feed into the reward to automate. So we start to see very active development of providing those reward functions. Whether it's programmable or in a sense human loop. But I believe that's the direction that the industry will be moving towards. And we want to build out this stack in the way we don't want to be opinionated how different companies want to plug in their reward function, but we want to make integration extremely easy and the rest of the feedback loop should be automated.

It seems that another important part of the platform is function calling. You want to talk about what it is, how you do it, and why it's important?

Right. Talking about that, we need to kind of talk about agentic development. So on GenAI, I think the first wave of application development of GenAI has been focusing on building user experience for humans to consume. Because the nature of the content, the results being generated are human consumable content, right? So, and the model was trained to generate optimized for human consumable content. So building agents is more complicated than being direct human facing because those models need to speak to each other. Or connect with external APIs to enrich the results. So the fundamental need here is to make the model output feed into some kind of programmable framework. That's a fundamental change in this new agentic development. So first of all, we need to teach model to generate. Programmable output. It's called constrained output generation and to follow certain schema. And usually it's JSON format for most of API integration. But we have seen many other customers, they want to generate XML, they want to generate HTML. Those are all schematized formats. So we support a new mode called grammar mode. That is any BNF format we can support. It's very generic. So, so that empower developers to extend beyond JSON. That's the base foundation. And on top of that foundation, then there are two different ways to build, to like satisfy agent development needs. One is people build orchestration layer to pull multiple models. Here, multiple models came across multiple different modalities. I can talk about examples there, or integrate models with databases, with search engines, with knowledge graph basis and so on and so forth. So they keep full, take full control of the orchestration scheduling. The other approach is expand LLMs, let larger language model dynamically orchestrate, create a plan, create execution, and tie the result back and continue use, serve the rest of the answers. But LLM is the main brain. Behind this. So these are two different modes. For the first mode, we, we provide a composable framework, especially across multiple modalities. We have seen use cases, for example, medical claim, medical claim processing historically has a lot of legal involvement because lawyers need to review people submit pictures of their medical bill and there are a lot of patient information. It's very complicated document processing and they have to review a lot of information and then they will need to generate a report that follows some kind of format, legal format to decide go, no go. And you know, what are the implications and so on. It's a very lengthy process and we know like lawyers' time is very precious. And expensive. So, but significant part of this process can be automated, combining a vision model which can read and see with a large language model that is tuned towards generating legal reports and composing them. And you can pick and choose which vision model you want to use because as I mentioned, no one-size-fits-all. Our prediction is the future will be a lot of expert models focused on solving certain problems really well. There are vision models really good at reading handwritten documents. Their vision model really good at processing PDFs, processing tables, weird tables spanning across pages and so on. So there are a lot of complexity in the document format. And there are language models really good at generating summarization in general or specializing in legal terms or specializing in finance terms and so on and so forth. So this composability allow you to pick and choose the best best and solve your use case the best. So we have this, we have this multimodal composable framework. So that's one for you to orchestrate. And then when it comes to large language model to do dynamic orchestration, that's where tool use or function calling come into picture. We have been working on function for calling for a long time and we are the first one to enable function calling for DeepSeq model. Models. And DeepSeq model come in as a raw model and we add function calling to V3 model, which create a lot of interest.

That was recent, right? That was announced just a couple weeks ago.

That was announced recently. And many people has been using it and give us very good feedback.

Let's talk a little bit about the low-level architecture part of the platform, in particular the GPU layer and the work you've done different CUDA?

Right. So we have been investing in overall the speed and the cost optimization. There are two sides of the same coin, shall we say, for a long time. And from the PyTorch days, and we actually have a lot of customized kernel. We did a lot in-house and are not afraid of, we actually wrote PTX. So a lot of kernel build on top of CUDA, but we also build kernels at the lower level. You think about PTX as writing assembly code. And those are extremely high performance kernels to maximize, to kind of maximize software performance towards the hardware limit. So that, yeah, so we have been building that for large language model. Of course, large language model model now has different classes. Mostly the biggest bucket are chat models and logical reasoning models. They have different characteristics. And then we have been optimizing for communication, any possible bottleneck that we have optimized for.

And the GPUs themselves, they sit on some cloud, or how does that work? Do you run on all sorts of Nvidias and AMDs, or you agnostic that way, or how How does that work?

Yeah, similar to PyTorch idea, we want to provide, we want to build on top of the best hardware possible and the different hardware skill has different strength. So we build on top of NVIDIA GPUs from the many, many generations to the most recent one, B200. We also build on top of AMD's hardware. And we optimize on top of ROCm is their CUDA equivalent layer. So yeah, so we have been deployed to many, more than 15 regions globally. Currently we're on top of 5 different clouds. We plan to expand to possibly 10 different clouds and a lot more regions globally. And in terms of deployment, we have our hosted, managed, fully managed single tenant API. We can also deploy into your VPC and into your cloud for privacy concerns, or we can connect VPC to VPC through PrivateLink or VPC peering. So all these possible way of deployment we have enabled for various different enterprises and startups.

Are you seeing a robust demand for, for, for the latter, meaning enterprises want wanted to deploy AI in VPC, on-prem, perhaps even locally? What's your sense of the market demand? Right. That's this whole thread of conversation around the fact that people just don't want to send their sensitive data to OpenAI or any, not to pick on them, but any kind of API provider. What's the reality of that from your perspective based on what you see in the field?

I feel there are market demand or constraint that's not movable. and there are constraints that's movable. So I think the default mode when we work with enterprises, the default mode is no data can leave our premise for good reasons. But at the same time, the market is very interesting. So first of all, model depreciation has never been faster. And every week, every week. And you You never know tomorrow what's going to happen.

Quite literally. It's not an expression.

So catching up with this fast-moving train of like model innovations happening across all different modalities, all different space is really, really hard and interesting. And the second is hardware depreciation is happening also very fast. In the past, every 3 years there's a new hardware. That's kind of the cadence we are thinking about. New SKU coming up. And this year there are 3, like each vendor probably has 3 SKUs planning to, you know, get into production. And with the moving target of new model innovation and new models typically optimized for newer hardware and new hardware advancement, and enterprise is asking us, hey, it probably is better to use our hosted API because we always move towards the top of the state-of-the-art models, state-of-the-art hardware, and they want to get onto the state-of-the-art technology. So we have seen the trend of enterprise moving to our single-tenant, heavily secured hosted API more and more. So that's very interesting. While we still deploy into their cloud, their premise, to run high production traffic.

From a business standpoint, there is this wide expectation across industry that the cost of inference needs to keep going down and will keep going down. What does that mean for you in terms of business as an inference infrastructure provider? Is there an expectation that you're going to be cheaper and cheaper and cheaper? And if so, how do you build your business over time.

Right. So I believe, I believe this overall AI infrastructure would go down by order of magnitude in terms of cost. And, and they should. Where it is today, um, is— I, I don't think it's a stable state to be where it is today. And this whole trend of infrastructure should become a utility is really, really good for the industry because today we see a lot of case that I think today they're the infrastructure cost basically start to separate out two different concept. One is product market fit, as in is this product useful? Does this product add value? It is a different concern to can you build a sustainable business? The fundamental reason is even if people can have a product-market fit, they get really good feedback from their users, they cannot scale it to the maximum degree because this is so expensive. And we literally have seen cases that a startup can scale to bankruptcy. And for big companies, for very big companies, companies, uh, like the GPU bill surfaced to their CEO and they're like asking questions, what the heck is going on here? Why we're burning so much money? And we should really think about ROI here. So because of that high cost of infrastructure, AI infrastructure for GenAI, it basically creates a high bar of what kind of application can be a viable business. If this bar can be lowered by 10 times, you can imagine there's so many more it will be 100 times more applications enter this arena to create a brand new experience to end consumers and prosumers. And by that we'll see a much bigger consumption across the industry. So I don't think I worry about my business. I think actually I want this cost to go down. It has been our primary focus to help application developers to build a sustainable business through Fireworks because we hyperfocus on cost efficiency, not just beyond speed and quality. So from total cost of ownership side, they can have a durable, they have a long longevity of a durable business. And that's what we care about.

So maybe to close, some forward-looking future-guessing kind of thoughts. So one, for Firework specifically, in the next 12 to 18 months, what's on the roadmap? What are you excited to build that you can talk about today? And then for the broader industry, any thoughts for, you know, where things may be going from your perspective?

Yeah, I'm super excited. We are doubling down on the File Optimizer. It's our customization engine across quality, speed, and cost. We have made a tremendous amount of progress in the past 1 year, and we are getting that to massive production with many customers, they now have a viable path, path to have product-market fit, both product-market fit and durability. And we are, we are going to heavily focus a lot more for File Optimizer on the quality side because we're already doing really well on the speed and cost and we want to maximize the quality gain that through customization, the quality should be much better than they build on top of any generic model, any generic model API because they shouldn't, right? So if you fuse use case specific data to it, it should just shine. And for application developer, I think fundamentally what is that mode, right? Their mode is probably not the user experience because it's very easy to copy. Anyone can study the product and copy. The mode is data is the more— the more is the data they curate through or the insight of the usage, like usage pattern and kind of create this flying wheel. And this flying wheel should feed back into the GenAI model they use. And we want to kind of really double down to make customization heavily push on the next level quality. So I'm super excited about that level investment. And we will launch a series of product features along those lines.

And then any thoughts on the future of the industry at large? Like the next 12 to 18 months, is that the year of agent that everybody's talking about? Is that the year of adoption? What's your prediction? What are your predictions?

So there's no doubt 2025 is the year of agents. Literally, no matter where you go, people are building all kinds of agents. So I think there will be thousands of agents being built focused on solving specific problems. At the same time, I also think 2025 is the year of open models. And it's clear that DeepSeq has created a big dent there, but the meaning is not about Deepseek itself by itself. It's the precedent it has set in the world about setting a huge, like a very solid baseline for open models, for open model providers, that it just needs to be better before anyone can open source new models. It also set a precedent for all model providers to be better. I'm very bullish on open models because I have seen the power of open source that PyTorch has been able to leverage. Deepseek, for example, just within one month of releasing their new models, there are— despite Deepseek model extremely hard to tune and optimize, extremely hard, there are 500, more than 500 variants published on Hugging Face, optimizing for local device, optimizing for cloud infrastructure. People tune DeepSeq model, people distill all DeepSeq model into various different kind of models. Companies like Perplexity also tune DeepSeq model for—

1776.

Yes. And we launched that model in partnership with them for much unbiased, accurate information for deep research. Link is another customer of ours. They focus on— they launch a DeepSeq model, tuned DeepSeq model for financial assistant they're building. So we see a lot of vibrant energy from the open source community. And that's why we believe in the future of modeling sits on open model side. And I believe that side is going to be much more active in creating those hundreds or maybe thousands of expert models that is specialized delivering much better quality in certain domain. And the interesting challenge then is between those active development of agents and active development of open models across these two, there will be heavy spaghetti in the middle and that complexity of navigating and doing the last mile quality alignment between these two and deliver real-time experience for the, for the Agentic user to consume is going to be the next level challenge. And we want to like solve, like take over, solve a big chunk of those complexity and simplify it for the Agentic developers in the future. And that's also going to be our this year's focus for fireworks.

So do you tell your friends and parents that you are in the spaghetti business?

We are. We love spaghetti. We love to simplify spaghetti.

It feels like a wonderful place to live in. Thank you so much, Lin. This was terrific.

Yeah, thank you.

Hi, it's Matt Turk again. Thanks for listening to this episode of the MAD Podcast. If you enjoyed it, we'd be very grateful if you would consider subscribing if you haven't already, or leaving a positive review or comment on whichever platform you're watching this or listening to this episode from. This really helps us build a podcast and get great guests. Thanks and see you at the next episode.