Putting AI into Production with Fireworks AI's Lin Qiao

Hey friends, we have some news from our sponsor, TextControl. They just released version 32 of their document processing library, which includes new core functionality such as document footnotes, SVG export, and more. You can integrate document editing, signing, collaboration, and PDF processing into your ASP.NET Core and Angular applications with TXD, TextControl. These powerful libraries will let your developer teams focus on their core competencies while TextControl handles your digital document processing. Check out all the new features and see the technologies live in action by visiting the demo at demos.textcontrol.com. That's demos.textcontrol.com. Hi, I'm Scott Hanselman. This is another episode of Hanselman Minutes. Today I'm talking with Dr. Lin Qiao, CEO and co-founder co-founder of Fireworks AI. How are you today?

Hi Scott, I'm doing well. Thanks for having me.

Yeah, I'm very excited to chat with you today because I'm really trying to understand these next generations of startups and medium-sized companies doing AI. Because I think it's worth noting that before you were the CEO and co-founder of Fireworks, you worked at big companies that are doing big stuff. You worked at Meta, You ran a large team of engineers doing AI. You worked at LinkedIn, earlier years. They must have been doing work in AI. And you also worked in research at IBM. But rather than building the thing you built at any of those companies, you chose to leave a giant company that's making big moves in AI, referring to Meta, and do your own thing. What, what, what's, what's happening in one's brain? How do you, how do you decide I have an idea and I can't do it at giant company, I'm going to do it at startup.

I think I need to be crazy enough to believe the direction I'm heading on is going to work. So I think I'm fortunate enough to be driving PyTorch at Meta for a long time, for 5 years. And here the focus is kind of really driving PyTorch from a research-oriented project for researchers into fully full-blown in-production. And it matters, production scale is huge, right? So, and there's a variety of applications and products building on top of AI, on top of PyTorch, on top of infrastructure we built from ground up for PyTorch. So that path took us 5 years. And then PyTorch at the same time is a great open source project. I think the team did a fantastic job driving the open source engagement. So that, uh, I'm fortunate enough to work to see many other companies, more traditional enterprise companies in the industry, start to, uh, to start to get on to this AI-first move. I still remember, like, we, we work with Walmart, we work with Disney, we work with John Deere. Um, they, they are all using AI to, to innovate. It's very interesting applications. Like, John Deere was using AI to to take pictures on the field. There's a machine rolling through the field and they detect random stuff on the field, like weed, and they just unplug them to clean up the field before they plant things. So that's really interesting. But at the same time, we also heard a lot of feedback like they don't have the right software to enable AI, they don't have the right hardware to enable AI, they don't even have the right team. They are more looking for us to, as kind of helping them to move faster And it's very clear to me the whole entire industry, majority of them are in that state while we are in this AI-first move. So the big contrast between the need here to kind of go faster, use AI to enable, deliver a lot of business impact and value to the lack of supply, lack of supply of knowledge of how to do it properly. And we feel like my whole entire founding team, we are impact-driven. We have been impact-driven for a long time and we're gravitated towards bigger and bigger impact. Of course, we can stay in Mara and there's no question there's huge impact to land. But from our observation, driving the whole entire industry going through this journey is much, much bigger. So that's kind of why we started this company and built the business from ground up.

When, when big companies— I think of big, I mean talking like, you know, very large, no pun intended. There's very large language models and there's very large companies that make them, uh, you know, they get a lot of press. Meta gets a lot of press when they release a model, uh, Microsoft gets a lot of press if they release like Phi-3. Um, where does Fireworks fit into the stack of small, large, all these different kinds of models? Are you the top of the pyramid that just makes them all friendly, or where are you in that that, that stack, right?

So there are multiple ways to look at this. There's also, when it comes to GenAI, there are companies building infrastructure for training, right? So, or inference. So we actually can choose to do both because PyTorch has been— we have been building infrastructure for training using PyTorch or inference using PyTorch at Meta scale. But we decide to focus on inference. The reason is Gen AI as a technology, I know fundamental disruption here is the content generated through those AI models are very similar or even surpass what human can generate, um, and that will create a whole slew of innovation in application and product space. So by nature, those innovation will be more consumer, prosumer, or developer-based by nature, and the matter is the biggest, I would say, B2C company. And, and, and we, we have seen that, uh, in those kind of B2C company, or like to, to see, or to pursue more to developer space, um, your inference challenge or inference band is going to be proportional to your end user, which the limit is the population, uh, on Earth, um, versus the training you are going to be proportional to the number of researchers. In relative speaking, that is much, much smaller scale. So we decided to kind of solve a bigger scale problem and laser focus on inference. The interesting transition that whole industry is going through before and after GenAI, there are a few things. Again, our target audience are app developers and product engineers, right? What doesn't change is All this to see, to person, or to developer application, it needs to be hyper interactive. And as a matter of fact, the response time starts to kind of shrink smaller and smaller because, hey, that's how we get used to React faster and we like those kind of experience. And today, GenAI is on the highest spectrum of the size and complexity, and getting this fast, extremely fast experience is really hard. And that breaks the product experience. And the second is when those companies, they hit a product market fit— actually hit a product market fit is not easy. Once they've hit the product market fit, they have to scale to large corpus of users, and then if the cost structure is wrong, they quickly bankrupt. It's literally, they have to kind of— and they didn't expect, oh my God, I'm running out of money. No matter who you are, you have deep pocket or not, you can run out of money pretty quickly. That's a common complaint we have heard about, and they have to press on the brake. And even they have a viable product, they cannot launch to have a viable business. So we first heard these two problems, and we're like, hey, we have— we're going to solve this problem first. Be the fastest and the most cost-efficient inference engine. I want to pause here and kind of see.

Yeah, I want to break down some pieces because I think that when the audience listens to yourself and people talk about AI, they're still often trying to catch up. Because very— there's a lot of startups that are basically just web apps that talk directly to the OpenAI endpoint with— and it's very simplistic. Anybody puts together a chatbot or a completion, and you're very consciously at Fireworks using the term inference engine, making inferences, drawing conclusions based on evidence and reasoning. It's not just completions, it's not just chats. Why did you focus on inference as being the most interesting problem to solve before we talk about scale and pricing?

Right. So this is the kind of choice between training a foundation model or training a generative model versus you have a generative model and you just ask it questions and then bake the answer into your application to power new product experiences, right? So, so those are the two choices. Among these two choices, we choose to, uh, to drive, uh, the platform to deliver much faster and more cost-efficient serving, as in you ask a question, get back result. That's how you interact with ChatGPT, for example. We basically provide the same product line similar to ChatGPT. So we are not solving the training problem, because it requires a different kind of setup where you have a lot of researchers, you have a lot of data, and you have a lot of compute, right? So typically what we observe is, and that business is really, really, really challenging. The fundamental reason is the model depreciation cycle is very fast right now. Almost every week there's the new model, someone launch a new model. That is better in like certain benchmarking results. And from the model builder point of view, they invest like months, quarters, or even sometimes years of research and tons of compute resource and get to this great innovation out, but soon it got subsumed by somebody else, right? So the model, uh, model is depreciating really fast. And second is compute depreciation really fast too. Um, usually in the past, the life cycle of a SKU, usually one hardware vendor, they have a new SKU out every 3 years or 2 years, right? That's kind of the cycle. And now we are, we're seeing how new SKU coming out like multiple times a year, multiple SKUs a year. Uh, and then there are a lot, several hardware vendors are like competing, uh, in this space. Um, so like hardware depreciation depreciate really fast also. So that makes kind of the modeling space for training really, really challenging. But GenAI is the notion that the concept here is the following, right? Before GenAI, for AI, before GenAI, any AI model you have to train from ground up, from scratch. As in, if you want to use AI, you first have need to have a machine learning research team to curate data, curate lots of data, and then train the model, right? And make sure it converges. And then there are a lot of data issues, you need to kind of go back. So that's a lot of investment. Now we have GenAI. GenAI, the concept is it's a foundational model. Foundational model as in someone else built this foundational model. For example, Meta keeps releasing really great Llama models. You don't have to do the same thing. You don't have to train from scratch. You just need to use it as is or infuse your data to align the model better towards your problem, towards your data distribution. And that alignment process is very lightweight. It's called fine-tuning. So that makes this technology much more accessible. Just GenAI is much more accessible to a broader set of developers compared with traditional AI, where a company needs to decide to found a substantial team, such a reasonable-sized team with deep talent to be able to do this model training. So with that said, because we focus on GenAI space and we basically leave the training problem to those research institutions more and more. So now it's kind of basically two camps: closed source versus open weights. And of course, we bet on open weights heavily, and we believe in that direction because I, I've been kind of on the open source side through the PyTorch journey in the past 5 years, and I deeply believe that. And then Fireworks is just focus on enable those state-of-the-art models, state-of-the-art GenAI models in the serving tier. So enable lots of ChatGPT-like apps and products to run on top of GenAI through Firewalls. So that's our mission.

So if I'm a startup who's listening, who is maybe calling Azure or another host, or maybe they're calling OpenAI endpoints directly, and they've already picked their model, they've gone in there and maybe they're using Llama 3.2 or they're using Llama, you know, 3.1. If they put— if they switch to Fireworks, they're getting what? They're getting lower cost, massively lower cost, massively higher throughput, and, uh, lower dollars per token. How? What is the secret herbs and spices that you're putting in between? It's almost like— I imagine an analogy, this may not be a good analogy, but like In the old days, we would put a virtual machine on the open internet. You'd open a port and people would call that virtual machine. And right now, you know, OpenAI has this API and everyone pretends to be OpenAI. They are an OpenAI-compatible API, but you would not today in 2024 put a virtual machine on the open internet. You would put a caching system in front of it. You'd put a reverse proxy. You'd put a series of layers to protect it on the input and on the output. Is that what's happening with large language models? You would not put a large language model on the open internet directly. You would need to wrap it in safety and caching and throughput and things like that.

That's a very interesting analogy. I really like it.

You can use that.

Thanks for the idea. So there are a few things, right? So compared with Azure or AWS or GCP, So first of all, our inference engine, because we wrote PyTorch code, right? A lot of PyTorch optimization. We're very kind of, we're the domain experts here. And all these GNN models are PyTorch models. So, and we basically have a special PyTorch version for GNN models. And here, of course, we also wrote a lot of low-level like CUDA kernels, ROCm kernels, AMD CUDA kernels, NVIDIA. But we also have a special orchestration and distributed execution engine that we implement inside of PyTorch codebase, but which is like specialized for GenAI. And we also implement various different level of caching. That's why I like your analogy here. If you can hit, if you have a high cache hit ratio, as in the prompts are similar to each other, then you don't have to recompute every time because this is really expensive. So leave that aside. Our specialty is actually a different thing other than, hey, we have our own optimized PyTorch runtime for GenAI. We specialize— our observation is every person, every developer, every enterprise, their inference workload is different from each other. Everyone is different. And if we treat everyone the same way, then we leave a lot of latency optimization, cost reduction, quality improvement, all on the table because we don't differentiate across the board. One analogy I want to use here is database has been a well-understood area. It has— it is declarative. It has SQL as a language where you describe what you want to get out of the database, but you don't describe how to do it. And a database engine take your SQL query and then go through a process called query optimization. This query optimization is gonna look at what's your workload, what's your query look like, what do you want to do, and generate the most optimized plan for low latency and low cost. So same idea here. Everybody's inference workload, as in kind of the, the request you send to ChatGPT, for example— is different from each other. It really depends on use case, depends on your internal datasets, and so on. We can heavily personalize or optimize Infrangeant towards your workload, and that's called Fireworks Optimizer, Fire Optimizer. Similar idea to Query Optimizer for databases. And that makes us stand out from everybody else is we do special, like, customization and personalization to the extreme. And by using File Optimizer, we let you choose. So you need to imagine here the design space is multi-dimensional across quality, model accuracy, quality, latency, and cost. It's a 3-dimensional trade-off space, and there's a curve you can get to. And then everyone wants to choose a different point in the curve to optimize for. We expose this curve to them, but everyone's curve is also different because your workloads are very unique and distinct from each other. So we basically present this personalized curve to everyone, and they can pick and choose which point they want to land. And that's Fire Optimizer.

The, uh, that reminds me of the old software engineering joke that you can have it good, fast, or cheap. Pick two.

We want you to pick three.

Yeah, you want to pick— I would love it to be high quality, low cost, and highly efficient. But by exposing that curve, you're allowing them that level of flexibility because, um, you know, right now a lot of I think we as an industry are still discovering what the UI should be for this. For the customer, if you look at like Bing Chat, they say precise versus creative. And for a developer, they might have a temperature slider. And if you go to the OpenAI playground, they allow the temperature to go to 2, which one could argue has zero value. But if you go to Azure OpenAI, they don't even allow temperatures above 0.8 because they've made a declaration that irresponsible things happen at temperatures over, over 1. So when I went into the Fireworks developer playground where folks can go and take a look at fireworks.ai and explore, you know, you expose a lot of those sliders and those, you know, knobs and allow people to experiment like anyone, but you do allow for potentially high temperatures, and, uh, I don't know if a high temperature necessarily is a creative— I don't know why we decided that that was the creativity axis.

Yeah, so I like— it really depends on your use case and application, right? So for some use case, for example classification, um, and did they use, uh, LLMs to classify, hey, what kind of, um, what, what kind of weather like for example, product catalog, right? What kind of category should that fit into? And they need a very— like, you don't want to hallucinate, you don't want to kind of build based on probability, you want to kind of this very, very accurate deterministic behavior. So you kind of temperature zero, no, no kind of creativity is needed here. But sometimes you want to— we have seen kind of use cases like You want to kind of paraphrase some kind of full email, right? So it depends on different role you want to play. If you want to make it more interesting, more unique, or more businessy or professional, then you kind of— the temperature does play a role in adjusting the outcome here.

Interesting. So You, you call out on the site that it is the best way to go from prototype to production. What are things that you think people are not thinking about when they go into production with language models? My first thought is responsibility to prevent problematic things coming out of the models, uh, when problematic things are maybe sent into them. What kind of safety and where's your stance on when it comes to what we would call responsible or, you know, low bias or bias-free AIs?

That's a really big question. I think that comes to actually like different model providers. So for example, we work with many model providers. Some are even proprietary models hosted on Fireworks. Everybody is putting a lot of effort in safety features. Like recently, Matter announced launch Llama 3.2, where they're launch partner. And I, I gave a presentation in Meta Connect, um, in the day of launch, uh, and there's a backstage and we, we just chat about, uh, different things. And, and clearly the conversation becomes— quickly becomes how to ensure safety, right? There's a lot of effort Meta put into the model itself, uh, because they don't have a hosted API they can control and the kind of model need really have the guardrail. So it goes into the kind of post-training, doing a lot of work to make sure the model just doesn't spit out bad things. Yeah, so I think that is kind of one thing is kind of the model provider need to do a lot of work. And I do see across the industry, people are holding a higher and higher bar. The second is, I will say, Safety, security in the enterprise, for different enterprise, for different industry segment means different things. Oftentimes, the industry themselves has their own guideline. That just means you have to customize the model to follow your guideline better. That's where it goes back to File Optimizer. We don't believe in one size fits all. And there's a lot of, like, we do see the future trend is everyone can easily customize the model to fit in, not just make it better quality for your workload, but also for the safety reasons, right? So you may have a different safety guideline from those model builder because it's similar to model. When you build a model, you make assumptions what this model is good for, right? Similarly, when you view the safety, guardrail, you make assumptions what the safety guardrail should look like. But different region, for example, has different bars and different needs, different industry segment also. So I believe in heavy customization. So you can add, adjust, and enhance your safety guideline accordingly.

Hey friends, in the interest of me learning more about AI and helping you learn more about AI, for the next couple of episodes, we're going to have Lendert van Doorn from Qualcomm. Help me understand AI. What is an NPU and why is it important today? Ah, so the NPU stands for Neural Processing Unit, and it's basically your AI accelerator. And so it can execute AI operations, these large language models, or all sorts of different generative AI models. In a much more efficient way than a CPU or a GPU can. So just to give you an example, with Microsoft's Recall, you run that continuously in the background. Running it on an NPU takes about 30 minutes of overall battery lifetime. If you were to run it on a GPU, it would take 6.5 hours or more. So it's all about energy efficiency. Very specifically dedicated to a single task, which is AI in this case. Very cool. So I'm going to go learn more about NPUs and how they can do things more cheaply from a battery perspective, more efficiently, and why I might want a Snapdragon processor in my next PC. Exactly. Thank you. And folks can check out more at qualcomm.com/developer to download those things and learn more today. Does Fire Optimizer— is known for this speculative decoding, does in running things in parallel and you're getting a really high hit rate. And forgive me if this is a dumb question, but does that have anything to do with, or does that guarantee a higher quality inference or a less problematic one if you're doing that kind of speculative decoding?

Right. So speculative decoding in a nutshell is basically you think about a pair of models. This is just analogy, right? So, sure, sure, a pair of models, small, big, right? Uh, and then by running them together, you get the latency of the small model, you get the quality of the big model. Basically, you get the better of both worlds, right? So, but for this to be effective, the, the small model need to answer most of the questions, right? And then sometimes, like, some— of course, it's— the granularity is not that question level. I'm just drawing analogy here. But sometimes if the small model is not good at giving answer, correct answers, aligning with big model, then you— most of time it becomes a big model answering the question, and then you don't get the small model latency, right? So basically, it's a small model hit ratio need to be very high. And then how to increase that, right? So Basically, the small model and big model, they need to align. The more aligned they are, then the more likely you'll get the benefit from both. But alignment is not done in, in vacuum. This alignment needs to happen in tandem with your inference workload. So both need to align with your inference workload. Basically, for specific narrowly defined problem you want to solve, Both are good at answering those, and then you get the benefit of both. You get the lower latency from small model but higher quality from big model.

So, okay, that makes a lot of sense.

Model Optimizer is, uh, provide an automatic way to do that alignment. Uh, so basically you don't need to even worry about that. We take care of that for you, uh, as long as kind of you use Fireworks inference engine.

Yeah, I know you, you've probably learned this about me in just the short time we spent together, that I'm all about analogies, uh, because if I can understand something in another world, then maybe I'll be able to understand this new world of AI. This reminds me of P-cores and E-cores, performance cores and efficiency cores, and the idea of asymmetrical design on a PC where you've got these processors that are going to use lots of power and lots of wattage and you're going to get great fast throughput, but I'm just checking email, so I'll just do that work on the E-core. And with asymmetrical design, you get effectively, if you do it right, the best of both worlds. And it feels like you are describing a kind of asymmetrical design in this, uh, in the speculative decoding, right?

But this asymmetric design, we can— it's also adaptable. It will adapt. So imagine your, um, your E-core and H-core you mentioned.

Yeah, P-cores and E-cores, efficiency versus performance.

So they are going to adapt based on what kind of workload you send to it.

Right.

It can adjust itself, right? Of course, hardware is impossible, but kind of imagine if that happens, that's a killer feature. So, so yeah, that's what we do.

Right. And the magic on the— in my analogy is the kernel scheduler and the magic in your analogy is the Fire Optimizer.

Right, exactly right.

And I like that you kept saying, and then you don't have to worry about it, because this is the thing, right? You're a customer service representative. You're trying to help people with customer service chat problems about your specific domain, and you've selected a very large language model that knows how to make songs about Star Wars in the form of Shakespeare. And none of that has anything to do with the business impact that you're trying to deliver. And that's why You want the quality of the large language model, but you want to ignore all of that extra corpus, that unnecessary information, and focus on the fine-tuned goal of your business.

That's right. So that's why I think at the beginning you asked a question, right, about small model or big model, right? I'm a firm believer that in kind of in the business world, when we solve business problems, We absolutely are going toward direction of small models, and as a matter of fact, it's going to go smaller and smaller because a business task is a constraint problem. It's not open-ended. Let's solve, like, let's solve a math, like, very hard math problem or anything. So it's very specific, and exactly as you said, when the problem is very specific, narrow-defined, you can throw away other things, and you don't need, like, um trillion parameter generative models, so like trillion parameter generative models, so instead like billion parameter or even sub-billion parameter generative models will be able to do equally good job for a specific narrow defined task. And then the rest will become customization. Small model plus customization is going to be the killer.

You know, I've— we, we as a, as a culture in the news cycle, as we watch the personalities, for lack of a better word, at OpenAI have their personalities, we have this fascination with the largest models. And Meta has been leaning in on saying massive context window, unlimited context, you know, paste your entire life into it, that's your context window, billion parameters. But I think that the really interesting thing is going to be what can I do privately on my own machine, on the pocket supercomputer I have in my phone, on the NPU that I'll have in my laptop. I am disappointed in our fascination with very large language models that we pretend are AGI, when I think really interesting work happens in a group of small models talking to each other and voting on things.

That's right. That's exactly— that's what's happening in the industry, right? So we have seen the power of small model can do big things. And again, a small model combined with your data asset and basically enable the developers and enterprise to have a path to convert their data asset into AI assets. And channel that value into their apps and products. So that's the transition the whole entire industry is going through.

Yeah, well, this has been really, really educational. I feel like I've got even more things to dig into. I'm going to continue to play at the Fireworks AI dashboard. Folks can check out fireworks.ai, just click try now, log in, and you can explore. They've got all the latest Llama 3.2 models available now. You can read the blog post and hit get started right there on the home screen, and you'll be playing with the platform very, very quickly. Thank you so much, Dr. Lin Qiao, for chatting with me today.

Oh, definitely. Thanks for having me, Scott.

This has been another episode of Hansel Minutes, and we'll see you again next week.