Transforming AI | Inference and Fine-Tuning | Lin Qiao on Fireworks AI

How do you see the evolution of agents and the agent workflow?

I think the future of GenAI app will become autonomous. In the future, agents are going to be self-organized, self-coordinated, solve problems together. We'll have a system it will automatically route and direct requests to the right agent API to process. So then the control is going to be distributed. I think that is definitely going to come. In that world, you can imagine there are multiple hops to solve a problem and the latency becomes even more important.

Why would someone use Fireworks AI versus any other inference as a service offering?

I, I think the biggest differentiation is we heavily focus on production.

How do you ensure there is less latency or low latency for customers across the globe?

That's a really good question.

Ladies and gentlemen, welcome to another episode of The Encounter, and today our guest is Lin Qiao, the CEO and founder of Fireworks AI. And I'm really glad to have Lin on my show. Lin, welcome to The Encounter.

Thanks so much for having me. Super glad to be here.

Awesome. So Lin, you worked at Meta before you started Fireworks AI. And I'm seeing the market with many players offering inference as a service. So tell us about, you know, how you actually started Fireworks AI, what was the motivation, and what is the differentiating factor when it comes to Fireworks AI?

Sure. So me and a few of my co-founders, we have been working at Meta for many, many years, between 7 to 10 years. Especially in the latest 5, 6 years, we have been building AI infrastructure to help Meta take the leap to transition to be AI-first company. As part of that, we built PyTorch. As you know, PyTorch is the dominating AI framework right now in the industry. And my responsibility is to convert PyTorch from a research-only project into supporting all the production workloads. It took us 5 years to penetrate mobile app using PyTorch to running PyTorch at huge scale on the server side. By the time we left, we were running more than 5 trillion inferences per day across all data centers globally. While we are doing this, we feel like really big impact delivered for Meta. But when we look at the industry, the whole entire industry is also going through AI-first transition, but they're going through excruciating pain because they don't have the the talent to support that transition. They don't, many of them don't even have GPUs, they're still on CPU. They don't have the know-how of the best software to run on top of GPU and drive down latency and cost. So that's the reason, the mission we started Fireworks is to support the whole entire industry to do this AI-first transition. We want to heavily compress the time. Remember at Meta, it took us 5 years. From 5 years to 5 weeks to even 5 days to enable enterprises and developers to build innovative, disruptive applications and products building on top of Generative AI. So that is our mission.

Awesome. So tell us about the platform, the offerings, and what can customers expect from Fireworks AI?

Right. So In the simplest way, we offer general AI model inference SaaS platform, and we also have a companion SaaS platform for fine-tuning. So these two goes hand in hand for us to deliver the customized quality, customized quality using your prepared data towards your use case, your workload, to deliver the best lowest latency for real-time application that you, um, build for end consumers, prosumers, and developers, for the lowest total cost of ownership, and enable you, empower you to build a, a viable business after you have a viable product. So across the board, quality, latency, total cost of ownership are the values we deliver to our customers using our proprietary inference SaaS platform and fine-tuning SaaS platform.

Got it. So essentially customers can consume a model as model as a service offering, and they can also use API to bring their own dataset and fine-tune and deploy the model. Is that right?

Exactly.

Right. So, You know, the market currently has multiple players offering very similar fine-tuning as a service and inference as a service. Compared to your competition, what is the key differentiating factor of Fireworks AI? Why would someone use Fireworks AI versus any other inference as a service offering?

Right, so I, I think the biggest differentiation is we heavily focus on production And when the developers are enterprise, they have seen great sign of a product market fit and the experimentation evolve into, hey, I'm going to launch this at huge scale in production. And that's where we design this Phase 4 is we really know how it's operating huge amount of volume. Like Meta is one of the biggest B2C company. And that's where kind of we understand deeply that interactiveness delivered through extremely low latency makes a product viable. And when you scale, you really need to control your TCO. Otherwise, you're going to bankrupt pretty quickly. So this is our heavy differentiation is to deliver this production SaaS platform, not just low latency and low TCO, but it's highly scalable, highly reliable. So we scale with a business. You can leave the operation aspect to us with a peace of mind and only focus on heavily competitive application product development space. So, so that's where we want to empower our customers across the board, all the developers and enterprises, to, to to continue to innovate because this is the, this new tidal wave for generative AI. Why everyone is paying so much attention is all the leaders in this space, they know that this is a new technology that is going to drive a titanic shift of the landscape. If they do not invest and they do not pivot, their company, even though they could be incumbents, can become obsolete and no longer stay competitive in this dynamic market. Our job is to let them focus on the innovation on the application and product side and leave the infrastructure operation and the scalability, reliability, latency, low latency insurance, and low TCO to us. So that, that's what we have been doing for years at Meta, and we would love to bring this experience to the industry.

Perfect. So you mentioned low latency as one of the differentiating factors, but assuming the endpoint that you are exposing is maybe in the US data centers or in UK data centers and, and the customers are consuming it from Asia Pacific, how do you ensure there is less latency or low latency for customers across the globe?

That's a really good question. So first of all, our inference engine where we control the execution itself, we do a lot of layers. We do kind of horizontal and vertical optimization. Let me give you some idea what I mean. So the horizontal optimization is at the model inference runtime level. As we all know, majority of the GenAI models are PyTorch models and we wrote PyTorch code for years. And we know exactly how to optimize PyTorch. In addition to that, we also, for the heavy hitter operators in the layers in the model, for example, matrix multiplication, attention, and cross-GPU communication, those layers, we handwrote kernels to heavily reduce the resource consumption, compute resource consumption. And then, We also do application-level optimization, and there are categories of use cases. They all come in different patterns. And at the service orchestration and the scheduling and kind of just the overall serving layer, we heavily— we built our proprietary technology to cache data so we don't always have to recompute every time to to deploy certain kind of configurations that is designed for capture certain patterns of your workload. And two, at Cloudflare level, we also enable cache to— yes, we work with our close partners of the CSPs to leverage their compute resource globally. So you're going to hit an endpoint that is closest to you. So we're kind of networking latency becomes an issue, right? So, so we actually drive real-time optimization across the whole entire stack. So we ensure our customer get the good fast experience so they don't have to worry about all these layers of system build out.

That makes sense. So what you're basically mentioning is the way you optimize the inference engine to deliver highly optimized, highly optimized LLMs that will deliver more number of tokens per second, which will directly translate to the speed and of course the latency. And second is you have a caching layer that's actually enabling better delivery and better performance. And finally, you also have a partner network where you're actually co-located in various other locations that are closer to the customer locations, right, or customer applications.

I would say on top of that, we have managed hosted API service, right, for enterprises, but we also deploy into our customers' VPC, right? There's nowhere closer than we can run directly inside their VPC. So we also support that mode. As a matter of fact, the more enterprise we talk with, the more we we kind of have to deploy in that mode because they are very sensitive about privacy. And the GenAI inference, it consumes a lot of data, right? And many of data has a lot of private information that the enterprise doesn't want those data to leave their VPC. So another way to think about real-time low latencies, we deploy directly into our customer's VPC.

Got it, got it. So if a customer is running on AWS or Azure or GCP, uh, your stack actually runs within the same subnet of the VPC of the customer, uh, yes. So that is basically a private or hybrid deployment, right?

Yeah, it's, uh, it's very, it's very secure, um, and, uh, it kind of, um, solve the problem that They know, it's also our differentiation compared with OpenAI, is OpenAI will not deploy into customers' VPC, and they want to be the hosted managed service versus we want, really want to respect our customers' privacy needs, and we can deploy into where they think is the most private setup for them to run inference.

Thanks, that actually clarifies. But one, one question that I have is most of the customers who are running GenAI in the public cloud will tend to use a managed service offered by, let's say, GCP, which is Vertex AI, or Azure AI Studio, or Amazon Bedrock. What would make them choose Fireworks AI instead of the native cloud provider-based GenAI platform?

Right, that's a really good question. So I will take a big step back and talk about what is our target, right? So we really think about ourselves as a better— we want to build our product as a better OpenAI for enterprises. And let me explain why. I think OpenAI has this really great mission to deliver AGI. That means it's a more complex or smarter AI system than human being. And the vehicle for deliver that is a small set of models. You can think about that as one model that solve hundreds of thousands of problems the best. So that is the strategy. But that's a really hard goal to reach because, I'm glad OpenAI is tackling that problem. It's very difficult to reach because when you train a model, you have to set the objective functions for the model to optimize for. And then it almost, it means that the researchers training the foundation model need to be have to decide ahead of time that among all the problems, which problem, which is, which subset of problem they care the most about, which subset of problem it's kind of okay, they don't have to be the best, which set of problem they don't really care about. And then they curate the training data based on those objectives. So it's inevitable that the end model, the end result of the trained model will be really good at certain things, will be okay with certain things, will be really bad at certain things by design. So have one model solving all the problems the best is really, really hard. And so we take a complete opposite approach where we also want to solve hundreds of problems that enterprise cares about, but we are gonna enable hundreds of smaller models. And the key here is to heavily customize those models towards a particular use case, and on top of that, further customize towards individual enterprise workload. So that process is a much viable process to, to deliver the end goal we want to deliver, which is, um, now our enterprise customer, they have a proprietary model, their own model, uh, that is on par or beat GPT-4 quality. They have the lowest latency, lowest total cost of ownership. So when we reach that goal, that means there's no reason, um, any enterprise wouldn't run on top of Fireworks. So that's kind of where we're heading towards. We are already leading the industry when it comes to low latency and total cost of ownership. We're building a stack where we customize towards, we're building a SaaS platform to customize towards use cases and individual enterprise workload. To reach really, really good quality. So here I want to clarify, by customization, we do not provide professional service. It is a SaaS platform that will do the customization automatically. So yeah, so that's our vision.

Got it, got it. Thanks for the clarification. So apart from offering the infrastructure and platform and the SaaS service, you also invest in foundation models. For example, the FHIR Functions V1. I'm a big fan of that. It's very capable when it comes to performing function calling. So tell us more about that and why did you diverge into building a foundation model by yourself?

Yeah, so I want to clarify a bit. We are not building a Fire Function for function calling from scratch. We build on top of existing foundation models and uptraining the model to make it more aligned to solving function calling problems. So here's the kind of big picture, right? Any given large language model, whether it's closed source or open source, it has limited knowledge. The large language model, they can answer accurately for certain questions. The knowledge base is bounded by its training data. As we all know, the training data is fixed set. It's fixed in terms of time range, beginning and end. It's fixed in terms of how people can crawl the internet and buy data from data providers. So it's always limited. With that limited knowledge, there are knowledge that is infused in other places. For example, there are other modalities, there are image generation, generation models which understand and generate images. There are video generation models, there are audio generation models, there are embedding models, there are image-to-text models, and so on. So there's a variety of multimodality models that also encapsulate knowledge. In addition to that, many of the knowledge and especially real-time information is hidden behind APIs. There are public APIs, there are also private APIs each enterprise holds. So many of those, so that's kind of the baseline we need to get into. And then current innovation on the application product side, building on top of GenAI, they have to, eventually they have to access the totality of the knowledge, not just the knowledge they can extract from a large language model by itself. So, so that means they have to have access to other modalities. They have to have access to other APIs and public APIs, private APIs. So function calling, or many people call it tool calling, is a way to bridge the knowledge gap. It's the way to bring the totality, totality of knowledge, and then you can build your application product on top of it. So that's why strategically We have put a lot of emphasis on in this area, and starting from last year, we released our first function calling model. This January, we released the second version, and in a couple of weeks, we are going to release the third version. The third version is really, really good. So we want to help our developers and enterprises to have this one surface area where they can— through that, they can get access to various different kind of knowledge as they want, and they don't have to worry about where to get those. So that's kind of, that's our mission, and that's the reason behind our investment for function calling.

Perfect. You also partnered with Naus Research to fine-tune Hermes 2 Pro, particularly with Llama 3. So when you already have FHIR functions, what's the rationale behind partnering with Naus Research to come out with another model with the same capability?

Yeah, our philosophy is we want to bring the best of the world into our platform. It's— we're less opinion whether we build those models or our partner build those models, but we are very opinion that we want to surface the best models on our platform to our developers and enterprise. So that's the motivation behind our partnership with Nausk. That's the motivation behind our partnership with Stability. And you will hear more from us. We're gonna extend our partnership with many model building companies, especially foundation model companies, to service the best models on our platform. With that said, we'll still continue to invest in function calling because I think we have a slightly different goal from general function calling model. Our goal is, And so today we already serve more than 100 models on our platform, and we're going to increase that number to potentially thousands of the best models across the industry. And then we want to have this layer that can wrap on top of all these, the best models we serve. So from the Totality kind of the easy access to our system point of view, that's where kind of our investment will get into, and that will be very customized towards accessing Buildiswrapper, accessing all the models we provide. So that has a special unique angle than a general purpose function calling. So I think there are a lot of overlap between these two goals. So we'll continue kind of push on the function calling space.

Totally agree. I think it's, It's great that you're partnering with other model providers and extending the function calling capability. So, on that note, I want to get your viewpoint on the rise of agents. I think function calling or tools are the foundation for agents. So how do you see the evolution of agents and the agentic workflow? Where do you see that going?

So, I think that is going to be a fast evolving area because I think I can imagine revolutionary applications will emerge in this space. Just like AlphaGo demonstrated the capability of autonomy in learning from playing chess or playing Go by itself, I think the future of GenAI app will become autonomous. So we just talk about function calling. The thinking for function calling is still you have a layer that is taking control of where to route your API calls into. But in the future, we'll have a system, it will automatically route and direct requests to the right agent API to process. And it can cascadingly do processing and routing and directing until the right answer is retrieved. So then the control is going to be distributed. I think that is definitely going to come. So instead of one function calling, pulling from different models API as a controller, in the future agents are going to be self-organized, self-coordinated, solve problems together. So in that world, you can imagine there are multiple hops to solve a problem and the latency becomes even more important. So right now it's like more than, more like one hop where you, hey, you ask a large language model to question in, answer out. In the future, there will be multiple agents like bounce off results and questions to each other and then solve a problem. So the latency budget to each of those steps becomes even much smaller. And that's where we believe Fireworks Lightning Fast Inference our function calling model, our fine-tuning service will pave the way for developers to get there.

Perfect. That's a good note to end the discussion with. I completely agree with you on the evolution of agents and how they're going to be autonomous and more self-organized. I think that is the future of agents. So Lin Qiao, that brings us to the end of this discussion. Thank you so much for being a part of the show. Great to learn about Fireworks AI and wishing you all the best as you compete with some of the big wigs of the industry and taking LLM serving closer to the enterprises.

Absolutely. Yeah, so for all of you that is interested in trying out Fireworks, we have— it's very easy. Just go to fireworks.ai and go to click on models and you can play with any of the models. Models on your playground or integrate with our API. Our API is OpenAI compatible. We are a superset of OpenAI API, so it should be very straightforward to get started.

Perfect. Thank you so much.

Thanks a lot.