Going from Single Model to Compound AI Systems ft Lin Qiao, Cofounder & CEO - Fireworks AI

My name is Lin Qiao. I'm CEO and co-founder of Fireworks AI. Welcome to our first Campfire Connect. Today we're going to talk about the industry trend of going from single model to compound AI systems. Here's a quick show of agenda. We'll get up to speed on Fireworks AI. Some of you have probably have already, already been using us, and some of you may be new to Fireworks AI. We'll also show a quick demo of the GenAI native applications built with Fireworks, and then we will start to introduce to a new concept called compound AI systems, and followed by a quick demo of Fireworks version of compound AI system using our function calling model. And then we'll do exciting campfire chat with Harrison Chase, Lanchen CEO. At the end, we'll do a Q&A. Please enter your questions by clicking through the activity button. Okay, before we start, um, we're going to start this session with a chemistry test. 10 points for everyone who guessed What this compound is, if you— some of you may guess correctly, it's penicillin. And this medicine created a massive impact on increasing our overall life expectancy. So it's very interesting analogy of building from molecule to compounds, right? It's not just penicillin. Composing molecule into compounds has been a major change in many, many ways. For humanity. The compounds are more functional. They have new and emergent properties that individual molecules do not possess. Compounds are more stable, which is crucial for formation of more complex and durable structure. And compounds are more interactive, enable new medical pathways and reactions that's not possible with single molecules. Something similar is happening with AI. Models are like molecules, and now developers are synthesizing them into compound AI systems, which are more functional, stable, and interactive. This transformation is necessary for us to cross the chasm from prototype to production. So we have been spending a lot of time playing with single models from the industry, and single models are limited. Because generative AI models are fundamentally probabilistic, it produces the next code token based on the highest probability. So those models are capable of hallucination, and that is causing problem for applications and products. And second, each individual model has its own unique strength. There are models able to understand or generate text, audio, image, embeddings for semantic search, or vision language model. So even within each individual category, there are specialty models. Some are good at classification, some are good at long context processing, like with good memory, uh, and so on. So just using one single model to solve a complex problem is really hard. And also, single model has limited knowledge. Because its training data is finite, not infinite. And when you ask a question outside of its knowledge, and then it becomes problematic. A lot of deep knowledge lives on internet, the news, the most updated information, or lives behind APIs, the real-time information about weather, about stock price, and so on. And also lives inside enterprise in database management systems, source system, file system, so on. So compound AI system is built to address all these challenges. The idea is simple: evolve from single model to a collection of components of various different models across modalities, combined with various different knowledge sources. And in combination, this system can increase many new AI results. In terms of accuracy and quality. It covers examples like Gemini, uh, using 32-step chain of thought to answer question and pick the best one, or ChatGPT Plus calling tools to enhance the knowledge. And I didn't invent this term. This term was coined by Berkeley and recently amplified by Databricks. Before we dive into compound AI system, I'd like to talk about the challenges for Application development. I believe many of you are application developers before and after GenAI. As we know, GenAI is a disruptive technology driving innovative applications and products like the previous waves of cloud-first and the mobile-first. And all these previous transitions completely shaped the industry landscape. In current wave, there are invariance and variance in the application design because of GenAI. First, many applications built on top of GenAI, they are consumer or developer facing. So you need to deliver very interactive response and low latency as a prereq for a viable product. And many of those consumer developer facing product will scale quickly. So low cost is also very essential to have a viable business. But generative models are on the highest end of the spectrum of size and complexity, so deliver very low latency and high cost efficiency is very challenging. What is changing for application developer is Before, products and apps are more deterministic because the logic is encoded in program they build. Now, generative models are essentially probabilistic, and it can hallucinate. So grinding the content generating is a new skill set everyone's learning. They also need to learn how to expand the knowledge beyond the generative models, individual models, to produce high-quality results. In order to address all these challenges, Fireworks is expanding from single model serving to a compound AI system. Here's a diagram of our product offering. On the lower right, we have Fireworks distributed inference engine. It has multiple components across far attention, our customized GPU kernel, to disaggregated serving, for us to scale computation beyond one GPU, and also semantic caching across multiple layers to trade memory for compute. And we have expansive deployment coverage of our inference engine building on top of tier 1 CSPs. The primary goal of Fireworks inference engine is to provide interactive latency and high cost efficiency of the first two challenges I just mentioned. On the lower left side, we have Fireworks adaptation engine. It has the primary goal to efficiently customize quality or latency with your own data. You can do supervised fine-tuning using curated dataset from your application, and we make the process super easy. To further simplify the work for you, Fireworks can enable self-adaptation based on your product workload. Using self-tune. We can further support personalization for both consumer-facing or enterprise-facing applications. By personalization, I really mean to tune a separate model for each entity, should it be individual person or individual enterprise or customer you are onboarding. Our adaptation engine designed to increase grounding and addressing the challenge of hallucination. Let's move up the stack. We have a very comprehensive model stock, covers multiple modalities from both open source and the proprietary model from the top tier model providers covering text, audio, embedding, image, vision language, and so on. Building on top of all the layers mentioned, our Compound AI SDK consists of our tuned function calling models, specially tuned function calling models by Fireworks, a support of JSON and the grammar mode for constraint output generation so it can tie multiple, tie multiple components very well together. And that's the, um, our Compound AI SDK stack. So with all these combined, we provide a comprehensive product offerings. Now I would like to show a quick video highlighting Fireworks product features and many GNN native applications built on top of Fireworks. All right, this is super exciting. So much energy is there. I cannot wait to see more disruptive and innovative applications building on top of Fireworks. So for many of you who are driving those innovation, please let us know your feedback and anything we can hear from you to improve, love to do that. Now I'd like to dive into each product feature and share more details there. We'll start from Fireworks Distributed Inference Engine. There are 3 main components. First, FireAttention is our customized GPU On the cost side, in comparison, we can also drive as low as 8 times lower cost compared with GPT-4o. And because we understand operating a viable business is important, you need to have the right cost structure. Just give you a quick example. To process 1 billion tokens per day, you are gonna save more than $6,000 every day and amounts to more than $2 million a year. 1 billion tokens per day may sound a lot, but it's actually very common for many companies we see. It just means if you have a prompt of 5,000 tokens supporting 10,000 users each hit the LLM's inference endpoint 20 times a day, it will aggregate to 1 billion tokens per day. So many businesses already hit that ratio. Through third-party benchmarking vendors, they also show Fireworks as leading the pack when it comes to low latency. Here, um, this case is about a most commonly used pattern of LM with RAG. So we have huge performance, uh, in this example, we're 9 times faster than Groq, which use a custom ASIC. We don't just do text, but also do image and multimodal. We're the fastest provider to serve SDXL. You can get almost 9 times improvement compared with do-it-yourself on the GPU you procure. We're also the only inference provider serving SD3. Stability's API built on top of us. Again, we put special effort to optimize for long context. So because this is the future, we can achieve from 4 times to 12 times lower latency than VRM, which is the most popular open source serving solution. Now I would like to move to Fireworks adaptation engine. Before we dive in details, I want to quickly show the trend and of how the model quality is evolving. This is a chart that shows the evolution of model quality based based on the ELO score published from LM6 Arena. The red dots are open-source models and green dots are— the red dots are closed-source models and green dots are open-source models. On the rightmost— leftmost side, in March 2023, there's no available open-source models. But 3 months later, Llama emerged. And between GPT-4, the best closed-source, and Llama, the best open-source, there are about 100 Elo score difference. And fast forward to May this year, that gap shrink down to 50. And you can also see from this chart, the activity in open source land is much more denser than closed source. It— this chart doesn't even include the fine-tuned model releases. On Hugging Face, there are more than 50,000 open source models built for GenAI, and this highlights the energy from the community. What's not in this chart is in many specific areas, open source are getting on par with closed source models, for example, coding. Fireworks want to— we want to offer the best models across the industry to our customers. The best we can have access to. Today, we provide more than 100 top-quality models from the base models to fine-tuned models across all modalities of text, image, audio, vision language, and embedding. This model also include, yeah, open-source models, of course, and also a few, actually a few proprietary models from top-tier model builder. Now comes to what is the Fireworks Adaptation Engine. It's a weird name. So Fireworks Adaptation Engine has the main goal to efficiently customize quality or latency with your own data. You can do supervised fine-tuning using curated dataset from your application, and we make that process very easy for you. To significantly simplify more, Fireworks also enables self-adaptation based on your production workload using automated tuning. That means the more you run your inference service with Fireworks, then your model becomes faster and smarter. Fireworks can further support personalization for both consumer-facing you personalize towards individual user, or enterprise-facing, you personalize towards individual company support. And there you can create thousands or even hundreds of thousands tuned models. And this is actually a very common pattern we have observed. For example, for B2B business, you would like to tune a separate model for each customer, each enterprise customer, to avoid data leakage across customers, and that increase the complexity of the whole tuning and inference process. But we will help you solve all these problems. To zoom in more to personalization optimization, we see many customers using LoRA adapters because it's the most efficient compared with the full model fine-tuning. The traditional way is to serve LoRA adapters and the base model one-to-one. However, in this case, there are many LoRA adapters. By many, I really mean thousands or even more. Serving thousands of models is very expensive. We implemented cross-model batching. That means all our adapters can share the same base model. This way, it significantly compresses the memory footprint and can lead to, lead to up to 1,000 times cost reduction. Now I would like to walk up the stack and talk about our Fireworks Compound AI SDK. I know many of you joined this event to hear about compound AI system and what is Fireworks doing here. So here we will dive in, and I would like to see if Harrison agrees with any of those, and we will have a conversation as a second part of this, this event. So Fireworks compound AI system, we have our special definition or implementation. Of this concept. It starts with our fine-tuned function calling model called FireFunction. This FireFunction model is capable of talking with multiple other models served by Fireworks inference engine across many different modalities. It's also very self-aware and knows when to call external tools for help. Including retrieving information from databases using SQL, or doing internet search, or using APIs to extract real-time information from my personal to-do list, personal email, or public sources, or access knowledge graphs. Our philosophy from Fireworks is to design a very simple API and hide the complexity in the backend. So we believe in the future trend of prompt as the universal API, and we design our Compound AI system around that. Let's take a look, look into a little bit more details. FireFunction is specially tuned to preserve multi-turn chat capability, so itself is strong LLM. In addition, we added capabilities of action planning, orchestration, and execution. The actions can be done in parallel or sequentially. The API access to Fire Function is very simple. It's just prompt. And many of you, probably all of you, are very familiar with this API. This concept is very similar to database systems where SQL is the declarative API. For user to specify what you want to query out of the database. The database engine will then do the heavy lifting to plan detailed execution in a distributed fashion with the right partitioning or right indexing or even materialized view to extract the best performance. So here, um, in production, to solve a complex problem Oftentimes we need to tap into multiple models or with multiple modalities. Here are plenty of examples of that. For example, we have seen many customer, they are automating call center or customer service, and they need to thread the models across audio, chat, and get API access to semantic search. In e-commerce vertical, Many companies are doing product catalog cleansing to generate high-quality assets, and those assets can use to boost search and ranking relevance. To do that, they need to use a combination of vision language model and large language model. We have also seen customers, they spend a lot of legal processing time and the cost to process the medical claim. And now they are implementing automation of medical claim processing using both vision language and large language model. In the gaming business, there are games require coordination across a team globally in real time. So the teams, different members, they are in sync about strategy and tactic. And that requires a combination of audio translation and the LLM. We have seen many sales process optimization with sales call summary and intent analysis using multiple modalities across audio transcription, summarization, classification, and so on. In addition to that, having the right API access to solve a complex problem is also critical in production. In the observability industry vertical, we're seeing many companies requiring access to production log using a search API or extraction API to their internal production log. There are many companies building assistant for research, for academic research, or for legal case study. They need to have access to internet and internal, external document store. And then write back the report to some documentation using API access. Many, many companies also doing developer productivity improvement, building developer portal, and that's essentially a Q&A engine for developer documents, and they need to have storage access for these documents. Companies also generating KPIs automatically with API access to databases through SQL, and API access to generate charts. Salesforce analysis is extremely important to drive business efficiency, and that requires a wide integration, a wide variety of API to Salesforce, HubSpot, Slack, Notion, and so on. So with all these pieces, the— there need to be a glue to tie this together, um, and the constraint output generation is very important to be the, to be the glue. And we already support two modes: JSON mode and grammar mode. And as you can see from this example, this is example of JSON mode. Output JSON is great, but not all the rules and structures can fit into JSON mode. Grammar mode lets you describe a desirable a grammar in BNF format. There are endless possibilities, endless possibilities of what you can do. Anything from output styles to entire programming language can be represented as grammar. I believe Fireworks is one of the few companies, if not the only company, providing grammar mode support. With this, I would like to show you a quick demo highlighting what's possible to build with Fireworks. Compound AI system.

Fire— Fire— is now multimodal. Today we'll dive into these capabilities and see how they enable natural interactive experiences. Let's see how it works, taking both an audio prompt and an image input. Can you tell me who's in this picture? Under the hood, Fire Function calls into Fireworks's new Fire Lava visual language image model to understand the image context natively. Let's also take a look at how Fire Function can combine multi-turn chat contexts with calls to external data sources and visualization libraries. Fire Function was able to accurately interpret the user's request, fetch the external information, and call into a chart-making function to create a visual asset. Let's see again how FireFunction can combine running chat context to iteratively improve on its output. Finally, let's see how FireFunction integrates with other Fireworks services, which give it a large array of multimodal output capabilities. In this case, let's make a creative image from the charts we've generated created so far. And there you have it. Fire Function acts as a building block for compound AI, orchestrating inputs and outputs from advanced AI model inference as well as more traditional data sources and computing functionality. We're really excited to see what you build with it.

This is very exciting. I just covered all the product features of Fireworks, from our fast and most cost-efficient inference engine to adaptation engine to ground generation, content generation, and then to the upper stack of Compound AI SDK for expanding knowledge. Fireworks is already deeply in production. We serve more— we process more than 140 billion tokens a day and generate more than 1 million images per day with extremely high reliability and availability. Today, companies ranging from startups— many of you probably— to Fortune 500 are using Fireworks to disrupt the status quo. Startups are using Fireworks to build new product and new experiences, and Fortune 500 companies use Fireworks to improve productivity. It's very easy to get started from fireworks.ai, our main web app, if you have not tried it yet. And I cannot wait to see the future you build on Fireworks. Thank you.