Democratizing AI with Fireworks

TV live. We're here in New York City for.local manga day B, and I have another special guest here, Lin Qiao of Fireworks. Welcome, Lin. How are you?

I'm great. Amazing. I'm hearing myself.

Yeah, it's a little odd at first hearing yourself on, on, uh, on the live stream, but, uh, welcome. And Fireworks is amazing. Tell us, give the audience a little bit of a background on Fireworks. How did it come about?

Yeah, definitely. So I started Fireworks with a group of 6 other brilliant engineers. Our background is we have been working at Meta and Google for many years, between 7 to 10 years building AI infrastructures. I was head of PyTorch for 5 years and took us 5 years to bring PyTorch to the fully to production, running more than 5 trillion inferences per day across data centers globally. Our mission to create Fireworks is to democratize AI infrastructure, make it accessible to a wider set of developers, application developers, and also significantly compress the timeline from 5 years to production to 5 weeks or even 5 days.

Wow. How's that possible?

It is actually very possible. And I will share a little bit more about how to access Firewalls. It's very easy.

Nice, nice, nice. So what does the ideal AI tech stack look like for you?

Yeah, so it goes all the way from— at the bottom layer is the new hardware because generative AI is on the extreme, on kind of very big and complex spectrum of AI. It has to run on GPU, which is very expensive, but also GPU is power hungry, generates a lot of heat. It requires interesting cooling. It cannot use air cooling. It has to use liquid cooling or immersion cooling using oil. So all this infrastructure orchestration is very expensive. So cost to serve is very concerning to build on top of GenAI. And that's where we come in and significantly reduce the cost. Of ownership.

So in what ways are you able to reduce those costs?

So we do significant optimization at the software layer. Of course, we cannot reduce the cost of GPU. That's out of our control, but we can reduce how many GPU you use to enable your application to run, especially most applications are consumer, prosumer, developer-facing. Once they hit product market fit, they can scale very quickly. And a lot of companies we support run into issue that it just can no longer afford the cost and it's not a viable business to them at all. But we can drop the cost by 8 times, 10 times and make it super accessible to, to a broader set of audience.

Nice. What are some of the main use cases that you're seeing?

Oh, there are a lot of use cases. Again, coming back to traditional application, they are very interactive. That makes application interesting. So that didn't change whether you use GenAI or not using GenAI. We also focus on delivering low latency for real-time application too. And yeah, and among, I will probably bucketize into 3 categories loosely. One bucket is a new chat experience through text or through voice. And that's where we see, for example, e-commerce is transitioning to You type the keyword to search for product into YouTube, to the search portal to retrieve the best product you want. A slew of new assistants, including legal assistant, helping lawyers to do case study, medical assistant to address the big shortage of nurses and the doctors for preventive care, to medical, to take exams and so on, to education assistant to teach foreign languages or other classes. Or automate customer service. So that's one school of application. The other is not generating human processing language, but more like generating other things. For example, code generating API calls and so on. So we are supporting a lot of innovative coding assistant, not generating code, but also do great code review, improve your code like Casey. And also people generating business metrics in a visual way, generating production metrics from production logs and general business logic. And the third bucket is not generating text, but also generating mix of text and image or videos. For example, generating PowerPoints, generating branding, marketing message with text and images. So also it's a whole slew of richness of application that's in market right now.

It sounds like the sky is the limit, like there's so many innovations going on right now in the AI space specifically. So going back to the, you know, a developer creating one of these applications, what are some of the challenges that a developer faces when trying to go from prototype to production? And how can Fireworks help with that?

Yeah, that's a really good question because we hear a lot of complaints. I think the first one is their application needs to be real-time in the sense that highly responsive. Usually people all know about OpenAI. They start with OpenAI to test out the hypothesis. Even if even a large language model is the right solution or technical solution or not. Once they hit the product market fit, then they really serious about, oh, this is going to be the great product. And then they realize, oh, the latency is so high. It just— it's not a great product. It's not responsive enough. And then they all migrate from OpenAI to Fireworks and they immediately see user— the latency drop by 4 times or 5 times and user engagement drastically increased. So, so I think that's a challenge they're facing is they have to kind of still maintain a highly viable product experience through low latency. The second is I mentioned the cost ownership is— that has changed. Typical application, like old application, they run on top of commoditized, heavily commoditized CPU. So the cost factor is very small from a hardware point of view. Now they're running on GPU, so it's It just shifts the cost factor a lot and then make it difficult for them to make a viable business out of it. And we come in and significantly reduce the cost for them.

Nice. That's amazing. So talking about the future, like, like I said, so many innovations going on in a very rapid pace. The latest thing that I hear a lot about is agents. How— what do you think the future will be? Is like, are agents the future? Or what do you think is going to happen next in AI?

Yeah, if you look at the 5 years down the road, anything can happen. I would say based on this velocity of changes, I think one inspiration is, hey, we all know AlphaGo is a testimony that AI can learn by itself and it can learn from playing against itself and it becomes so much better. I believe the future is right now we're thinking about developers control the workflow of how to use AI. In the future, there will be versus different autonomous agents specializing in doing certain things and orchestrate organizing among themselves to solve a problem. And that's where Fireworks, we are focusing on that. So to solve a problem, there will be multiple round trips across agents and latency become even more important. Absolutely. And that's why we believe Fireworks paved the way through our low latency offering. We also have function calling for agents. We'll pave the way towards that future.

Nice. So can you think of any community projects that are using fireworks in their products that you want to plug right now? A project that is using fireworks in a way that is inspiring, or are there any specifics that maybe you could talk about?

Yeah, so I think there's so many. I wouldn't pick which one's my favorite, but maybe I will pick a recent one. Okay. That is basically a company called AnySphere. They are building a very innovative product to do coding in a drastically different way. And basically, we know that code review is high-quality code review takes time. It becomes a barrier to kind of move forward. There's a lot of productivity lost there. And they have a very fancy automated code review product built on top of fireworks, and that requires extremely low latency. I would say working with them, one of the benefits is they push us so hard and we deliver. So I strongly encourage every one of you to try out Cursor, their new product, and we are definitely a huge fan of that product.

Yeah, I've seen it. I've used it.

Oh, wow. Yes. Oh, that's amazing.

Yes, I love it. So, yeah, and that's great to get this feedback, you know, as everything is evolving, getting feedback from users. That way you can, again, they push you because they're trying to do this thing with your product and you want to help them to succeed as well. So the feedback is amazing. Same with the products at MongoDB, we love to have feedback and see you succeed. That's the point, we're trying to help you succeed.

Yeah, definitely. By that, I want to actually talk about our partnership with MongoDB. It's extremely important because I think the future of this gen AI wave is actually disruptive applications. And Mongo actually has a lot of reach, a great community of application developers. And in the new gen AI era, the models are not like— it's not going to solve everything. It actually has a problem of hallucination. And one way to control hallucination is through a technology called RAG. And MongoDB already stores a lot of operation data and it provide vector search engine that's the best for retrieving relevant information and from the RAG component, feed that into Fireworks LLM inference serving to produce the best quality result. On top of that, in conjunction, we can deliver 4x lower latency and 8x higher throughput. So you get the best of quality, latency, and, and the cost by using the combination.

Nice, nice. Yes, MongoDB can be the memory for your AI application. It's amazing to add that ground truth there to your application to make sure that it doesn't hallucinate, like you said. Yes, exactly.

Yeah, definitely. So I would encourage all of you to try out Fireworks. It's very easy. Go to fireworks.ai. There's a self-serving onboarding process, super streamlined. You also get automatically filled with free credits to get started. We also publish a blog post to talk about how to use MongoDB Atlas Search combined with Fireworks. So it's very easy to get started.

Amazing. And for those who are watching, there are links in the video description to those assets that she mentioned. So thank you again, Lin, for joining us.

Definitely.

It's a pleasure to have you. Yeah. And we'll be back after a quick break with our next guest.