Lin Qiao, Fireworks AI | theCUBE + NYSE Wired: Mixture of Experts

Welcome back, everyone, to The Cube here at our NYSE Studios. I'm John Furrier, your host of The Cube. This is our mixture of expert series where we bring in leaders who are experts who are making the AI wave happen as well as other accelerated disruption enablement that generative AI is certainly making happen. Obviously open source and the rise of developers now taking advantage of the infrastructure. Always great to have the leaders share their thoughts. Lin is here, CEO and co-founder, of BioWorks AI. Lin, thanks for coming in remotely to theCUBE here at the NYSC studio as part of our NYSC Wired program and community. Thanks for joining us.

Thanks for having me, John.

So first of all, we covered on siliconangle.com, my site, your recent funding, $250 million at over a $4 billion valuation. Quite a nice run over the past 2 years or year and a half for you guys. Congratulations.

Yeah, thank you. This is a big moment for us. Lightspeed and Index co-led the round with the past round led by Sequoia and Benchmark. I think there's a lot of trust from our customer partners and investors for us to continue to grow very fast. And there's a lot of momentum that we have, we have accumulated and there's a lot more for us to build into.

Yeah, the thing that I love right now and I want to get your thoughts on is the market around open source. And how open source developers are starting to lean in and look at the models and how AI-native companies are thinking around building companies. Obviously having a data mode is a huge advantage. You're starting to see the enterprise open up again. You see the big large-scale hyperscalers and the neo clouds doing well, but it's going to be the applications that are dealing with these models, this AI-native. What's your— thoughts on that because there's the new school of AI native and then there's kind of like the SaaS moving into AI enabled and then kind of transitioning. Talk about those two dynamics in context to how developers and builders are thinking about the models themselves.

I would say open model is strategically very important for developers to build the state-of-the-art application. The fundamental reasons are, uh, The open model providing a huge variety of choices. At the same time, open model quality is converging with closed model. The fundamental reason is model quality is decided by two factors: data and the training algorithm. Data is pretty much the whole entire public internet and data provided by labeling company, which all different, various different model providers, whether open close are using. So there's not much differentiation. Um, on the training algorithm side, um, the researchers are moving around, so there's not much, uh, secret sauce remaining. Um, so that's kind of explaining why the closed models, uh, are all converging. But at the same time, we are depleting all possible data, uh, that is in public domain or that is— that can be generated by labeling company. Um, however, At the same time, there is much bigger data set we're not tapping into today. That is the private data locked inside application, locked inside enterprise. The foundation model developers or researchers have no access to. I would say the next era of model improvement and model development should heavily index on customization, and that's where Fireworks, when we started, we have set our mission into customization. Our long-term mission is called autonomous intelligence, where we deliver application-customized inference. We have already delivered application-customized inference for speed and cost efficiency. And these two are extremely important factors. I'll get back to it. And we just launched a new product called Fireworks Reinforcement Fine-Tuning. Fine-tuning. It's based on reinforcement learning to customize for quality using applications' private data. So these three combined, we are giving agent, agentic developers or application developers a tool to activate the data, the user engagement data, user preference data, intent data to build a much better model, much faster model, much more cost-efficient model to run their business. And extremely important, for multiple reasons. One is the agent developer, they need to have full control, full control of the quality of the version, of the predictability of their mode. And the second, they need to have full control of cost. And many companies, they have great product. And it's funny, in today's stage, having a product-market fit and having a viable business are completely two separate things. You can have a brilliant product that your customer is willing to pay, but that doesn't mean you're going to have a viable business because many companies come to us saying they cannot scale. They have 10x more people they have held in the waiting list and they cannot open up the floodgate because they're going to scale into bankruptcy. The way they burn money is so fast on top of GenAI. It's not sustainable business at all for them. So, and that's why we provide this three-dimensional customization across quality, speed, and cost efficiency. So agentic application developers can leverage the tools we provide from Fireworks to have the best product experience where they build on top of their customized model with the best quality and interactiveness to the best speed and build a viable business with the best cost efficiency.

Lin, I love that description because you just explained the enterprise problem, which is they have all this data that they want to unlock value out of. And they also— it's proprietary to them. They also have workflows. They want to apply agents. They got to come in and understand how to, I guess, distill from the best, but also apply targeted generative AI capabilities to that data. And we saw a lot of POCs die on the vine because they were too hard to wrangle. So this is kind of an interesting point. So what's your thoughts on that? Because when you start to see the enterprise get it right, they're going smaller, not bigger. They're going targeted on workflows where they know the data. So this kind of highlights that point. I mean, there's a lot of data that has not yet been trained or can be inferred on. What's your reaction to that?

Right. So I would say this is not just an enterprise problem. It is also the problem for AI-native startups. Many— most of the AI-native startups are actually consumer or developer-facing. And there are a lot of product data for them to understand what is their consumers' or developers' preference, intent, engagement metrics. So almost like all these application developers, I think all the application developers are very familiar with A/B testing these days and the product analytics. This is no different, right? So with GenAI, with various different models, with various different customized models, they need to have that telemetry. So, and also kind of continue to evolve their product by understanding user. So from that point, there are a lot of data being generated. From a user understanding point of view, and those data are not activated. And I would say there's a huge loss on their side if they're not activating those data to make the model they use, underlying model they use, to be smarter, faster, and more cost efficient. To answer your question about how to, like, what kind of enterprise probably has a lot of data as well, or already a lot of data sitting. And small data, a small model, a large model. Here's typical flow. So first, like, there's no question, like, there are a lot of interest to customize. It's just kind of customization is not easy. And we derive a lot of best practices in addition to the toolchain we provide from our offering. So first is leverage your data to customize model. And initially, usually, you want to customize a small model to see if, your data is actually valuable, is the right data for model customization. And small model can help you iterate really fast and for you to see if there's a good trajectory. If there's a good trajectory, then next step is actually to customize a large model. The reason is large model has much stronger logical reasoning capability. It can absorb the knowledge much better and then can learn much faster. So then you But you use a small model to kind of iron out your data issue and create a fast loop to like curate a high-quality dataset and then start to tune the largest MoE model. Usually those kind of tuning is very complicated because the model is very complex. Our underlying infrastructure requirement is very high. So that's where kind of we offer, we just completely take that concern away. From the researchers or developer. And then you have the largest, the most smartest customized big model. And then you go small from that point. From there, you can distill into a small model and teach a student to solve a specific problem from the biggest teacher model. So there are kind of, it doesn't go like straight line into a small model. It goes multiple hops. And that's kind of the best practice we have seen and very effective that way.

On your success, obviously the funding is great validation. It's good to have fresh money in the bank and get great investors. What's the success formula for you guys? Was it on the adoption side? Is it developers looking for that kind of observability for AI native or is it enterprise? Where is your sweet spot right now in terms of the momentum? Could you share some data on that?

Right, so when we just started, in my mind, I was, thinking about sequential adoption, that startups were most tech-embracing, would adopt that first, and then digital natives, which is tech-forward, then traditional enterprise. And usually they want to kind of wait until everything is, the whole entire toolchain is ready and so on. So that's kind of my initial assumption. But right now it's happening all across the board. It's kind of a reflection of how explosive this market is. And we have seen many traditional enterprises also move really fast. With that said, we have adoption across the board, the best AI-native startups and digital-native companies and the traditional enterprise as well. So to us, it's kind of very eye-opening to see cross-board explosion of creativity using GenAI. The people are reimagining what would be the new user experiences. They are redefining what does recruiting look like, what does outbound marketing look like, what does education look like, what does medical look like, healthcare look like, what does coding productivity look like. But coding is not just coding. Coding is not about generating code. It's also about generating business workflow, generating SQL, generating designs, generating website. So the variety of use cases is off the chart. And that's where we shine the most, is we shine the most where the biggest amount of sparkle of creativity lies and they need the best state-of-the-art tools to help them move really I love this market because, you know, the classic segmentation, okay, we'll start here, we'll sequence to the next position, we'll get to the enterprise.

You're a horizontal value proposition because people are transforming every part of their business, whether you're an enterprise or a startup. A startup could say, let's reimagine what recruiting is like. Let's build an AI-native app. Let's work with the models. And then, you know, Put the energy, entrepreneurial energy into doing that, and you change the category. The enterprises are sitting there saying the same thing, saying, hey, I want to change how I do business with my customer, hence it's kind of a reinvention, it's a transformation. So you're kind of horizontal. You're enabling that app transformation. You're transforming old to new. Now entrepreneurs are new, but they don't have the old. They're going to crack the code disrupt the old categories and their applications. And we're seeing successes out there on theCUBE and SiliconANGLE where these narrow verticals are completely transforming. That wouldn't have probably been venture-backed, but you can make a great feature with AI and actually change the makeup and the economics. That's the magic.

Yeah, that's the magic, and it's also kind of, This is the once-in-a-lifetime transformation, industrial transformation we're going, we're living through. And that's extremely exciting. I think we have been going through various different kinds of fireside chats. One of the questions I got asked is, what does next year's startup look like? My prediction is next year, the startups are going to build on top of many startups. So because That's where the maximum amount of creativity is going to derive. And it will change our day-to-day life. We already see that happening.

It's like a mosaic. It's like a mosaic of startups that together are generative. Can you imagine starting a startup and being like, hey, build me a solution, code-assisted, that then integrates in with another application? Start up and then sell to the enterprise?

It's already happening in our company and we have so many startups built on top of us and we're also using their next-gen AI-powered product as well. So it is a very— in this fast-moving time, we are building on top of each other. We compound each other's results. And this is something I've never seen. In the past, and this is a part that's—

It's an entrepreneurial shift. I mean, most startups have a formula. You get an idea, and they've changed how they do it, obviously accelerates it, but you're getting at how people are working together, and Jensen Huang at NVIDIA, I know his name, was an investor in your company. He calls it extreme co-design. I think this is a phenomenon that is this generation because co-designing is not just API calls. There's actual things happening between two different companies or platforms. You're kind of teasing that out, but this is a feature, not a bug. What's your reaction to this co-design thinking or systems thinking? Because if I'm going to work on top of Fireworks AI, I got to like know your system, right? Yeah, I have to like It's not just an SDK or an API or an agent interface. There's benefits to co-designing. What's your thoughts as an entrepreneur on this new phenomenon? Do you agree? And what would be your— if you don't, what would you add?

This is a very interesting point. So we have seen so many startups or enterprises build on top of us. And there are many of them took off explosively. And there's one common pattern among those companies is they do not consider model as a commodity or as a utility or as just as an API statically. They are actually co-designing their product and model very closely together. The fundamental reason is model and the product are misaligned by design. Model is developed by researchers in a frontier lab. And product is developed by this application product company, and they optimize for completely different things. So the misalignment is going to directly result in lack of quality, speed, and cost efficiency. So the successful ones figure out a way to leverage their product data and align— use that data to align the model. They use. So they basically boost up the model quality, speed, and cost for their products specifically through customization. And we call it product model co-design. And the end result is model, it become their IP. And model is not static. Model continues to learn from the new product data. So they create a data flying wheel. So this approach is extremely effective. However, this approach is very complicated right now because the whole entire space is a void and they have to figure out a lot of things by themselves. So that's where we build our product to. We want to kind of make it so easy, such a no-brainer to plug their product data and build this product model co-design in a very straightforward, quick to start way. Time to value will be within like a couple minutes. It should show the result. So that's kind of where we're heading towards.

And you get the fine-tuning thing. This is interesting because I've been having conversations about, you know, this has started. We go back months ago, DeepSeq made the big impression, but the distillation, what they did was interesting, right? So you're starting to see if you really get into the models and tie it in to your point, there's value there. It's almost, it's like coding software down to the chip level in the semis. So you get better access to the raw kernel, whatever word we want to use, the asset that could be there. Now, okay, now that's kind of an overgeneralization to kind of compare that, but what you're kind of getting at is that the more you lean into the model, you do it. That sounds hard. What's the—

It is hard actually, and that's why we're coming in to solve the hard problem, to make that a reality. The end result is Every single application company, every single product team, they will have their own foundation model. So think about that. It's a very powerful position to be in.

Yeah, well, we love the cube. We have our own Cube Foundation. While this interview will be converted into linguistics, into, into data, we'll get to capture that. We have our little language model, but we're just a small little island. But now I can connect it with other things to add value. It's a multiplier effect. How are people reacting to that? Do people— are people getting this? How would you articulate the uptake? And then where is the gap? I know you guys are doing a lot more research with your use of funds. How do you build on that? How do people understand this? Do they? And how do they build on top of it faster and keep up with the changes in the— on all the models? Right.

So actually, that's a huge challenge. On one hand, the benefit of open models, there's a lot of optionality and the application developer, they don't get locked in. They have a lot of choices. On the other hand, there are too many choices. Every, almost every 2 weeks, there's a new state-of-the-art model topping the leaderboard. And people constantly ask us as the trusted advisor, what do we do here? Do we keep chasing them? And then it feels like a lot of work. So that's where we come in and solve all the problems. So if you build a way to customize model, you can easily just don't want to change anything. You can port to try any new models.

It's like horse betting. You bet on one horse and you lose all in. If they don't, if they get superseded by the next horse, the next model, you got to pivot. So you're saying you can swap out almost port the customization. Any model. So Vertex Gemini hits this, OpenAI comes out, Tropic, the different, and also there's different use cases. The models actually have different applications too.

Exactly. You're absolutely right. So just to ask people, today's prompt tuning, prompt-based application building, people are very used to swap out all the frontier labs. Like they want to have optionality. And plug in all different offerings, API offerings. For customization, same thing. We want people to have optionality to test and try all different kind of open models they can tune and integrate with their agents.

So you're saying getting your model IP, basically what you're saying is build an IP around your model, work with whatever model, whatever tools best for the job. And then keep focused on that piece of it. That's the engineering product integration with the models. Did I get that right?

Exactly.

Okay, great. First of all, how do I get Fireworks going right away?

Very simple. Yeah, go to fireworks.ai and we have a public platform that's completely self-serve. So it's a very easy sign up and then you can use all the features. From our public platform. And we also have support. We have a chatbot. You can ask questions. We also have a Discord channel. Feel free to ping me if you have any feedback as well. My first initial, last name at fireworks.ai.

Well, I will get on this immediately with my team with the Q by AI model. We'll get on Fireworks. Great stuff. Final question for you. What's the plan for you on the use of funds? I saw in the release and in the news you're going to do some research expansion. The role of research has become less academic, more academic targeted towards real use cases and more engineering applied. What's your plan?

Yeah, so we are absolutely more focused on applied research. In that sense, there are actually a wide spectrum of academic research and we want to adapt them into the real-world use cases. In addition to that, we are a very innovative group of people. We create new algorithms and a new way to run the tuning process. The tuning is mostly post-training, post-training inference alignment, and inference speed and cost-efficient research. So we have our own innovation, many of our own innovation along the way as well. So on that front, we also are hiring the best talent across the industry who are the expert in post-training as well as numerics and inference optimization and building extreme large cluster and operate cloud infrastructure. As we plan to significantly grow our fleet size. So we're hiring across the board.

Lin, thank you so much for being an expert in our Mixture of Experts series on theCUBE. We really appreciate you and congratulations. And hey, you know, make things simpler and easier to use and reduce the steps it takes to build is always a great formula. Congratulations.

Thank you so much.

All right. I'm John Furrier with theCUBE here at the NYSE QUBE studios. Thanks for watching.