The Practical Realities of AI Development

All right. So today we have Lin Qiao, CEO and co-founder of Fireworks AI, which you can find at fireworks.ai. I guess their description on the website is, we are the fastest and most efficient inference engine to build production-ready compound AI systems. So Lin, welcome to the podcast.

Thanks for having me, Ben.

And today we're going to talk broadly about trends in the foundation model space. But I guess before we start, we should establish your bona fides, bona fides for the audience, right? So Fireworks, I'm a user, so I know it's a great service. Lots of models, I think over 100 at least on the LLM side, and then other types of models, visual models, speech models. But give us a sense of, if you're comfortable, some metrics as far as usage so that people can understand when you say something, it's coming from, you know, from actual high volume. Right.

Yeah. So we're under hypergrowth in the past 1 year. In terms of growth, we have grown our developer base by 18 times within a year and the revenue by 30 times. And right now we are processing more than 2 trillion tokens per day. Yeah. So, yeah, so we are, this is kind of the time I think to us it's very important to, to land very big impact in the industry. And this is the moment where creative, innovative applications start to emerge and build on top of GenAI models, those foundation models. And inference becomes the fast and efficient inference becomes extremely important, but also high-quality inference. Three dimensions, quality, speed, and cost are constantly in developers' mind. And we can dive into more details. And then that's where we specialize in, in providing the biggest value.

So to give our listeners a feel for the types of people who use your service, right? So broadly speaking, you don't have to go into details, but We're not talking only about Silicon Valley people, right? So we're talking some people are in regular companies and enterprises. So give us a sense of the demographics of your users.

Yeah, so we are, we are developer-facing company. So we build a GenAI developer cloud. And the developers come from AI-native startups. There are a lot of creative minds. They are experimenting brand new user experience that never exist before without this foundation model technology. We also work very closely with developers from digital native companies, and they have a very broad use base already of consumers and developers, and they are renovating their existing product with GenAI, and we can dive into more details. And we're also at the same time working with traditional enterprise where they are more looking to how to improve productivity overall and the many traditional enterprise has a very large task force covering various different spectrum of the work. So that has been an active development across the board. I would say to me, that's a big surprise in terms of progress we have seen across the industry. In my mind, it's always a linear way of startups are most tech-advanced, digital native. They are tech-embracing, tech-forward, and traditional enterprise will like to wait until everything gets mature. So So that's my adoption curve in my mind. And right now it's across the board. So that's kind of very unique, very unique phenomenon in this era.

All right. So let's go through some of the topics that I wanted to talk to you about. The first one is open source and proprietary models. So there seems to be a convergence in terms of capabilities, right? I think the most recent Stanford AI Index They compared the difference in performance from open source and proprietary in the previous report and the most recent report, and the gap has really shrunk. And so, first of all, do you agree? And then secondly, what does this mean for developers?

Yeah, definitely. So I think beginning of this year, so like before, When the first time when, uh, foundation models get— were adopted, it's only closed source, it's only proprietary. There's no open, uh, models. Um, only I think the first one is March 2023, Meta open, uh, open source the first, uh, Llama model. Um, that was very exciting moment, but, um, there's clear gap between open and closed. Um, and that gap remains until beginning of this year. Uh, we see a slew of open models from from DeepSeq, from Kuen, and many other labs. So this is a very exciting moment. So why open model versus closed model, right? So first of all, there are many use cases. It doesn't matter if the model quality continue to be better because the quality is already good enough to solve certain use cases. So that first of all.

And then there's fine-tuning as well, right?

So yeah. And second is a lot of developers or enterprise, once they hit product-market fit, then they want to own their AI strategy. They want to own the AI and fuel their data through their AI strategy. So they own the end-to-end. They do not create external dependency to, or use a centralized model provider where their data may actually benefit may be leveraged to benefit other people's application. So own your own AI become a fundamental areas of thoughts that many leaders and founders putting to their product thinking. And then the question is how to make own your own AI easy. And it should be really kind of very low barriers of interest to get started, and the result will be really, really good. Again, because the gap between open and closed models are shrinking, and that makes closing the gap actually easy. And there are— then the technology to close the gap can also— will also get standardized, and then it will make the toolchain much more accessible to average app developers. Today, the developers building interesting apps, they are like, they need to understand AI. They need to possess that deep knowledge. And my prediction is in the future, probably towards the end of the year, beginning of next year, more and more developers who doesn't need to invest deep time to learn AI will be able to leverage AI as a utility. So that's where we focus on. Fireworks, we focus on building this developer cloud to significantly lower the barriers of entrance towards better quality, better speed, lower cost using GenAI.

So, so actually, I don't know how you feel about this, but the way I kind of describe it is that there's pre-training and post-training. And the reality is most of us will live in post-training, very few teams and companies will actually pre-train foundation models, right? So number one. Number two, so on the question of this open versus proprietary, the one thing that concerned me up until recently, as you pointed out, so the Llama models actually got better, but then the cadence, Lin, of release was not frequent enough for me, right? So I know OpenAI and Gemini, they release on a regular cadence. Anthropic to some extent as well. But now that we have Deepseek and Qwin, it seems like, especially Deepseek and Qwin, they, they're really cycle. It's like almost every 4 months they have new models. So now the question is, will Meta increase their cadence? And if they don't, What does that mean for open waste models? Because the one thing that I still hear from enterprise, I'm not saying Silicon Valley or digital native companies, is that they're still a little hesitant about the Chinese models, right? For, I don't know why, for security reasons or whatever, right? So I just talked to someone last week actually from an enterprise and they were saying, yeah, no, we're not comfortable. We're going to stick If we could use the Meta, if Meta only released on a regular basis, maybe we'll look at Meta, but because they're not, we're going to use one of these proprietary models. So two questions, cadence, and then this kind of psychological barrier for the Chinese models.

Yeah. So I think DeepSeq is interesting, right? So beginning of this year, it created a huge wave. It's not just because DeepSeq model is really, really good. It is really good for the first time. I think many benchmarks, it beats closed models for real on complex tasks. So that actually force all labs, not just Meta for open model, but also closed model provider to— it holds a much higher bar from this point on. So I think in that sense, it actually is trigger a huge momentum of model, like base model advancement.

And DeepSeq has a history of regularity, right? So they're regular in terms of their release cycle.

Yeah. Yeah. We have been supporting DeepSeq model before, like DeepSeq become famous this year.

Yeah, yeah. I know. I mean, I've been following them. That's why I like the fact that you can count on them coming out with a new model, I don't know, every 4 or 6 months, right?

Yeah, yeah. We, we have the, we have the reputation of enable those state-of-the-art model on the same day.

Yeah.

On our infrastructure, same day of their model release. So our user can experience with it immediately. Coming back to the cadence of open model, especially from Meta, I wouldn't be able to speak for Meta.

Yeah.

But I do, I came from Meta. I do know Meta has a very strong team. And has a long history of doing AI way ahead of any other industry. So I think from just talent point of view, I still have high confidence that Meta will figure out how to— and I know they really want to bring this to a broader community.

And they have very good track record of building the community in open source We just want them to give us new models on a regular cadence like the other players, right?

Yeah. Yeah. So actually the more they hear about the needs for enterprise, the more they, they'll be motivated to, to kind of release more often. So I, I would kind of, we're their launch partner. We work very closely with them and happy to bring this feedback to them.

So, so, so, I think a lot of people in the West are sleeping on Qwen. I think Qwen is also upping its game. What's your sense?

Yeah, great spec by Alibaba. I think they are, they have very good coding model. They have good logical reasoning model. Yeah, so I, and also they have that cadence as well, right? So yeah, yeah, yeah, they are. So more interesting, I feel from open source point of view, It's open source is not a privilege from the West, right? So it's kind of, I feel like across the board, you know, the whole globally, we have very strong community of people believing develop in the open, releasing, I don't know, these manufacturing, releasing model weights and the benefit of bigger community. And then there's, there will be a lot of derivative. Come out of that.

By the way, I also like the new Gemma models, which are the open weights models of, I guess, of the Gemini family, right? And as you probably know, I mean, developers need a whole range of models, some big, some small, right?

Yeah. So Gemma, and I also heard rumor like that OpenAI will open source some of their models in their model family. So I'm looking forward to that too.

So do you hear the same kind of concern from enterprise about these Chinese models and security?

I think not.

It's more of a, like, I think, I'm not exactly sure why. I think it's just much more the fact that, okay, so let's We know that the Chinese models are good, but if the other ones are just as good, maybe it's safer to use the other ones, kind of is my interpretation, right?

Yeah, I think nothing's standing still in these days. There are, from our vintage point of view, I've seen enterprise moving how they think about things, right? So because before they're typically very strong as in, hey, you have to for data privacy and security reason, you have to deploy into my VPC. And, you know, no data leaves my premise, right? So, but because this GenAI wave, they're more open to use hosted, our hosted API. And because things move so fast and they will change how they think about the bounds and so on. So I will say, The Chinese model is new to them. So they become conservative and want to kind of double think. And that's normal. But all I see the trend is including enterprises, like startups are on a completely different spectrum.

Yeah, yeah, yeah.

Enterprises, they are also changing the way of thinking. So I think we're at the early, very early stage of adoption here. But I bet after a few years, the landscape and how enterprise is going to embrace this technology in a way that's very different from today is very likely to happen.

So, so it seems like in the last 6 months, the 2 main developments in these foundation models is one is reasoning, right? So there, the open weights model seem to be close to being on par, if not on par or better. And then the other development is multimodal, right? So there, I don't know. So you can dissuade me from this perception, but my perception is the proprietary models are still better.

I would say the proprietary models are— first of all, I want to establish a baseline. There's no model that's the best across the board.

Yeah, yeah, yeah, yeah, yeah. I think the— First piece of advice is don't architect your application so that you're model agnostic, right?

Right. And the fundamental reason is like one model solve all the problem in the past, like compared with others is a flawed claim.

Yeah.

Because it's not how training happens, right? So think about model quality in an extremely simple way. Extreme single way is your training data distribution, the distribution from training data is perfectly aligned with distribution of your inference workload. Then—

Which is never the case.

Which is never the case, right? So, so it's like because researchers need to build hypothesis on what kind of problem they want to solve the best. And you cannot solve all the problem in the best way. The world is too complicated to model that way. So you have to think about the problem solving in a lower dimension. And that's why different model will be really good at solving certain kind of problem really well. So that's kind of the worldview. It doesn't matter whether it's closed model or open model. They are all very strong in certain dimension. And what we saw, the closed and open model, smallest gap area is coding.

Yeah, yeah, yeah, yeah. I agree. Yeah, yeah.

I think there's fundamental reason because coding is very easy to verify.

Yeah.

That it's correct or not. With that signal, you can reinforcement learning.

Yes. Reinforcement fine-tuning even, right?

Yes. Yeah. Right. So that's kind of fundamental reason. And that technology is well adopted in many labs. So, but the interesting part is many of the complex problem in the world, you can map that into a coding problem. Okay? And then guess what? We have very good coding models you can pick and choose from, whether it's a closed model, open model, right? And then once you solve the mapping problem, then you can go very far. So I think we have start to see that a lot of innovative applications start to think that way, right? And we have seen like they push huge mileage in the adoption.

So by the way, when I say multimodal, I mainly mean audio, video, right? So from that perspective, at least on the foundation model side, maybe it's just because they're easier to use, the proprietary models seem to be what I'm gravitating towards, right? So Gemini, OpenAI. So I'm not sure how the open weights models are doing with those type of data. So can you— do you have a sense of how good the open weights models are with video and audio?

Yeah, so I think it depends on the use case. If it's a very general purpose use case, that you just want to put some simple audio on top of LLM-based application experience, then you can go to OpenAI and Google to get that hook on easily. But we often see application with voice. It's not just like voice in, do some kind of process, easy simple processing like translation and voice out. It's not a very simple flow like this. Oftentimes they want to inject pretty interesting processing of this information in the audio latent space and in the LM latent space. So then if you have one model and, and for people to tune with all the signals combined is a really hard problem. So instead, if you have 3 models, audio in, LLM for intelligence or intelligence at the audio space, and then audio out, voice out, it's a much, much trackable problem. So we have seen, like, depends on the complexity of your use case and people prefer different constructs. And that's why in audio space, often it requires a not It's not just a simple API, it requires a pipeline that developers need to manage. So yeah, so that's audio. And the vision—

And Lin, in the audio space, so maybe it's a UX, DX, developer experience problem more than anything else, but what's the equivalent of like an ElevenLabs for open weights models, right? Where it's much, it's, it's, uh, simpler, right?

Yeah, so that's we're building, right? So, um, I, I think ElevenLabs has a lot of data. They, they are trying to tune, uh, their own models, uh, that combines, um, you know, in and out together, um, and, uh, end-to-end. Yeah, end-to-end together. And we have been focusing on just kind of the decomposed flow and easy customization. Uh, I think those are just different design points and we can also use ElevenLabs model, right? So whatever works for our customer.

Until someone on the open, open weight side does an end-to-end model that you can use, right?

We were definitely open for that. And that has been a very common question, I would say.

Yeah, yeah.

Among the audience. So the key value of end-to-end is latency.

Later.

Yeah. So for a lot of real-time audio application, they want very fast speed, but we have also enabled streaming across 3 models. So as long as, you know, your time to first token or time to first 20 tokens is fast enough, then the rest will be— you just, you don't need to be extremely fast. You have to maintain a human speaking pace, right? So So yeah, so like real-time audio is definitely a rising application domain.

So how do developers then balance— let's say they're already committed to open models. Let's just say they're even committed to DeepSeq, right? So DeepSeq, there's R1, which is reasoning, and V3, which is non-reasoning, right? So when should you use one or the other?

So yeah, DeepSeq V3 is the non-reasoning model.

It's because the reasoning is more expensive and slower, right?

Yeah. So the reasoning is basically like you ask me a very complex problem and I'm speaking loud how I'm thinking through. And Arthur in that sense is kind of quite verbose. It will spit out chain of thoughts. In 3 different ways.

But, but, but the, the scenario I'm painting is I'm a developer, I have no experience in machine learning, right? So how do I know which one to use?

Oh, I see. So usually people use logical reasoning model, which basically by the model thinking through different way to solve a problem multiple times. The solution will be much better, right? And the logical reasoning model I use for solving complex tasks, for example, planning. If you're building agents, typical flow is you need a step to plan what are the executions, right? So sometimes you plan like map out to 30 to 70 steps.

Are there people who are basically, basically Here's my prompt, and then there's some router that will route it to the right model based on the prompt. It'll go to R1 or V3 depending on the prompt.

Yes, so this is actually like being used by many, many engineering developers is, you know, they want to first kind of build out a plan, and once you have a plan, and then you route to, you know, each step you route to a specific model to solve those. They are the executor, right, or worker. So, and DSV3, DeepSeq V3 is a very good conversational or coding, a code, coding assistant model that we have seen a lot of developers use it as is, or a lot of developers, they tune the model. And then by the way, like tuning DeepSeq is not very easy because that model itself is very big. And just managing the kind of training infra is hard. I think we are the only provider offer DeepSeq fine-tuning right now.

Which just V3 or R1?

Both.

Both. So R1 is reasoning. So there's supervised fine-tuning and then there's reinforcement fine-tuning. Which do you offer?

We already offer supervised fine-tuning. We'll offer reinforcement soon.

So reinforcement fine-tuning. So I'm trying to think through what— so supervised fine-tuning, it's the UX is dead simple, right? Here's my labeled example. So I upload it. I walk. I go away. And then I come back. It's tuned. So reinforcement fine-tuning, walk me through the UX.

Yeah, so the fundamental way for reinforcement learning to work is you are not saying, here is absolutely the right answer for this prompt, right? So supervised fine-tuning is here's the problem, here's the generation, that's the SCORM tube. And really—

Yeah, it's a very kind of— for anyone familiar with machine learning, it's very familiar, right?

Right. So reinforcement learning is saying, I will, for the prompt and result, right, I'm going to rate you. Are you going the right direction or not? Basically, that's kind of the—

Yeah, yeah. So what would be the expectation from the developer? What do I have to give you?

Yeah. So that is a very interesting question, first of all, because it depends on what kind of developer, right?

Yeah.

So if it's machine learning engineers who are—

By the way, this is why I think this is mostly a UX, DX problem, right?

It is both a UX problem for the ICP. It has to be specific target ICP skill sets, as well as a pretty deep technical problem on both the algorithm design and infrastructure.

Well, the technical problem I give to you, right? So you guys will take care of that, but the UX—

We will also solve the UX problem because this is about adoption.

So if my problem, Lin, is something that has a clear validator like coding, done, right? So it seems like that's doable. But if it's a little more subtle, what do you guys do in that case?

So there are cases where there's objective metrics to validate on. There are cases there's subjective And so, and also depends on ICP. If it's a machine learning engineer who understand the training process, then the UX will be, you, we give you tools to manage reward, right? Reward function, right? And oftentimes reward is multifaceted. It's not just one reward. There's a rubric. Around it. But for average application developer, if you ask them to write reward, they are like, and look at the metrics and adjust it, they're like, they're completely lost.

Yeah, yeah, yeah.

And that's not going to work. And we have to build something.

Yeah, that's why you need to hire some really good UX people, right?

I will say it's not just UX, it's completely different product design.

Yeah, yeah, no, but I mean something that there's some innovation that has to happen.

There's a lot of innovation that has to happen here. But I'm very bullish that a brand new experience for application developer to improve their quality using to align the model distribution with production inference workload distribution, that space that product is solvable. And, you know, yeah, yeah, yeah.

So I guess we agree that supervised fine-tuning, done. It's a solved problem for developers.

I wouldn't say it's a solved problem. Actually, we talk with so many developers. We offer, first of all, we offer fine-tuning platform. It is not easy to solve that problem, to label.

And oh, yeah. Yeah, I get it. Yeah, yeah. But, but it's at least within the realm of if I can label, you guys can help me. The reinforcement fine-tuning, there still needs— there's some innovation that needs to be done. All right. So I don't want to end this podcast without asking you about agents, which is the hot topic this year. So any, any high-level observations or even granular observations about agents and any observations from particularly the people who are using Fireworks?

Yeah, so this is your, I would say, Fireworks grow very, very fast, but also because a lot of that is coming from application developers building agentic workflows. And Our current observation is people are building single agent. They're solving single agent problem, not multi-agent yet. And I know many people ask me about multi-agent. I feel like we need to take one step at a time.

That's exactly how I feel.

And because you don't have the stability, if you haven't have a stable single agent to build on top of, you're building on top of a sandcastle. Right. Coming back, but we do see a wide variety of single agents solving all kinds of problems emerge.

So, so, so these are people using Fireworks who are already, so are these pilots or are some of them in production?

There are many of them already in production, right? So they are building coding agents where you don't write code, you prompt.

Live coding. Vibe coding.

Yeah, say what you want and boom, you generate, or boom, modify your codebase and give it different. So, so that is, that is very natural progression.

So what other examples do you have besides coding?

We also have many companies building, like, I would say, document process agents where The documents are in— not in plain text, right? In PDFs, in screenshots, uh, in pictures, images of, of certain documents. Um, and though, though— and then document processing.

So this is like information extraction and OCR kind of agents?

Um, beyond that, uh, it is, for example, medical claim agents. So those All flavors are legal.

So are these— so again, are these pilots?

Oh, those are already in production.

Wow. Interesting. And they're obviously, since they're on Fireworks, they're using open weights models, right?

Yes, they're using open weights models.

So the people who are going down and deploying agents in productions, they also do fine-tuning.

Almost always they either use our fine-tuning platform or, or we help them to do advanced fine-tuning. Yeah, so, so that's kind of where the last mile alignment is very important to get the quality you want. And we have done so many. Our observation is we need to significantly simplify that step, and supervised fine-tuning is too complicated. For average application developer to do it themselves. And our job right now is kind of lower the complexity here.

Why don't people just use proprietary models to generate fine-tuning data?

Yeah, so—

At least that was 2 years ago. That was what people were doing, right?

Yeah. So not necessarily the quality is the best always, right? Because even proprietary model doesn't cover all the grounds. And oftentimes it requires a human expert to label data. So, yeah, so it's very time-consuming. It's also very— you have to invest money. And at the end, you may not get the quality you want. And then it's a very frustrating experience because it just kind of takes time, takes effort, there's no guarantee, and so on. So I think what's needed to happen here is this quality customization needs to happen quickly. You can iterate quickly, and it's low cost and low lift. So that transition needs to happen, and that's what we have been focusing on.

Do you have examples of developers on your platform platform who've built agents in production that are using tools, tool-use agents, kind of, you know, tool-use agents? Yeah, I think many, or that, or that's too early and people are still in the pilot stage for that?

No, no, many people use function calling tools in production. And yeah, we have many customers.

What about MCP?

MCP is quite early. Yeah, definitely see.

It seems like it's a mixed bag, right? So some MCP servers are really good, some not, right?

Yeah, because it's early, right? So I think that's expected. But something like MCP is needed in the industry to create a standardized way for tools to integrate with. So that's kind of the meaning. Itself, it's just more like a standard, and you have to kind of put the meat behind it. And how to do quality control of your MCP server and so on, I think that's still early in getting adopted. As you remember, OpenAI did a similar thing, and there are a lot of traction. And after a while, it starts to kind of stabilize. So I think we're going through that curve with MCP right now.

So we talked about proprietary and open, right? So the other topic I wanted to ask you about is the— and, you know, I mean, I think you've alluded to this. Teams need to have a collection of models, right? So not one model will solve their problem. So are you seeing people, Lin, gravitating towards, let's say, the smaller Gemma models, or taking a larger model and doing some distillation? In other words, you know, if you can distill a model, fine-tune it, you can run it cheaper, it'll be faster. So are people taking that step now, or Is that something like, I'll do that later once I am comfortable with my application, I'll refine and make my model even more efficient?

So it's always almost like after they find product-market fit, then to scale, and then they will invest a lot more effort in customization. And we make customization very easy as a platform. But when you are in pre-product-market-fit experimentation, it doesn't make sense to invest time to optimize. So I think that kind of makes sense. And yeah, so—

So what's your sense long-term trend, Lin, of these smaller models? Let's say 5 years from now, will we have these 8 to 10 billion parameter models that are going to be as good as the models we have today? So in other words, so are the smaller models going to be get better and better over time?

We're definitely seeing smaller models get better. But smaller models is just laws of physics. It has orders of magnitude less number of parameters. That just means it can learn, absorb information.

Maybe—

But it has capacity.

Yeah, yeah. Maybe for very targeted tasks. Yeah.

So for now, we are— We are— our platform, our kind of quality customization side of platform, has been used to actually improving small model quality. Therefore, the speed and cost is much lower. And this is much needed for a lot of our customer who are real-time application, real-time consumer-facing, developer-facing application, where speed is extremely important to optimize for. And the large models, has good quality, but it just cannot satisfy their, their real-time requirements. And therefore they have to go small. Once they go small, they have to bridge the quality gap, and therefore they use our RFT or SFT platform to get there. So yeah, so I think that's the trend on how to make that customization, quality customization to small model much easier and streamlined, and that will drive a lot of adoption.

So, so in closing, is there anything, uh, that's happening in, uh, on the Fireworks, uh, platform on the foundation model space that we haven't discussed that you think people should be paying more attention to? So we discussed about reasoning, uh, multimodal, uh, open models. So anything else along those lines that people are sleeping on?

Yeah, so we, we basically, we have, we are a multi-line product platform. So our, the lowest level of our product is virtual cloud infrastructure where we build on top of, I believe right now, 7 cloud providers and spanning across 30 different regions where our developer doesn't need to worry how to and where to procure GPUs and know how to, if a region is completely dark and worry about, you know, sustaining their user-facing SLA and so on. So we take away all this complexity from them and they can just kind of only think about building applications on top of foundation models. So that's the lowest layer.

So I guess, I guess my question is more, are there things happening in the foundation model space that people are not aware of that you think they should be paying more attention to? So let me throw out one example to you. So should people pay more attention to AMD hardware or inference, right?

Yeah. Yeah. Yeah. So I think in the long run, as any free market, there should be competition when the profit margin is high. So I think in the long run, there's no question.

What is the state of inference for AMD right now in terms of ease and capability and software stack? And what's your— give us a sense. Yeah.

I think a few angles to look at it. One is hardware. I think AMD is closing the gap with NVIDIA, the hardware point of view. And then the next question is, can software extract the flops and memory bandwidth from the hardware as designed. I think they are also working really, really hard to close that gap. And yeah, so I think—

But you're not on the Fireworks platform. It's not like there's a lot of people asking for AMD.

Usually people don't ask for specific hardware because we provide the abstraction. We're responsible for kind of directing the traffic.

Or you guys yourselves maybe be, you know, if you can get a better deal with AMD, you would use it, right? So, and if it's something that's usable, you would use it, right? So.

Yeah. So I think as we, the whole entire team started from PyTorch, right? PyTorch is, has integrated with many hardware backends and different hardware has its, it's like no one model fits all. Also, I believe like hardware has different SKU has its own unique strength. If fitting to, if it's one SKU fits the best into certain workload pattern, and we will do the best matching and mapping to specific hardware strength from the application workload. So that's kind of benefit of using Fireworks is we take over that complexity and do the best job of extracting the most out of hardware.

And with that, thank you, Lin Qiao.