Closing the Loop Between AI Training and Inference [Lin Qiao] - 742

The purpose of those models is to serve product. The model by itself doesn't have any meaning. If model training doesn't move the needle for product, it's not meaningful, it's not useful. Product A/B testing is the ultimate judge whether your model investment is paying off or not, is successful or not. So actually, the fast iteration experimentation loop has both training and inference combined together to declare victory.

All right, everyone, welcome to another episode of the Twiml AI Podcast. I am your host, Sam Charrington. Today I'm joined by Lin Qiao. Lin is CEO and co-founder of Fireworks AI. Before we get going, be sure to hit that subscribe button wherever you're listening to today's show. Lin, welcome to the podcast.

Thanks for having me, Sam.

I'm super excited to dig into our conversation. We're going to be talking about a pretty broad variety of topics, but really digging into AI inference and enterprise generative AI. Before we get going, I'd love to have you share a little bit about your background.

Yeah, so, um, I started my career as a researcher, uh, after I graduated from— I get my PhD, um, and, uh, by my work, um, for research, it become 3 different products in, in the company I worked at, and I become software engineer, uh, and then I joined Facebook, now Meta, to drive the AI initiative. It's kind of full circle in my career to do research, go build systems, and come back to system research, co-design space. So that has been a very fun journey. And that the visibility we got from driving the AI work at Meta really gave us confidence this is going to be a huge industrial transformation. And that's our motivation to create Fireworks, to help all developers in the industry using AI in a very easy way.

Awesome. So what is Fireworks and what led you to start the company?

Fireworks is a GenAI SaaS platform where you do not need to set up anything, do not need to worry about how many GPUs you need to kind of deploy. To run your application, where is the location, how to manage reliability, cost, performance optimization, none of those. You just onboard to Fireworks and integrate with your product and boom, it will auto— optimize itself. It will help you save cost and scale quickly to millions of developers or billions of consumers, and it will manage all the production operation complexity and let you focus on application product development.

Let's maybe dig a little bit deeper into that. Talk a little bit about the way you expose the platform to developers.

Yeah, so a little bit about our background. We have a pretty big founding team, 7 of us, been working in AI space and building AI infrastructure for both training and inference for a long time at Meta and Google. So we really see the developer needs are not uniform. It at least broke into two phases, if not more. In the most simplistic way, the first bulk of developer work is experimentation, and that's where you want fast iteration, simple integration, lots of control. And validate your hypothesis. What can work, what may not work, and fail fast. So that's kind of the speed of iteration is the most important, productivity is most important. Once you find product-market fit, maybe 1 out of 10 ideas survive for you to scale into production, then it's a completely different set of problem you need to tackle. That's where you need to worry about cost efficiency to have viable business. You need to worry about latency to bring this interactive experience of consumer-facing, developer-facing product live. You need to worry about quality to the next level, quality to the next level of, hey, how does model quality really feed into, mold into the product needs? So all of that becomes a different set of concerns. So we are, we have seen issues only focus on one versus the other.

Meaning only the fast iteration or only the high efficiency scaling.

Right, right. So because if you only focus on large scale, then people who are at the early stage of experimentation, will not be able to use that platform. You cannot force a later stage product into early stage in the funnel. And second, if we only focus on early stage, then when people want to scale, they need to hunt and find another platform to leverage, and there's a lot of migration overhead for them. So to build a seamless, easy-to-transform, multi-stage product for developer, is absolutely necessary. And we have validated that during the PyTorch days. The big part of founding team has been building PyTorch for 5 years at Meta, and PyTorch now is the most popular framework in the industry, especially when it comes to GenAI. Almost all models are written in PyTorch and deployed in PyTorch into production.

I was actually going to ask about PyTorch, so let's maybe take a second and go there. I feel like you undersold your background a bit. You were the head of PyTorch and kind of guided that team through a pretty interesting time in the evolution of that product. I would love to have you talk a little bit about like lessons from that experience that are important to you as you, you know, first frame out the problem that you articulated for Fireworks, but then as you're building the company. Like, what did you learn that was most salient to your current experience?

Yeah, so I think there are a lot of lessons learned, I would say.

I can imagine.

Because at that time when we started, there's no hardware, no software, no team focusing on AI first, right?

Meaning at FAIR or?

And yeah, and Meta in general. Obviously, Meta was on the cutting edge, on the forefront of creating a technology space, innovating there ahead of the industry. So there are a lot to figure out. And during that process, we try many things. Some idea doesn't pan out, and we learn a lot. And that's kind of lead to where we are today. I think one of the big lesson is we thought these two loops, as I just talk about, the iteration, fast iteration experimentation loop and the production loop can be completely decoupled. We should use completely different system optimized for different things and build a bridge across. Okay. By building a bridge, that means we are going to do conversion. We're going to convert. The research outcome for doing experimentation or a lot of model tuning training results, convert that into like production, a different set of system optimized for production, and then you get the better of both worlds, right? The idea sounds plausible. However, this conversion is taking forever because these two systems are not designed to be compatible with each other. A lot of compatibility is numeric compatibility. You want to preserve the quality, high-quality results from experimentation, from customization, from training, whether it's pre-training, post-training, into production. Any conversion is going to cause loss of precision, but at the same time, it's additional work to do this conversion, and that can actually slow down significantly. The second is if you look at the product lifecycle, model training doesn't— the quality work doesn't finish at offline model training, as in you have a bunch of eval datasets and you measure the offline metrics and the metrics look good, you're happy. No. So actually you have to continue into deploy the model into inference and the product point to the model to start to do A/B testing. Because those, the purpose of those model is to serve product. The model by itself doesn't have any meaning. If model training doesn't move the needle for product, it's not meaningful, it's not useful. Product A/B testing is the ultimate judge whether your model investment is paying off or not, is successful or not. So actually the fast iteration experimentation loop I just mentioned has both training and inference combined together to declare victory. And then you know, hey, do I have a product-market fit or not? And then I know, we know whether we need to scale into production or not. And then the second loop starts. So even the first loop requires training inference fast iteration. And if you break it into, oh, training user system, inference user system, then you are like being stuck because that velocity, you lost it right there. So that's where like have a cohesive system that do training inference alignment and cross-deployment quickly is a necessity for experimentation. That's a big lesson learned. And the second is the inference system for experimentation needs to be the same as inference system for large production, because once you find a product-market fit, the immediate thing product team is going to do is, oh, I did— I route 1% of my traffic for doing A/B testing. It looks good. I want to ramp out. Ramp it up quickly to 10%, 20%, 50%, 100%. No time. I cannot wait. So, so that's why kind of the whole end-to-end need to be the trend. The transition need to be extremely smooth.

And even if you could wait, the product team, once you did whatever conversion, the product team would still want to start at 10% and ramp up slowly because that's part of their testing process.

Yes. And if this conversion takes a month, they're like, oh my God, like we are just doing nothing but waiting for this to happen and this is not working for me. So, just so you know, like the product team is heavily depending on the fast iteration with their users to understand the new product design is landing well with the customer or not because there are a lot of hypothesis. What product feature become popular? What, drive user engagement, drive time span, drive a lot of metrics, but it's not validated until they see the A/B testing results. So that's the nature of fast iteration. So there's a fast iteration with the end user also that needs to happen that cannot be slowed down. So that's where like this whole GenAI SaaS platform design cannot just be one-sided on kind of any one phase of the development lifecycle, of the production lifecycle. It needs to be end-to-end, very smooth transition. So I will say that's gonna be the biggest lesson learned. And that's why in Fireworks, we always think about ourselves as we're building towards the end-to-end developer platform. And we start with inference because inference is the foundation of this end-to-end. Inference is foundation because if you think about post-training, so we build on top of open models. One thing I want to clarify is Fireworks, we do not do pre-training. We do not build foundation models because today the open model quality is really good, especially this month is the gift showering all high-quality models. We can talk a lot more. I'm very passionate about that. So we, in the open world model world, a lot of people are doing tuning and customization. And part of the popular tuning technology is called RFT, or reinforcement fine-tuning. And if you take a deeper look at reinforcement fine-tuning. It is training, it is inference for rollouts, and actually even for tuning, it has a lot of inference dependency. And obviously after tuning, you need to deploy to small-scale A/B testing. That's inference. And then scale to full production traffic. That's also inference. So that's why inference is a foundation of this end-to-end. We started with inference at the beginning, and then we added supervised fine-tuning. We added reinforcement fine-tuning, all different kind of tuning customization technology built on top of inference.

Interesting. When I think about inference as a starting point and then your evolution to more of an end-to-end platform and the needs of post-training, I think of like the, the very different abstractions that you might offer to do that. Like, you know, for inference, I might just want to plug into LangChain or do like an OpenAI compatible thing. But then for fine-tuning and training, you know, I might want to take a different approach or have some, you know, very tailored user experience for this since it's much newer and Um, you know, there's not a dominant kind of approach to that, that user interface or that, or developer interface. I mean, can you talk a little bit about the abstractions that you offer developers and like, you know, how you've kind of managed the offer best-in-class experience, but also, you know, help them avoid lock-in or remain compatible to other directions they might want to explore.

Yeah, absolutely. So the inference abstraction and post-training abstraction is completely different. So I will start on inference. Again, think about there are people using inference for pre-product market fit stage. They just want to validate if GenAI is the right tool to solve their product problem or not. So that's why we choose OpenAI compatible. API, obviously everyone choose OpenAI compiler. It's just kind of easier to speak the same language. But then when they graduate from product-market fit validation phase to scale, then there are a lot of customization for inference itself to be high speed, cost efficient, and also quality need to be really good. So we call it three-dimensional optimization. We build a 3D optimizer to drive the customization for our developers without they understand deeply how to customize their inference deployment to satisfy their needs. So that is a layer of abstraction we created, very similar to a concept of query optimizer for databases.

SQL databases?

Yeah, for SQL databases. It basically, the data engineers or analysts, they just need to write SQL, which is an abstract way of describing what semantically is the query. And this database engine has a rule-based or heuristic-based optimization path to convert that into a very efficient execution. Right? So, but the idea is very similar, but database query optimizer is one-dimensional optimization, only optimize for latency. So we drive three-dimensional optimization for inference across quality, latency, and cost all at the same time. When we just started, the bad news is extremely complicated optimization space. The space is big because the technique we need to tweak, pick and choose from, the options we need to pick and choose from has a lot of components. Each component has its own subset of options and they stack with each other, leading to more than 100,000 different combinations. And then it becomes a search problem. Optimization is always a search problem. It's a very search problem with a large search space. And then how do you find, how do you navigate that search space to find this one option efficiently is the problem we solved.

Can you give some concrete examples of points in that search space or, you know, the different, I'm assuming we're talking primarily here about different backend configurations, but Maybe the first thing to do is to have you elaborate on, you mentioned these 3 dimensions, but those 3 dimensions are kind of a front end for lots of dimensions of change on the back end. Like, what are all of those dimensions of change?

So there are many dimensions. For example, for inference, we do fully disaggregated inference. That means we chop the model execution by the system bottleneck, right? Some part of model execution is fully bottlenecked by compute, some are fully bottlenecked by memory access, some are fully bottlenecked by networking, and we disaggregate them and scale them independently, right? And then how to scale them depends, really depends on the workload pattern. And the performance characteristics that the product team is looking at from the product requirement. So, that is one way to look at it. On top of that, there are various different numerics you can apply in terms of position. And when you apply position, obviously, you're— and this technique is called quantization. You can quantize so many different things. And there's a lot of options in that space, not just you can pick different position, you can also quantize weights, quantize activation, quantize communication, you can quantize many things. But at the same time, when you quantize, by default, your quality goes down. Sometimes if you quantize naively, your quality goes down by a lot. So a lot of product use cases cannot have that trade-off. They cannot trade off quality with speed, quality with cost. They want to preserve quality, do not degrade, and then improve the other two. So then we have a tech, you know, and then we have a way to preserve the quality while doing this optimization. We also have different kernels. Implemented for optimizing for different context length, you know, the long context, shorter context, and, you know, for generation speed. There are different options we can pick and choose from there. Obviously, we also have optimization for different hardware backends because different hardware have their unique strength we can leverage. And yeah, so that's why kind of the search space is very big. And you cannot manually do all these kind of try out these combinations. So it needs to be done in a systematic way.

It also sounds like something that— there are aspects of what you're describing that don't seem to lend themselves to doing that kind of optimization on the fly, meaning it's more like design time optimization, or maybe better, deploy time optimization. Like I have specified a model, I've told you how I'm going to use it. Now I press the deploy button and then you like figure out the best way to deploy it as opposed to I'm sending in an inference request and it could happen in one of N ways based on, you know, the content of that request. Is it more the former than the latter?

It's a combination. Some of the optimization we can do during runtime, some optimization we're going to push before we deploy. Yeah. So, so it's like how compiler work. There's JIT. There's AOT, so yeah, so we're doing a combination of both.

Okay. Okay. And then it sounds like most of what you've talked about is like infrastructure tuning types of, or both architecture and tuning types of optimizations, and they are independent of not independent of the model, but they are kind of assuming a static model. And now I'm thinking of this in contrast to like at Google I/O, Google announced a Vertex Model Optimizer, and the idea is to provide a frontend to all of the model choice that a user has. And so instead of specifying a specific model, they say, I want to prioritize speed here, or I want to prioritize quality here. Those, the dimensions of those two offerings are overlapping, but the, the approaches sound very different. Are you also thinking about that kind of model abstraction?

That's a very interesting topic because, um, oftentimes we see a desire from many developers. They want to be model agnostic. When they build a product, they don't want to get, you know, just bundled with a model because this space moves so fast. Every, almost every week there's a new model being announced, whether it's closed or open. There's some models breaking some record somewhere. And there's a huge amount of desire to be model agnostic.

Within a given product, there's, you know, a given product might have a dozen or more tasks, and each of those tasks may be best served by a different model.

Absolutely, yes. And the interesting observation is it also depends on the product lifecycle itself. In the early stage of product development, when, when people are not sure, you know, which part of this product feature can be hardened yet. They typically would pick a powerful model to begin with, just kind of ensure the best quality. And over time, they got a lot of user feedback, right? There are a lot of, again, there's A/B testing, there's kind of product metrics to monitor each product feature, how people's reaction look like and how they kind of continue to polish it and tune it to the point it start to emerge, oh, those features went too hard. And typically those feature has a narrower problem definition and that's where they start specialize and customize. So we see a clear transition and continuous transition. It's continuous process. It's never done one shot from, hey, I have a new product idea, I want to try it on the most powerful model to, oh, now I know exactly what I want to build and I'm gonna narrow down and use a special Customized model. And then the customization process can be I pick another smaller specialized model from off the shelf, or now I have bootstrapped my product, I have a lot of data from production to tune a model specific for my application. And there's a lot of power to unleash from there. Because no one has that data, not the frontier labs, nobody. And with that data, you can overfit your model into your application. That's what you want to do. Actually, in this case, overfitting is good. And therefore the model is designed for your product, make your product better, and your better product drive more user engagement. You collect more data. Make the model better, better product, and so on. So this is a virtual cycle we start to build. So we see a lot of success for early pilots being able to master that virtual cycle, and they emerge as leaders in current generative app space. So what we are convinced to do and to offer value to this community is to make those tools accessible to everyone. So it should be easy, it should be standardized, but when it comes to standardization, it's a messy, it's a very messy place. One is, just let's talk about one technique that's getting very popular and also we offer on our platform, it's called reinforcement fine-tuning, RFT. So the most critical part of OPT is to write an evaluator that takes a model output, give a score, right? That score is called reward. You give a very big positive score, that means you really want to, the model doing great job, you know, you can nudge it to the right direction. Or the model, a big negative score means the model is, you know, really bad, the result is BS, and you want to strongly discourage it going that direction. A lot of reward you can write in code and in like, it can be executed in isolation, but a lot of time the reward is it cannot write in code. You have to call into an application-specific internal API to extract knowledge. In information that is deeply coupled with the product. Okay. Because a lot of time those kind of score generation or reward generation depend on internal state.

What are some examples of those types of reward or reward evaluation functions?

Yeah. For example, If it's a math problem, right? Or if it's a coding problem, it's very verifiable. You know, yeah, you know, absolutely here is the ground truth, right? So for highly verifiable rewards, you can either code, sometimes you can just code in your reward function itself, or you call into a, you know, ground truth API to get the result back, right? So that's kind of, that's one way to do it. But sometimes the reward is subjective, right? The reward may need to come back from the product. It's a multi-turn chat and people engage with the product, thumbs up, thumbs down. There's a way to read, hey, you know, so then it becomes a reward environment integration problem, which is messy. So there are so many different places you can extract signals and do integration. Whenever it becomes integration problem, then it's a lot of work. It's just a lot of work. So I think this is a big space calling for standardization and we are gonna put our effort to drive towards that direction because we believe this is a much-needed void and much-needed effort to put together by the whole entire community, not just us. So we'll kind of launch something towards that direction very soon.

And it sounds like that might be an open source type of tool or—

Absolutely. Yes.

Okay.

Absolutely. I mean, the only way to run this is through open source, and we are very familiar with open source given our background and anything that benefits the whole much bigger community, open source is the best tool.

You know, we started off talking about kind of this convergence you observed from the PyTorch days between inference and production and the desire for teams to do online testing, A/B testing of a model once it's put into production. And what it sounds like you're describing here from a reinforcement fine-tuning perspective for these very application-specific problems. So again, not coding, not math, is like almost bringing the fine-tuning into your online application operations and, you know, doing online training of the model. Like, how close are we to that type of an environment where, you know, the model's constantly being trained, it's constantly looking at signals that users are, are given up and down and like using that to refine itself.

Yeah. So I'm very bullish on that direction, right? I think, I think we should have that. The biggest challenge today is for us, in order for us to have that, people need to write what is the expectation. For the model to grow into that expectation. Okay. It's like, it's like when, if we want to test our code, we need to first write unit test, right? And unit test is basically, I expect this piece of code to behave this way, right? And I'm going to test it. And then I know, oh, if the code is doing a good job or not, right? So that's kind of similar to reward.

I hear you starting to describe this scenario where what you're saying is like, we've just kind of kicked the reward can down the road. Like before, like it was like this micro reward. And then if we, once we integrate this all in, you know, we still have to describe this reward function, but it's more of like this long-term over time reward function that's, you know, maybe just as hard or harder to describe than the a micro reward? Like, is that kind of the direction you're going?

I don't think we can kick the can down the road because without a good evaluation mechanism defined, many things are harder. All the things we talk about is going to be harder. For example, earlier we talk about, hey, people do want to be model agnostic. But without evaluation criteria defined when they switch models, they don't have a way to measure, should they switch or not? Anything is going to regress or not? Is it going to improve? I don't want to be surprised. So that is evaluation. And evaluation is also needed when, like for RFT, because you need to have a way to tell the model if the model is, you know, in comparison to expectation, is it doing a good job or not? So that's where kind of before we do anything, I think one of the common practices that is spreading to the industry is to have developers write evals. So that's a starting point. Without that, a lot of things they do is like shooting in the dark. A lot of things they do, they will start vibe checking, but vibe checking is not reliable. It's not consistent. It's not repeatable. It's not systematic. So I think that before anything, the future, I'm very bullish about that future we talk about. To close the loop automatically. But the first thing is to enable everyone to write clear evaluation criteria. So, I think that's a transition the industry is moving towards. Many app developers, they are very familiar with product analytics stack. Hey, you know, I can start to trace this and and build data pipeline to generate product metrics. And I know how to do A/B testing, and then I know how to look at the result and make decisions and do a lot of product-level experimentation. But how do I write evaluation for this model is a new skill I need to learn. I haven't worked on that before. But we see people are kind of picking that best practice faster than ever. So, this is a kind of exciting moment, but also in the same time, it's a starting point of like do this automatic closed loop thing.

So earlier you weren't necessarily saying that the metrics or reward function for this closed loop thing was going to be unnecessarily harder. What you were starting to say was that we need to be evaluating in order to even hope to get there. And then once we have as a community, as a practice, like a discipline of evaluation for models, then starting to insert this kind of RFT or whatever the technology is into kind of a live optimization loop, then it becomes not just easier, but possible from the first place.

Yep.

Yeah. Interesting. Interesting. We started to talk about kind of interfaces and abstractions and you kind of dug into inference and talked about it being OpenAI compatible. On the post-training side of things, it's a lot more open there in terms of kind of the field. I don't know that there's quite an accepted standard. You know, when you think about kind of designing that experience, what are you emphasizing? What's important from a developer or user perspective? And how have you seen, you know, what have been the required elements for folks that are having the most success with tuning?

We, we are aiming at a broad set of developers, but in the adoption curve of a new technology, It almost always starts with power users. People are self-learners. They can go really deep and pick it up and adopt it, right? So for the power user, the pattern is they want to control everything. They want to tweak a lot of knobs without being blocked out of optionalities. And if you give them a higher level abstraction, they'll be so frustrated. And they will not like it. Instead, they want to kind of go as low as possible. So, the system design is just kind of build building blocks, expose knobs for them to tune. But then when you want to reach a broader set of audience where lots of people, they don't want to tune low-level knobs, they want to focus on developing product. And just have simple integration and things should just work. And by things should just work, it means out of the box it has pre-configured setup that can bring you to 80%. May not be 100% or like 99%, but 80% to begin with is good enough. So that's a high-level simpler abstraction that for broader set of audience. We have tested this hypothesis many times, and this setup is the kind of best to cover both ends. And the good news is the higher-level abstraction can build on top of the lower-level abstraction. It's kind of staggered system. And in that sense, it's not wasted effort to build lower level. Yeah, so we believe in this design, and we also think what's happening during training space is also following that pattern.

What are some of the knobs and levers that folks operating at the low level are wanting to configure?

So for example, in the reinforcement fine-tuning, there are many parameters, just kind of the tuning parameters, but also for some For the initial setup, so for example, we have a web app, you can write your evaluators, the code of reward function, and then you can test with your initial small dataset. That's kind of easy. That's simple. And then once you are confident with the direction of the evaluator or reward function, and then you start to upload a real dataset, and then you start to kick off a training job. And then the training job has a lot of parameters that you can set, but we have the default ones. You don't have to worry, just kick off a training job and go with our default ones. And then we also have default parameters, metrics you can look into, but you can add your own metrics. Oftentimes, like, next-level researchers, they want to add a lot more details because the metrics can be overwhelming because if it's not actionable, if you look at those charts and you don't know like, what do I do? Like, it's going up, is that like, or going down? Even if I understand it, but I don't know what's the action to change it, then it's overwhelming, it's useless. So, that's what I mean is the first mile for a broader set of audience And, you know, we designed that that way, but we have like people can click open the parameters, they can set their own thing if they want. So, so that's kind of hopefully that will bring various different, a wide range of experience to, to a broader set of audience that way.

I'm imagining that ultimately, I guess this is going back to that kind of closed loop discussion we had. Ultimately, I want to just kind of hook the system up to my eval parameters, my data flows, and say, go optimize this, do a new train every week, or whatever, every day, every hour, and just keep going until the performance is better. Do you have a sense? So as I say that, I'm wondering, is that interface and user experience, like, is that even well-defined? Is it, you know, once everyone has their evals in place and gets comfortable with fine-tuning, you know, are most of those problems solved? Or like, what is left to solve, uh, for us to get to like this, you know, self-tuning, uh, type of ideal that we've discussed?

Yeah, I think for the self-tuning, I think automate that to 70%, 80% will happen pretty quickly, but optimize that to 99% would take a much longer time. So I think that's kind of, it makes sense. It's a natural progression, but we believe in this direction is If we take a big step back, in this new wave of GenAI, the product development philosophy is also changing, as in many people we've seen, they've been building great product, have huge amount of adoption over small amount of time. Is they do not think model as a commodity or as just utility you plug in as is. They think about model as a critical part, critical assets of their product. And they are actively bridging a very interesting gap. The interesting gap is the data they collect from application cannot be seen or accessible to the frontier labs developing models. And you can think there are two parallel threads of data distribution that's happening. One is the frontier labs, whether it's closed or open source model providers, they have a dedicated research and data team to curate data based on some assumptions of how application is going to use this model for. So they, they query data based on some data distribution, and that's where kind of the end result, the model quality is going to focus on. In a different company, the application product developers, they are building new user experiences and that produce another set of data distribution, right? By design, these two are completely different from each other. By design, they diverge from each other, right? There's no— because hey, they solve different problems and there's different entities, it's different. So the mismatch between these two data distribution is going to result in worse quality, worse latency, Worst cost, right? You leave a lot on the table. And we firmly believe there's a huge void in that space to standardize, to bring more tools in the platform to close the loop, to close loop because we believe like a large portion of data actually is not on internet. Large portion of data lives with the application itself. And if you do not leverage that, and then there's just a lot of, you know, data is the moat and data flying well is the moat, and you just leave that on the table. And whoever can leverage that quickly and in the best way, they are going to emerge as winner faster. So, so that's part we want to standardize and productionize soon.

We said that we would come back to this early on in the conversation. This seems like as good a place as any, but also relevant to your last comment and thinking about the model as a commodity versus an asset. And that is open models versus closed models and all of the activity that's been happening in the model domain. You know, talk a little bit about, you know, first commodity versus asset and open versus closed. Like there's not necessarily a one-to-one relationship between these. There's a complex relationship between these potentially. You know, how do you think about, you know, if your model is going to be an asset, do you want it to be open or do you want it to be closed or based on an open model or a closed model?

Yeah, I do. I, you know, if we take a big step back, the model quality is essentially data, right? It's data quality. That's pretty much clear because I don't believe any lab has perpetual secret sauce in terms of, hey, for the same amount of data, they just can make the model better in a meaningful way and no one else knows how to make, do it. As you can see, people move around a lot and they bring that knowledge with them and they spread the knowledge across the industry. So then essentially it's about data. And the public data on internet has been exhausted, right? And then there are many data labeling companies that are pretty much providing services to all the labs. And then in that sense, there's not much differentiation.

Until they get acquired by Meta and then they lose their contracts with the other labs.

Yeah, but the— there are 10 different data labeling companies, right? So no one has the unique advantage, right? So it's because this is such a high potential space and a lot of investment come in, and that means a lot of companies gonna work on similar space to bring supporting services to the broader audience. And then if you think about that is if all the labs essentially has access to same amount of data, then the only diff— difference is in the app space, they cannot have access to those application-specific data, right? The app space of data is gonna be verticalized because that is That is asset, that is unique competitive edge and no one will share outside. Then the next level question is how do you leverage that and turn that into your own model advantage, right? In that sense, open model is much easier to tune because there are a lot of tools building around open models in the community. That moves really fast. I have, there are many stories and that's why I believe in the open source community is there are many stories even for the originator of some open model provider and open source project provider at some point, they flip to close up, they build their own software, they close it up for internal use. And there are cases again and again showing the open source project move much faster than closed source project. I've seen that at Meta. So Meta is a really open source friendly company, especially on the infrastructure level. But there are cases, the open source project, there's too much maintenance. To keep the community up and they decide, hey, internal demand is huge on that project, let's just close it and focus on internal need. And it turns out the open source side move much faster because the sheer amount of resource is much bigger no matter what. So that's where kind of I believe and we're betting on as a company in that direction to to kind of bet on customization based on open models because the whole entire community is working around that. And we want to provide the easy access of the SaaS platform for people to deploy without considering or solving any of the infrastructure problems. So that's our deepest value proposition for— and now we're kind of not just inference provider, but also deeply in the customization tuning side.

What are you most excited about in terms of, you know, recent models? Like it's, you know, on the one hand earlier expressed excitement about all this cool stuff coming out, but then you also articulate a bit of, you know, the underlying model is still a commodity. It's when you, you know, customize it with your own proprietary data, then it becomes this asset. Kind of rationalize those two positions. Am I correct that you kind of feel both ways about models?

I feel excited about open model because we know that there's a war on the model capability, or there's a war on data on the public, you know, from like lab accessible data, right? There's a wall, right? And that just means, you know, the model more or less is going to converge on some dimensions. And that just means the open model is going to be catching up with closed model closer and closer, which is happening today. And all the model providers, when they launch when they announce their model, they will show benchmarks against the best, right? It doesn't matter if it's closed model or open model, they all compare themselves against the best across industry. And they will say, hey, we beat the best in those metrics. That makes me excited because now they can say it. Before, the open model only compared with open models and compare with each other. Now they are like declare victory against the closed models and it happens more and more often these days. So that's a sign that the models are all converging. Under the assumption models are all converging, then the customization becomes, on open model becomes much more interesting because you do not need to customize to catch up with closed model. You can customize to beat the closed models. So that's kind of a very interesting shift of the dynamics happening across industry right now. And that's kind of— our belief in that direction never changes. And we have been betting on the open model war to converge with closed model very, very closely.

And which recently announced models are you most excited about? I'm assuming the KIMI-K2 and the Quen, the new Quen coder, and are those what you had in mind when you lit up earlier?

Yeah. Yeah. We are super busy, super swamped these days. Yeah. I think summer will be busy. There will be more open models coming up from various different labs.

And is that the idea? And putting this in terms of your business, that as each of these new models comes out and demonstrates its capability on benchmarks, then you have to then do some work to support them on your platform and make them available to your users?

Yeah, we always aim at supporting those models from day zero. Whenever they launch, it's available on our platform so people can evaluate those models quickly. And then we will roll out the surrounding functionality. For example, a lot of those models are really good at two calls, and we need to enable two calls to make it, you know, work well with our service APIs, with on-demand, some kind of reserved throughput setup and all this. And the context window length has become bigger. And when to enable that properly. And obviously more longer context window, it just means more pressure on processing those tokens, generating those tokens and so on. So yeah, so a lot of next level optimization. Some models come with new model components that we need to next level optimizing for. And yeah, so I think many labs give us great surprises as they are very innovative. They are, when they train models, they are putting inference into consideration, how to accelerate inference during training time, the different model components, they run just much more efficiently. Economically. Yeah, so I think this community is very active right now and really looking forward to more state-of-the-art model launches from Open Model World.

Can you talk briefly about when you're presented with a new model, like what the engineering workflow is like to to support that new model?

So our inference engine has a lot of building blocks. So first of all, we need to kind of pick the right building blocks to assemble for a particular model. And then as I mentioned, this 3D optimization have many options, and then we need to go through this search space and find an option we like and run a lot of quality testing to make sure, you know, the quality is correct. There are, when we assemble those pieces, there are not like numeric arrows that lead into accuracy issue. And so more like focus on quality correctness at the beginning. And then there are other functionalities. There is constraint generation correctness. There is, there are many add-on capability on top of the raw inference. There's tool calls. There is LoRA adapter. There are many other things we also need to make sure it works well with that particular model. And then we launch it and then we start to do continuous optimization. So that's kind of the typical workflow.

Order of magnitude, how long does it typically take to support a new model?

Couple hours, like, yeah.

And, you know, besides the things that we've talked about, new models, you know, what else are you excited about kind of looking forward? For, you know, GenAI?

I do think making customization accessible to all developers is the most exciting trend. And like we are fully committed to make that happen. But I think it's not just us. The bigger community is actively thinking about that. There are many, many players covering different sets. Different part of this journey for customization is actively building tools. I do think that's the next phase of heavy development and adoption is going to happen there. People will not just be satisfied with using off-the-shelf APIs. They want to actually fully leverage the production data curated and in a closed loop, closed loop as before. and this is a different loop, but the mentality or mindset is similar to before, of, you know, closing the product analytics loop. And this is a kind of new loop that they're going to close. And therefore they have, they have their, use their own data to have their own model and build their own, you know, leading position in a heavily competitive space or application. Development. So yeah, so we would love to see those app developers be— continue to be very successful in leveraging their own data.

Awesome. Well then, thanks so much for taking the time to share a bit about what you've been working on and how you've been approaching inference and tuning for GenAI. Very cool stuff.

Great conversation.

Thank you so much.

Thank you.