Fireside chat on everything inference

Hi, I'm Tuan. I'm the CEO of Base10. Today I'm here with—

My name is Amir Haghighat, CTO of Base10.

Hi, I'm Bolath. I work on product here.

Yeah. So we've been building inference products for the better part of 5.5 years now. Is that right?

Almost 6.

Almost 6 years, which is wild. Obviously the market's evolved a lot in those 6 years. I mean, do you want to just to start off with, you have a little spiel where you talk about where we come from and why we're doing what we're doing.

History lessons. Yeah. So yeah, the company is almost 6 years old. The product has always been in the ML inference space, and within that, it was always focused on inference, as in we stay agnostic of training the models. We cared more about what comes next. The thing that changed were the types of models that customers were deploying on this. And so early on, it was classic machine learning, predictive models, regressors, classifiers. And the use cases for those were mostly back office use cases, fraud operations, trust and safety, content moderation. But about 3 years ago, that started to change. We started seeing a lot more customers deploy deep learning models, BERT-based models. And that was exciting for 2 reasons. One is that these models were suddenly being used in the path of the end user of our customer. Suddenly the customer cared a lot more about high throughput, low latency, SLAs, uptime, things that we were building for, but things that the back office people didn't care as much about. The second reason it was exciting is that we saw a pattern that these models, these new architectures, the models were becoming useful in ways that you couldn't predict. And oftentimes their weights were open too. And so we said, look, if this continues down the line, we're going to end up with models that are both very useful and open in different modalities. And so we decided to build for that future. And that meant building a lot more depth of product in really two pillars. One is around optimizations at the model level and the different techniques there to get the most out of the hardware. And then the second time around horizontal scale infrastructure and cross-cloud and build a lot more depth there. And that was good timing because by the time 2.5 years ago came, 2 years ago came, and a lot of companies were using these generative models in production. We had a good product for them and started signing them up.

Yeah. And so, yeah, and we've always focused, we've always focused on this problem from what, like 3 pillars you'd say, right? You said 2, but there's a 3rd one, right?

That's right. So the pillars, performance, performance at the model level, infrastructure. But really these things have to be wrapped around in a very good developer experience that presents the power of these 2 pillars to the customer so that they can optimize and get the most out of the models for their own use cases because the use cases are very different based on modality, based on whether they're latency sensitive or throughput sensitive. Yeah.

And so I'd say the ChatGPT moment, I think actually what was more interesting in a lot of ways was the Stable Diffusion moment because all of a sudden you had these open source models, which could be considered state of the art.

And Whisper too.

And Whisper. I think there's always been this debate around open source models and closed source models and custom models and whatnot. I think we're probably a little less dogmatic than others. We kind of think that the future of AI is both open and closed. And what we're really focused on is that, look, there's going to be a ton more models. We're probably in early first innings. We're pregame in terms of where we think we are in terms with AI adoption. And these models are going to have to run somewhere. And that's, you know, in a lot of ways, it's like, you know, we've been very, very focused on inference, which, you know, I guess I was going to ask you actually, can you maybe, Boal, this one's for you. Can you define what inference is?

It's not for tourists, I can tell you that.

Inference is not for tourists. Inference is essentially like how you take a model that was trained and the way the model is trained is just to repeat everything and get it to do something interesting, right? You get it to do like, it's not like training, you just repeat, like your goal is to mimic, but inference, you could take something that was trained to mimic and teach it to generate. You could teach it to generate different types of things and all of that is an inference.

Yeah, so it's the actual generation of one of the large models. And when we talk about inference, you know, we talk honestly about all the things it takes to actually generate those things in for, for users, for end users. Um, and so one of the, one of the north stars of our company, um, has been, hey, we're going to build whatever it takes that's going to lead to more inference. Um, and I think one of the challenges from the outside in, or one of the perceptions from the outside in is that, oh, this is actually a really simple problem. You take a GPU, you map a runtime onto it, voilà, your model's running.

But are you saying that VLLM plus H100 equals inference?

Yeah, basically. But when we go and talk to our customers and when we're running our workloads in production, there's so much more than that goes into it. Amir, do you want to just talk very quickly about the two different things there that we've talked about and just go a bit deeper. So there's performance and there's infrastructure.

Infrastructure side.

Yeah.

But to go back to that equation, and this is interesting because I see this in the market where people think that they have some GPUs and put their models into VLM and that is production-grade inference. And we know that through experience to not be true. I think with those you can get 80% of what you want the model to do and 95% reliability. But for mission-critical inference, that's not enough. And you need to go much beyond that and get closer to that 100% number. So what does it take? It is the—

What is the inference stack?

Exactly. And so honestly, this is why we built the Base10 inference stack is to get you to 99.99% reliability, is to get you to have the power to get your models to do exactly what you want and be optimized for your use cases. So what does it take to do that? So there's really two components to it. There's the inference runtime and then the inference optimized infrastructure that that runtime runs within. So let's talk about the inference runtime. There is a lot more that you have to do than what you can get out of the box using open source. So you need to have a lot of different runtime optimizations underneath that. You need to have a lot of decoding strategies that work really for your use cases, whether they're latency sensitive or extremely high throughput. And then a lot of kernel optimizations, that also needs to happen in order for you to really get that sort of SLAs that you want. At the end of that, you have a model that is running on that one GPU very efficiently, and that's great, but that's not enough because that one model running on that one GPU very efficiently is going to get too much traffic that it cannot handle. And then at that point, you need to horizontally scale it. And that's not an ML problem or a PyTorch problem, that's an infrastructure problem. So that's where the second pillar of the Base to Inference stack comes in, which is the inference optimized infrastructure. So what does that do? So really very high level, 3 things. You need to horizontally scale up and do so very efficiently and very fast. There's a lot of cold start work that needs to go into that to ensure that when you get a burst of traffic, you can quickly scale up. That's great. Then you have routing strategies that you need to apply, and you need to ensure that that's being done efficiently. So for example, for a lot of code generation, just doing a round robin routing is not going to really do you well. You need to do KV cache-aware routing, geo-aware routing, et cetera. And the third thing underneath all that is to get capacity where you can, to go cross-cloud, cross-region and be able to gather capacity from anywhere that you have access to GPUs and not be confined to just a one cloud or one region kind of strategy. Combining all of these together, then you can get the models to be optimized specifically for your use cases and be able to get to 4 nines availability.

Yeah, I think just to put it in some customer context, We have had a customer where they've been optimizing for latency. They've been optimizing for latency. And you realize that this is like the inference stack goes just a lot further than just like, hey, can I just run it in a highly efficient runtime? So I'll give an example here. So this customer, they fine-tuned a model, they needed to deploy it somewhere. And they had multiple versions of this model because they were constantly updating it. So they needed a workflow to be able to deploy that model. They needed to observe how well it was performing. So they needed an observability stack that's on top of it. They needed to use prompt lookup, which is a speculative decoding method to get the most performance for their use case. And then they needed to geolocate all their hardware close to their users so that they could minimize the hops.

Totally. For certain use cases where tens of milliseconds matter. So if you're typing, if an end user is typing and you want to suggest something in a code editor, tens of milliseconds matter. If you are doing an AI phone call, tens of milliseconds matter. And so that's where all these different tactics and strategies come into play.

And then the Infrastacks really becomes the kind of the culmination of those three things, right? So it's like at the lowest level, the bedrock's infrastructure, then on top of that you build the runtime and there's a lot of work that goes into runtime to allow all the different prefill and generation strategies to get things as fast as possible. And that's when you can use stuff out of the box like VLM or SG Lang or TRTLM that you can run on Base10, or you might have to do something a bit more custom. So for example, for embeddings, you might want to need to run a hyper-optimized embeddings runtime for which we've built PEI. But then on top of all that, you need the actual developer experience. So you can give that control to do all those things back to the user. And that's what we've built, which is the inference stack. And that has become the bedrock of what Base10 is today and why it supports companies like Descript and Writer and Abridge and Bland to power things in production. One, we've dealt with this, we've kind of delivered this stack very much in the context of I'm a user, I have models, I want dedicated hardware and I want dedicated deployments.

Generally for their own custom models that they've pre-trained or fine-tuned.

Yeah, I guess one thing I wanna say is like, I'm glad you brought up like AI phone calling and I'm glad you brought up embeddings 'cause like the baseline for stacks is not limited to LLMs, like it's not limited to tokens. So like for voice, for example, the routing is different because you want consistency across different generations of the same voice. We do the same for images where it's relevant, for transcription where it's relevant depending on how the model's implemented. So I just wanted to be very clear that like basically InfraStack allows these like compound AI use cases because it lets you become optimized for each part of that system.

Yeah, and a lot of them require WebSockets and just the REST call is not enough, which they also support it.

Yeah, there's a ton that's going on there. And one thing we've heard from customers time and time again is, hey, I'm just getting started. I've been using OpenAI, I've been using Anthropic, I've been using the model providers. How do I transition from that to my own dedicated stuff with the right cost performance? Trade-off? And what not when I don't have scale just yet, or I'm still experimenting with scale. That's where the model APIs that we're launching come in today. We're really excited about this. It's been a bit of a sprint for the company to get these out over the last few months. Bola, do you want to just talk a bit about the motivations for the product and what we're launching today?

Yeah, yeah. I think you guys talked about like we've been around for like almost 6 years, like like, why now? Like, why are we doing this now? And I think something very exciting happened in January, and you probably heard a lot about Base10 around that time, which is DeepSeq dropped. And when DeepSeq dropped, like, not only did it move markets, but like, there's something that was very obvious to us because we've seen a few of these shifts, like in our lifetime, like AI shifts, like just within the lifetime of the company, which is like, hey, there is like a very clear open source competitor. To quality that you could only access through closed source. And that became very clear to us. And there's a bit more to say about how it was trained that we'll get into later. And that was really like, why now? Is that we could finally put up a model API that gives you a clear alternative and a good place to ramp up and try new things where you're not sure if they will work, where you need the generality that you can only get through kind of closed source before. And we want to make sure that everybody can access that and everybody can get a world-class experience trying to build their products on it.

Yeah. And so, and what do you think those things are? Like as you're using these model APIs, what are the important things for customers and what?

Yeah, like you're trying to build a product, right? So you want to be reliable. You actually kind of want it to be predictable. Like you want intelligence, but you don't want like unregulated intelligence, right? But so a lot of that is like, you need structured outputs, right? Like you need to be able to show that the next system that's gonna consume this response, like has a very clear directive what to do. So there's structured calling, there's tool calling, there's just knowing it's not gonna go down. Like it's like nobody wants to be in the middle of a demo and then just be like, oh, sorry, like let me just look at this page. You know, it's like just—

You go to the status page and there's a Christmas tree of, you know, green with lots of red around it.

Yeah, and like you just don't want that to be, your status page, right? It's like you just want to make sure that you're insulated from that. And honestly, just performance. It needs to be fast. Everybody knows GPUs were a critical part of the AI journey because it was the only way you could do things fast enough for it to be relevant, right? And then that just keeps going through any experience that you want to build on top of AI. And that's a key thing for us and why we spend so much on that Base-10 inference stack and making sure that that's really what powers like this experience.

Yeah, and we're launching our model APIs with a bunch of cool partners. Do you want to talk a bit about that, Paul?

Yeah, so actually Retool is bringing open source models to their platform for the first time on top of Base10. Again, like I mentioned a lot about what DeepSeq enables and some of these newer models being able to compete in the open source stack. And we're excited to continue working with partners to bring every modality, every model that you might need to build the kinds of products that we know our customers are building. We're hoping all of those will be accessible through the API.

For sure. And then maybe, maybe, so we don't bury the lead, it's available today, performant, competitively priced, and we'll be constantly updating the models available there to keep pace with market, ideally even be ahead of market.

Awesome.

So that's the first part of what we're launching. The second part that has become more and more important to us as we have gone through, built this product and heard from customers is one thing I said earlier was we will build anything that will eventually lead to more inference. And we've historically stayed away from training over the last 6 months. We've got lots of inbound from customers, which is very much that, hey, you've built this amazing inference experience. I also have workflow for training. Is that something you can support? Hey, I care about fine-tuning. Can you help me there? Yeah, Paul, are you keen to do the honors and tell us a bit about training and why we're building this and what we're releasing today and who we're partnering with?

Yeah, so over the last 3 years, right? The use cases that are really at scale, that really need Base10, need something very specific. So Amir talked about some latency things that people often need. There's reasons why you need to have a dedicated deployment and we love doing that. One of those reasons is that an off-the-shelf model cannot solve your use case. And some of the most successful use cases on Base10 today are fine-tuned models. And over the last 3 years, our customers are asking us, hey, you guys solve infrastructure for my inference. Can you solve, can you just, we had a lot of pain getting to this point. So we have this fine-tuned model that we could then not have pain in scaling. Can you just help us? Just help us solve.

Yeah, they would SSH into the box and run some fine-tuning scripts.

Yeah, or they need commits for a year to try out something new, right? Because the compute you need is actually quite large, right? And again, it's been 3 years. So the question is why now? Why are we doing this now? And going back to January, one of the things that was very interesting about DeepSeq was how it was trained. And this new technique called GRPO.

What's that, Will?

It's quiz time. GRPO stands for Group Relative Policy Optimization. Well, we can break it all down, but I'm just going to TL;DR, like, like, let's do it. What, what GRPO is, but I think one of the key challenges with fine-tuning that, that people have faced is like, what is the amount of data that you put into it? Like, how does it need to be structured? What does it need to have? Like, there are a lot of vibes, like, on, on, like, like, what data leads to good models? And GRPO just said, hey, like, let's, let's like, let's let the model deal with the vibes, right? So like, so you tell me what you want your input to be. You tell me what you want your output to be. And the model is going to generate anything in between to get you from that input to that output. And all you need to do is tell it, here are the right signals that you need to hit that we're looking for, for you to be successful, right? You could tell it some things are good, some things are bad. So these are the policies or the reward functions that you're optimizing. And the way you get there is by generating a bunch of candidates in a group. The model will generate a few candidates in a group and then you will score them relative to each other. That's the relative part with your policy, which is these reward functions. And then you optimize.

That's all the words. That's all the words. We kept hearing about this from customers. So what are we building? What have we built? What are we releasing? Who have we partnered with?

Yeah, so we've built the infrastructure solution that solves all of the pains that you might have. So one thing that our customers have said is, hey, we actually need multi-node training because these models are getting bigger. And even though we could do adapters and we could limit how much memory we need, we still like, we have a lot of data or like we might need to deal with a lot of data. Like we need multi-node. So we have multi-node and we have it on demand. Like as in like you could kick off different jobs that need different amounts of multi-node on Base10 that again, like don't have to run for a year. Like you don't have to pay for them ahead of time to get that year, right? Like you figure out like what is the experiment that you need to run? Like what is the code that you need to run that experiment? And we'll solve everything downstream of that. So that includes the infrastructure, the job orchestration, it includes the caching of the models and the weights in clusters. So repeated experiments don't take as long as starting from scratch. We take care of automatically managing checkpoints and giving you inference from those checkpoints. So I'll kind of just take a tangent here just because this part's important.

This is actually cool.

Everybody has seen like, oh, my loss curves are cool, my evals are bad. You do a few iterations, you're like, oh, my training loss is good and now my eval is good, right? Like my accuracy is good, my eval loss is good, like everything, right? And then you take this checkpoint and you deploy it and then you're like, oh my God, my product broke. Like what actually happened? And I think like this is a common experience. And part of what is painful there is taking that checkpoint and going through all the pains of getting it into inference and then going through all the pains of making sure that it's integrated to your product and like do it there. So what we have is like you could take a checkpoint and with one click get an endpoint that is ready for inference that you could A/B test in your staging, in your pod, in your dev and like get real-world feedback as to like, is what I'm trying to do with my fine-tuning yielding like kind of, what do they call it? The outcomes that I'm expecting, or is it moving the bottom line? Sure. Yeah. That's what every, like, it's like, and for you, like as a product person, that's like engagement on the product. It's A/B testing. It's everything, right?

And monitor the performance characteristics around latency and time to first token.

So we build a training API that integrates very nicely within our inference, with the BayesNet inference stack. It's available today. Can you talk a bit about, I think we've partnered with, is it Oxen? It's Oxen and MixBread.

Yeah, so MixBread, again, we've built this product for post-training, but MixBread is a foundation model company that builds embedding models. And we just wanted their feedback, but then they're like, we want to use this to train our models. So we've partnered with them to make sure that that happens and they have the best possible experience and can get to that value. Faster. And Oxen, as I mentioned, specializes in working with the data and how you version your datasets and everything. And we've partnered with them to make sure that you could get access to these best-in-class compute with these best-in-class techniques without having to start from scratch.

Amazing. Yeah, we're very excited. So you can check out both the model APIs and the training product today. We're really excited to hear what you do with it. It integrates nicely with the inference stack. You can reach out to me directly at tuan@baseten.co or Amir Haghighat@baseten.co and Bola at baseten.co if you have any feedback. Yeah, really excited to see what you do with it.