Self-hosting & scaling models

Foreign.

Welcome to Practical AI. If you work in artificial intelligence, aspire to, or are curious how AI related technologies are changing the world, this is the show for you. Thank you to our partners at Fastly for shipping all of our pods super fast to wherever you listen. Check them out@fastly.com and to our friends at Fly. Deploy your app servers and database close to your users. No ops required. Learn more@flyeio.

Welcome to another episode of Practical AI. This is Daniel Whitenack. I am the founder and CEO at Prediction Guard and I'm joined, as always by my co host, Chris Benson, who is a tech strategist at Lockheed Martin. How are you doing, Chris?

Doing very well today, Daniel. How's it going?

Oh, it's going great. I spent the afternoon in sort of a brainstorming session with a couple of our team members here at Prediction Guard and it was a ton of fun. So talking about a lot of prompt engineering things and how different models perform and that sort of thing. So it was a good time.

I'm glad you're doing that because you know what? I just want things that just work, you know, I don't have to think about it. I'm glad you're thinking about it. I think we might have someone else to talk to who knows how to make things that just work.

Yeah, well, a lot of the models that we're running sort of just work for us in terms of inference because we're hosting some of our models in base 10. And we've got Tuan joining us from base 10 today. How you doing, Tuan?

Hi, Dan. Hi, Chris. Nice to see you guys again. Thanks for the kind words, Dan.

Yeah, yeah, for sure. Well, it's exciting to have you actually back on the show because it was, I believe I looked it up, it was like May, June 2021, when we recorded and released the last episode with you. So how you doing? What's new and how has base 10? How's the ride been?

Yeah, it's been crazy. I feel like the last. I think it was like May 2021.

That's like a millennium in AI time, you know.

Oh, my God. If it actually does feel like it was a different job ago. Yes, it feels like the job before the. If that makes sense. No, I think being crazy, I think the last two years for everyone here have probably been a bit of a whirlwind and you guys are pretty on top of current things in machine learning and AI. And I think, I imagine just like for you guys, it's hard to keep up at times with what's going on.

I think, you know, we only do one show a week and you know, it's getting. We almost need a daily show. There's so much content now, it's not enough.

Yeah, don't give our listeners ideas because I don't know if, I don't know if I can do a daily show. But, but it is a lot and I think so. I'm looking back previous at our episode and last time we talked about sort of the easiest way to create ML apps, that was kind of part of how the conversation was framed. And I know just from working with you and talking with you as friends that a lot has changed and you've seen some things within how people are deploying machine learning AI systems that now base 10 is really focused on. Could you give us kind of like the high level view of base 10 and the type of problem, the type of solution that you're offering?

Yeah, yeah. I think it's just worth probably pointing out before I go into base 10, specific things. One of the key things that changed since we last talked, I think.

Sure.

If you think of the year of 2012 to 2020, you know, data scientists were the ones doing a lot of machine learning. I think, you know, that's changed for a number of reasons. I have a lot of thoughts on that. But probably the bigger changes are, you know, the, the emergence of good open source models. Dan, I know you do a lot of work with that and you know, like we've seen Hugging Face as a community evolve into this like really, really vibrant place where if you want to get a sense of how fast things are moving, it doesn't take long to, you know, take screenshots of Hugging Face every Monday morning and see how the trending is changing and you'll see that things are pretty different every week.

I don't know if it was Daniel that said this or another friend of mine, but the analogy was that Hugging Face has become to AI kind of what GitHub has always been for software developers over the last decade or so. It's just, it's the place to go to find it. Anyway, I didn't mean to interrupt you there.

Yeah, and the good about it but also the confusing too. It's like you have random person clones model or copies model and uploads random version of that model that like maybe works.

Yeah, 100%. I feel like the game you have to play with Hugging Face is like. But does it run? You know, does the model run? But does the model run but you know, I think open source emerged and I think like stuff like whisper showing up and you know, some of these OCR type replacement models showing up, you know, they're probably the more interesting ones to me, not because what they do, but because they end up just solving a lot of open problems, you know. You know, if you think about transcription as a problem, think about nuance and like how long they were working for that. Yeah, like literally 20 or 30 years of, of work just kind of. All right, that's solved problem, now let's move on. Oh, we've actually solved multi language with the same model too.

Business models come and go, don't they?

Yeah, yeah, it's wild. And I think like the last piece is just around you know, like the ChatGPT moment for AI. Interesting for a number of reasons. I personally think it's someone who built infrastructure that it's most interesting because if you want to call that the iPhone moment of AI, I think it's a bit different to that because it's so early in the journey. It's like if the iPhone showed up when we were all using 5110 from Nokia, the world would be very, very different I think because consumers and developers, their first taste of machine learning and AI was through ChatGPT and GPT APIs. The stakes are just, you know, it's just harder to build something good. Like, you know, people don't want to use a model that takes 12 seconds to run. You know, like high speed production inference is, you know, taken for granted when you're using this model. And then when you kind of combine that with okay, open source models need to be run somewhere. We personally think that's like, okay, well there's a massive infrastructure opportunity there, or maybe not even opportunity that just a fact that a whole new stack will be built to support models to be able to power these end user experiences. And I think like that's kind of the core insight kind of going into base 10 and talking a bit about base 10 around like what's changed is that, you know, we, we kind of two years ago when we were talking about data scientists and we weren't talking about engineers. I think that's pretty key to our story, which is that I think we came to the realization that every engineer needs to grapple with machine learning now as opposed to maybe a small market, lower data scientists. I think going from smaller models that run in memory to larger models is another big focus change we've had. I think there's a bunch of language Stuff and NLTK stuff. And you were doing a lot of that work, Daniel. But for the most part, everyone was using Scikit Learn and Scikit Learn models, for the most part, run memory on CPUs. On CPUs.

Yeah. If you think just in that time period, the amount of maturation that's occurred in this industry, it still bought. You said something a second ago, which kind of hit me, and that is most people out there in the general public, you know, are really just getting into this, you know, with chat, GPT and stuff, and we've come a long road already. But I just, I'm listening to you and it's amazing how far we've come in such a short time.

Yeah, it's insane 100%. And I think that, you know, going from small models to large models as well, it's just like that change kind of happened pretty quickly. You know, I saw one who did a bunch of work with small models. Like they have their time and place, but they just aren't that fun anymore. Like something that runs in memory just doesn't give you the same feeling. And then I think the last one is just so much of the stuff that was happening with machine learning outside of faang, I'd say was like FAANG had some production use cases around like ad serving and search and whatnot, but outside of that it was mostly just internal workflows. You'd go and work on fraud and content moderation and recommendation systems. And I think, you know, going from, hey, every product is going to have some sort of machine learning in it, and every existing product will definitely. And 90% of new products will be built with the, with a new pillar that is machine learning and AI, which wasn't the case, I think, two years ago, which is just crazy to think about.

Yeah, it's completely changed in that time.

I think one thing that was brought into focus for me while you were talking was that as soon as you make this leap to kind of larger models and you make the leap from some closed API that's very fast to maybe running your own model, there's two things that become immediately clear. One is the infrastructure challenge around that, which I think is the workflow around that, and the model hosting Base 10, of course, is an expert in that. And the other side of that is the product concerns around running these things, which I feel like we always have great conversations too. And because you're on the one side of that and I'm probably on the other side because, yeah, when you're using ChatGPT, or even the OpenAI API. They have layers of protections on the prompts and on the output. They have filters to make sure they're not responding in certain ways. And there's all these product concerns that people don't think about. And then they take a llama 2 model or something, they run it and then there's like, oh, this doesn't respond. Like, this is not a product. Right. And so like the infrastructure is a piece of that. The ability to iterate very quickly with models of a variety of types, I think is part of that infrastructure challenge. How do you see that infrastructure piece of this kind of playing out?

Yeah, maybe just before I say go into that. When you were saying that reminded me of like I bought a drone in 2014 and I was so excited. DJI Phantom 3, I was so pumped and you know, I flew it around a bit and they had basically had autopilot mode where you didn't have to do anything. Like it kind of stabilized itself. And then there was this button that said manual mode and it had like all sorts of warnings. And I remember saying like being like, oh, how hard can it be?

Right?

Oh, I really like, jokingly, like I was in a safe space, but I put it into manual mode while I was up in the air and it just fell to the ground very, very fast. I don't know. Chris, do you have some experience with this?

Well, I'm in aerospace professionally, so I know a little bit about that. And yes.

Chris, weren't you like the TV host of the drone racing competition or something like that?

Okay, there you go. There you go.

Yeah, a few years ago, yeah, I was one of the hosts of the first drone racing league. They had a championship series and they were using instead of you as a human. They were navigating obstacle courses and stuff like that. And what I learned through that experience is when you have how much autonomy is required to make even small things fly. Well, yeah, and yeah, so I, I sympathize with you for going on manual there. Oh, no, don't do it, don't do it.

And the analogy I think holds here. It's like the close, the close API, it seems so great. And then you see something like a Llama 2 or a Mistral and you're like, okay, I'll just rip and replace this. And it's like, nope, that's not going to work for a number of reasons. And I think that's kind of how we see about it. Think about infrastructure based sandwich is that running models in production is very Very difficult. It's difficult for a number of reasons. I could get started but like from like a user requirements perspective, like latency and throughput are paramount costs of what something you want to optimize. Data privacy is a whole nother beast. Security comes into play that orchestrating this across a bunch of different hardware, orchestrating this across clouds becomes a problem. Benchmarking these things isn't easy. And that's even before you go into all the evals and the, you know, the kind of like the guardrails you want to put around the scene to get it running. And I know that's on the stuff that you think a lot about, Dan.

So just as an extension of what you're saying and Daniel, you mentioned it at first, but if you could also talk a little bit about what the difference is about just host like. Like just having the model hosted and kind of the idea around what you have to put around it as to make it a product. Because I don't think most people talk about, I don't hear a lot of conversations about that. And it's a big set of gotchas on what to do, you know, and kind of what's involved in that. What's your thinking around that when people are looking at doing that?

I can talk well about the first one and I have thoughts about the second. Dan's going to be the expert around the latter piece of that. But to get a model running in production, there's actually a ton of work you need to do from infrastructure perspective. But and this is before we talk about the workflow stuff, you know, you need to figure out how to containerize this thing and get this image running. Like, you know, as we alluded to earlier, like, just taking a model hugging face and expecting it to run is not a thing. You know, there's a bunch of requirements that these models have. There's quantization inside the code, there is different base images that they might need based on Torch and Pytorch and Python versions. And then, you know, you can really find yourself in a bit of a pickle just trying to dockerize a model. So the first thing is you need to figure out how to get this in some sort of containerized form so you can run it elsewhere once you have that. The truth is that you need to spin up some sort of service that can deal with variable traffic. And the reason why that is, is that traffic, you know, these things tend to be expensive, they tend to be bound by compute. So if you get smashed with a bunch of Requests. It's not like you can just have one model and it will queue out there, will timeout, the whole thing will slow down, your product won't work. So you need to figure out how to scale up and down with traffic. And then you need to figure out all the security concerns that come with all that. That's just on the serving layer. Now you need to start thinking about the workflow layer that sits on top of that. And I think, you know, version management is, you know, non existent for the month of hooking it up into cicd. It's to really treat this like a, a service or a microservice that has putting your model at an API need observability and logging another whole set of features. And what you realize is that taking a model and getting it working in production behind a reliable, secure performant API and maybe cost efficient as someone has done this myself and someone has built a company to try to abstract this way, it is easily for one model, couple of people have had counts, work for a couple of quarters if you're lucky at scale. And I think that is the most efficient organizations that can hire people with kubernetes, Kubernetes experience to be able to do this. And so that's type of things that we try to abstract away from our users where it's like, you know, you figure out the Python code, we'll figure out everything else and we'll give this model this first class treatment so that you can version around that you can log around it, that you can observe it and you can call it, but you don't need to think so much about that. You get that that's now to the point where you have something behind an API and ready to consume. Now there's a bunch of stuff that needs to happen to make sure that you know, it doesn't start saying random stuff that you protect against hallucinations. It's not just ingesting pii all the time. Dan can probably just talk really quickly about that as well, I'm sure.

Yeah, I think part of the reason why I'm always excited to talk to Tuan and his team at base 10 is because they are experts in this. All of those layers that we just talked about. I was actually on a call with someone the other day and we were talking about like spinning up some microservices or something and I think my comment was like, I just really don't want to care about kubernetes because I don't want to like wake up lying in a Ditch crying in the fetal position. Like that's how I view that, like whole world. So props to you and your team for dealing with that side of things. I think that's what's allowed us to then on the prediction guard side in a lot of ways to bring up a model quickly and then have the time to think about some of these other things. Tuan, I don't know if you can comment on. I have my own perspective from trying to run models for my company, but it would be interesting to hear the perspective of different Personas that are coming into base 10. Like are they people that are sort of application developers that are not infrastructure people? Are they like data scientists? What are the types of people that are coming to base 10? And maybe along with that, as you mentioned, closed APIs are getting used a lot, but still people are coming over to think about like hosting their own models. One question would be like, why? Like who are these people and why.

Totally. I'll answer the second one first. No, I think I'll answer them together. Is that it is more and more just engineers. I'd say, like, I don't know if there's any, there's any distinction now between like I'd say it's less and less data scientists, your traditional data scientists. It's more and more people with some ML exposure, product engineers, infrastructure engineers who have tried to build it themselves and have really felt the pain. I think from a product engineering perspective, like why people want to use open source APIs, I think cost is one big thing, is that OpenAI tends to stack up over time, I think, or anthropic, I think the other one is data privacy and security, is that you don't want to just be piping over all your data to OpenAI today. And especially when you start to talk about B2B use cases and enterprises. And I think there's like, probably the more interesting one is that there are just like a long tail of people working on weird models. People are fine tuning models. Fine tuning OpenAI models is not great. You get a bit more control with that manual mode with open source models. And so it's kind of like the long tail of use cases I'd say are coming more and more. And these can be engineers, they can be machine learning engineers. It can be honestly like a lot of audio models, like different modalities that there's not that much exposure to with closed APIs and a lot of custom stuff as well.

You mentioned, you know, like shipping data over to OpenAI and I have talked to gazillions of people who have that as a constraint in their businesses. You know, because the, the attorneys for the business are like, nope, you don't want to send, you know, your proprietary information and stuff over that. I guess you would not have that issue at all with base 10, would, would you? I mean that's, that kind of goes away altogether when you're hosting in that way, right?

Yeah. You have ownership of your data. We don't log any of that data. You're treating the model as just like a, like a map of input, the outputs and nothing else.

Yeah, that would really solve a lot of people's problems by taking an approach like that.

Yeah. And I think the second piece there is that once you adopt the basin approach, you can then start to think about like self hosting, deploying it within your own VPC. So, you know, we have customers that deploy base 10 within their own AWS account and data never leaves kind of their boundaries or their accepted boundaries.

Yeah, and we've kind of, I think you've framed the concerns that you're looking at with base 10 very well, these sort of infrastructure scaling concerns of hosting your own model. Could you maybe take a step back and just describe, like, if I go into base 10, like how have you architected the approach? Like to. I'm an application developer, I want to run some random fine tune of llama2 that I've created somehow. What is it like for me? What does that look like with the way that you've structured this and what's some of the thinking behind that in terms of the workflow and how you want it to be for people so that they can treat, I guess, that model as a first class thing. That is a first class asset in terms of what they're monitoring and logging, that sort of thing.

Yeah, for sure. Our goal is just to make it easier and you know, try to take away as much of the complexity, but you know, maybe more importantly is to allow you to have a bit of control. You know, I think base 10 started as like a one line to deploy your models. Like, you know, we don't believe that anymore, you know, either. We think that, you know, actually having a little bit of structure around it gives you action, bit of structure up front, gives you a lot more flexibility a bit down the line. So we have an open source library called Trust, which is basically a, it's abstraction that if you write your model in, you get kind of everything free. And so basically you need to write two things. You need to write one Python class With a load function and a predict function. And this is vanilla Python code. It can sit within your monorepo, you can specify requirements as you want. There's nothing base 10 about these files. You know, you can run them outside of base 10. I think that's very important for us as well. But once you write that load and predict function, it does two things. One, it tells us, hey, what are you trying to do here? And you know, we can load that up and when we deploy a model, we load that function when we infer we run the predict function. But more importantly, within those functions, we allow you to compile stuff down, we allow you to kind of do the tricks that you need to do so that you know you still have that control and you know, within that you can write pre processing and post processing functions that allow you to, you know, maybe like strip out some data, log something, monitor something. But really it's still giving that control at the product and application level while still abstracting out the thing we want with trust. Once you have a trust developed and you can go to Trust B Co and check out a bunch of these, it's a pretty simple abstraction and you could just push that up and we kind of give you all the workload version management around that deployed trust.

Yeah. Could you speak then to like, that's like the prep kind of that goes into, oh, I've got my weird model, I'm writing this Python class, I'm going to deploy it on somewhere. And I know that like one thing that I think is really cool, how you've made base 10 trust is like you mentioned, it is open source. And so you can run trust things and deploy in a variety of ways. One of those being like base 10's hosted infrastructure, which is of course easy, but it's also generally a great sort of framework to package your models. But let's say that you do kind of go the base 10 route. You deploy this through the base 10 client to base 10. Could you kind of compare and contrast? Like let's say I just tried to run my model in a fast API, API in an EC2 instance or ECS or like whatever that is in my cloud. What is going to be different about what I look at when I kind of look at my model in base 10 versus running this API somewhere else? And how does that make a meaningful difference or what are you trying to do in terms of making a meaningful difference for the day to day for people?

Yeah, well, I think what you're doing is that. So besides, you know, you can run that model on FAST API. Great. You got this model, you give it an input, it gives you an output. Fantastic. Let's carry on. But it's like the depth of features and the creation of workflow which are really important here. And so like the depth of features is that, hey, if you can do that with FAST API, great. You still got to set up auto scaling, you're still going to have to set up observability, you're going to have to serve logging and whatnot, hardware management. But I think the workflow is probably more important to be honest, which is I do. We're creating a defined way for you to publish new versions of this. If you want to ab test two models, you can have two models running at the same time. You know, it's really that the removal of boilerplate and the addition of some workflow so that you know when you are deploying this in production and you need to roll back a version, you don't need to go and scramble to find that fast API file that you're using before. We've all been there before. And so it's that creation of workflow is probably, I think, what a lot of our customers probably use us for. Besides, you know, I think the production grade inference is a given. But a lot of the differentiation comes from that workflow.

Just to totally boil it down, you're saving them a lot of work right there.

Yeah, 100%. Yeah.

There's a lot of kind of grunge work. It reminds me of Dan liking to do his data massaging that I'm always teasing him about. But joking aside, you're basically saving us all sorts of work so we can get into production faster, get it up and running and know that it's production grade all the way through with a minimal amount of effort and know that.

It'S just there a hundred percent. Like we're working with this customer right now. This is worth noting that they were a, this is a pretty late stage startup, raise hundreds of millions of dollars AI native product. They've got a team of four AI infra people to manage this and they've been working on this for about two years. You know, we were able to replicate and get a more performant API up and running in two days. Wow. I think that is kind of what we are trying to. It's the performance, the workflow, it's the maintainability, but it's also just the speed to prod.

I don't know how many of your users do this, but the fact that this Might be sort of revealing about me as a person and also reveal some utility of base 10. But like I can literally like last night I'm sitting on my couch and I can log into base 10 on my phone and that just change the auto scaling from like two replicas to like five max replicas and the timeout and like all those things of like the auto scaling of my Llama 2 fine tune from my couch like in between Halo games. So that was, that was terrifying. I don't know what that reveals about me as a person, but certainly like that ease of use I think is really interesting. It's like that proverb when you try to solve a problem and then you're like, I'll solve this with regex, and then you just have another problem to solve. It's kind of like that. It's like you want to deploy your model and then you want to deploy it with Kubernetes and then you have like a whole nother problem to solve and like solve the auto scaling stuff and all that.

And then like I think on top of that it's just all the SRE work that you have to do for that service. Like, you know what happens when it goes down, you know what happens when you need to migrate something over what happens when there's a new GPU you want to use. There's just so much. I feel like we've really turned the corners and I think stuff like AI and ML that really has helped here because people want to move fast is like, I feel like we've put the build versus buy debate a little bit to rest for a bit where we just don't hear it as much. It's like, hey, we want to build it ourselves. Like people are just like, we want managed solutions.

Yeah, you gotta go fast these days because if you don't, somebody else is going to get there first and you're not going to have a business.

And the market is remarkably talent constrained. Again, like Dan, you're saying this and this makes me happy because so, you know, dense background is in data platform and dealing and dealing with all these things, you know, like it is.

I've cried myself to sleep in fetal position.

And so really it's just that ease of use and like it's ease of use and the ability to scale with you. Like, and that's probably like the two things which we try to bring to our customers. And I think just even outside of base 10 is probably the, the biggest opportunity I'd say in machine learning infrastructure right now is Think of all the user stories that are important now that weren't important 12 months ago and maybe just take a long, a slightly longer term view than what can I build around OpenAI APIs which is like stopped a lot of attention. These are all places where base sand is thinking about going. Or like, you know, we, we will partner with people we're doing. If you think about people, the emails layer, think about people at the fine tuning layer, you think of people at the training layer, at the observability layer, the logging layer. It's like there's an entire new stack here to be built and that's a massive opportunity. Mosaic's a really interesting company to look at, to be honest, because they were training their models, they were helping people train model remarkably early, but the value was very, very clear to a pretty sophisticated buyer in databricks. And so to any folks building tools around here, so much value to be added.

It's Greenfield, I have another question to ask you because you are living in kind of that world of model deployments and the various ways that people are doing. People are fine tuning their own models or just using open models. I'm wondering how you see the trends going. So you've already talked a little bit about open models being available, small to big models, and how people are hosting them. There's tons of people that are also exploring this area around running models kind of at the edge or in various environments or on laptops. And also people that are exploring kind of. You already mentioned quantization and running models on CPUs potentially instead of GPUs. I'm curious, as someone who hosts a lot of models in the world, what are you seeing in terms of this trend? Because you hear about these topics, but I don't really have a good sense of. Obviously people are exploring those things, but are those people just extra loud on the Internet or. Certainly there's use cases. Chris knows them well for running certain things at the edge. But for many people out there that are maybe building a SaaS platform or something that's less relevant, although maybe the costing. You mentioned cost optimization as well around that sort of thing. So yeah, how are you seeing as someone. I guess my question is, as someone who hosts a lot of models for a lot of people, what are you seeing as people's concerns both in terms of that costing, optimizing models and deployment targets? I guess totally.

I think what we're saying is remarkably early, to be honest. There are opportunities, people are deploying stuff on edge, but I think there's not enough of them today. Just kind of think about like a generalized opportunity there. And so I think there's a company called Octoml that started with edge deployments, and I think they were just like a little bit the cloud, because that's where the opportunity today is. I think all the stuff that's happening around running these models on less and less hardware or optimizing them X way or Y way, it's remarkably intriguing. Right. It's pretty crazy that we can get a model that we can barely run on, you know, the biggest GPU we can find, and some person figured out how to compile it down or rewritten it with C kernels, and all of a sudden it runs anywhere. That's pretty fantastic. But I do think that, you know, we're still in the research phase there. We're in the experimental research phase. And like, you know, we've seen a lot of people deploy those models. Yes. We've seen a lot of people play with those models. We've seen a lot of interest in those models. But I can't really think of too many examples of people running those models in production just yet. But it seems inevitable that over time, oh, it will happen. That is the arc that we are on. Yeah, these models are getting smaller and smaller and yeah, when I'm not on.

The podcast, I'm in a world where it's all about things moving around in time and space. And a lot of those things will have AI capability on board going forward. So that's. I agree with you. There's a lot of research going on and it's not a solved problem in one. There's not a set of best practices yet, if you will, but it's an area that is majorly ripe. I'll be really curious to see if you guys or another company is able to leverage all the expertise you've built up and experience you built up in the cloud and kind of move out into those areas.

Bring your own device. Yeah, yeah.

There's millions of devices out there just waiting for you.

Yeah, I say like the, the challenge there is just around, you know, how I'm guessing Lockheed's devices are a bit of a snowflake and, you know, we can't, you know, you can't build for one type of device and just go and apply that to the next device there. And I feel like there's probably some generalization that needs to happen at the OS layer before we can do that. But I am also completely uneducated on edge stuff. So you probably have a lot more to say that than I do.

Well, and it's interesting too, I sort of ask this in a leading way because one of the people that we're talking to, I'll kind of genericize this, but they run some equipment at the edge in manufacturing and they have like a hub at the edge which is air gapped, which doesn't talk to the Internet, but their whole like next generation of things is going to be Internet connected. And when I was talking them about like doing some things with large language models in that environment, you know, essentially where we got was, hey, well it's going to be more hassle for you to figure out some of these like model optimization things and all of that than to just like set up an API in base 10 or something like that and, and just connect it out if that's where you're headed anyway. And it's not like a military security concern like one of these situations. So yeah, I think we're probably see both and but for a lot of people it's like kind of how I've categorized these things in my mind is yeah, some people will want to run Kubernetes in their own infrastructure and they have the expertise to do that. And if that's you, then like great, you're one of maybe a few people on the planet. I don't know.

Good on you.

Good, good on you. And similarly, like with Chris, if you're running a lot of models on the edge, which I know certain people are, and in certain industries, it's really important, that's great and that expertise will be there. But I think for the, I don't know, my sense is that for a lot of majority of people, like separating out that infrastructure concern of model hosting is really, really a useful way to think about things. So I don't know, we'll see. I, I'm always bad at predicting the.

Future, but we talked to this one customer, they were saying that for some reason the CEO bought a bunch of GPUs and they literally have machines in the office. They're like, oh well, we're going to do the. I think Amazon has like the kubernetes anywhere or something where you can basically.

Like a hybrid sort of thing.

Yeah, yeah, you know, we ran away from that opportunity, suffices to say, but I think these people are thinking about these problems. I don't know if there is a solution here just yet.

Yeah, well, as you kind of think about so obviously, and I know just from our discussions, like you're helping a Lot of people and really significant use cases in the space already with what you're doing with the infrastructure side of model hosting. But as you look to kind of the next, I can't even say like the next years because things move so quickly. But as you look towards the future and like what is not yet solved on the infrastructure side with model hosting and what are like you and base 10 really excited to dig into what comes to mind and what are you thinking about?

Yeah, I think within the containers like that layer is. Gets very interesting. I think you know vlm, which I'm sure you've played around with TGI which, you know they're both great. I think they're still far from ready for PrimeTime just yet. VLM, TGI and Tic LLM Nvidia just put out. I think there's going to be more and more of these frameworks and supporting these frameworks I think is going to be very, very key for us and what we're really excited about so we're going deeper at that layer so you can kind of bring your own framework on your container and really benefit from that. So you know, we're gonna have first class support for TRT LLM pretty soon and we already do it for TGI and BLM and I think that side's pretty interesting. I think you know we have a big launch coming up. I'm happy to talk about it right now actually but around multicluster so that's basically being able to one use your own compute to bring your compute to base 10. So the control plane sits on base 10 and the workload plane considered GCP, Azure, AWS or some combination of the three and then we'll keep adding clouds to that and so I think that's very, very exciting especially in the enterprise that because you know, people want self hosted.

That's huge.

It is, it's really nice.

That's gonna be big and then kind of just beyond serving and I think with really signed, you know we've already had one foray and we learned a lot and you know we, we, we end up retiring but we're going to get into fine tuning at some point. I think like that's, we keep seeing like, just like I said about kind of like the edge device stuff and the compilation stuff is that fine tuning is still an art. I'm a little bearish on like APIs that say give me your data, I'll give you a model. Like I think you need more controls. As someone who built an API that said give Us your data, we'll give you a model with blueprint. You know, I think you need more control. You need control over your base models. You need more control over even the fine tuning scripts to customize that model. We'll start to think about that very soon. You know, we're already doing a bunch of work with customers to make sure that we're marching in the right direction. So I'm very excited about that, which is that, you know, over time base 10 becomes this place where you can run your models. Great. But then you can also start to collect data sets around your model. Just imagine if you could just give your model to base 10 and then, you know, you either opt in and we basically write all your input and output data to S3.

That's beautiful.

Yeah. Like essentially a level of caching for model inputs, outputs.

Yeah, exactly.

Yeah. The multi cloud thing really will be big for enterprise, by the way, just to that point because I think most enterprises across many industries are recognizing that their future is a multi cloud world. It's no longer tied to one. And if you have base 10 able to do that hosting and run a control plane and you can deploy into any of the cloud clusters that you happen to be, and maybe different parts of the company emphasize one or the other, then that takes a lot of challenge that they're currently facing out of that. So it's pretty sweet.

I mean, also opens up opportunities, right? Especially in a GPU can train world. Yeah, you can get them from wherever you want. And then, you know, I think once you have data sets as well, then fine tuning just becomes obvious which is like, okay, now I can I fine tune this and you know, or maybe it's even like, hey, hook up your OpenAI endpoint using base 10 so we can collect that data set and then we can create that fine tuned Mistral or Llama to keep you the more you want. So I think there's a lot of interesting things along that whole stuff. And as I said to you guys earlier, there's so much opportunity here for people building in the tooling layer, brand AI and ML. It's very exciting overall.

Yeah, well, we appreciate you taking time out of doing great work in that layer to talk to us and share with our listeners. This is a great conversation and hopefully we have you on the show in less than three years from now, but if not at least three years from now, hopefully sooner. So thank you for joining us again and giving us an update and some insights around this and really appreciate what you all are doing and appreciate you taking time.

Thank you. Thank you so much for spending time on the show. I thought it's fun to be on it. Foreign.

Thank you for listening to Practical AI. Your next step is to subscribe now, if you haven't already, and if you're a longtime listener of the show. Help us reach more people by sharing Practical AI with your friends and colleagues. Thanks once again to Fastly and Fly for partnering with us to bring you all Changelog podcasts. Check out what they're up to@fastly.com and fly IO. And to our Beat Freakin residents, Breakmaster Cylinder, for continuously cranking out the best beats in the biz. That's all for now. We'll talk to you again next time.