Serverless GPUs

Foreign.

Welcome to Practical AI. If you work in artificial intelligence, aspire to, or are curious how AI related technologies are changing the world, this is the show for you. Thank you to our partners at Fastly for shipping all of our pods super fast to wherever you listen. Check them out@fastly.com and to our friends at Fly. Deploy your app, servers and database close to your users. No ops required. Learn more@flyio.

Welcome to another episode of Practical AI. This is Daniel Whitenack. I'm a data scientist with SIL International and I'm joined as always by my co host Chris Benson, who is a tech strategist at Lockheed Martin. How you doing, Chris?

I'm doing fine. It's been interesting times, though it's not what we're going to be talking about today. Been watching the showdown between, you know, Google and Microsoft over ChatGPT and Bard and you know, things are happening as we're recording this, so I thought maybe.

You would have been too distracted with the new Harry Potter game.

Well, there is that. Yes. We all have our secret little things.

That we do to keep entertained.

Yeah. Yeah. Well, I forget one of our recent guests brought up that quote of like, you don't need to do machine learning like Google and you're talking about like Google and Bard and all of these things. And when you think about those things, you think about like, oh, these like data centers full of GPUs and these huge supercomputers that they've got at their disposal to do things, which isn't the type of GPU infrastructure that most practitioners have access to. And that happens to be maybe the topic of what we'll get into today. A little bit. Excellent with Eric Dundeman, founder of Banana Serverless GPUs. Welcome, Eric.

Thank you. That was a beautiful lead in. Definitely want to help people get the Google level infrastructure without that level of effort. So glad to be here.

Awesome. Yeah. Well, we're really excited to have you. I have to say I did spin up a model in Banana leading up to this conversation, so I'm pretty excited to talk about it. But before we get into the specifics of all the cool things that you're doing, I know that our listeners, like I say, they're probably very familiar with GPUs and why they're important to AI machine learning modeling, but maybe they've just heard of serverless as this cloud thing. That is a thing that people do in the cloud. They do serverless things and they've never thought about serverless GPUs could you just step back for a second and describe first off, for people that might need just a very brief intro, what do you mean when you say serverless and then kind of take us into serverless GPUs? Is that a new thing? Has that existed before? Curious to hear your perspective.

So I love your specific phrasing. What do you mean when you say serverless? Because serverless is one of those terms that nobody has really pinned down. Exactly what it defines our working definition is this idea that when you need capacity, when you need servers to handle your requests, when you're in periods of spikes in surges of use, you have more servers, when you have less use, you have fewer servers, and when you have no use, you have zero servers. And the idea of this is to make it so that you as an engineering team and as a product don't need to think about your compute as a fixed cost. It allows you to essentially view it as pretty much per request, pay as you go. Funny enough, serverless really does mean servers running under the hood. But the less is that you just don't need to think about it. You think about it less. Happy to dive into what the details of that mean in regards to GPUs, but serverless has been around for about 10, 15 years. I don't know my exact timelines, but it's been a concept within CPU based computer serving things like websites backends. And people have been wanting this to exist for GPUs for a long time and nobody's really cracked it. And that's the challenge we've been working on.

I know that like you talked about websites about like backends, that sort of thing. Just in general, when we're Talking about serverless GPUs in your mind is a use case that you have mostly on like the inference side or on the like training side of what practitioners are doing, or is there a little bit of both?

The vast majority, at least from what we've seen, is on inference. And I think inference is where the value of serverless comes in the most. There's other tools for training where it's not as latency constraint, where you could use other infrastructure orchestration tools, but for specifically inference. Serverless is one of the keys to the kingdom if you could really do serverless well. So we just as a team have chosen to focus mainly on inference, inference, real time inference. So if there's a user at the other end waiting for a response, we're the ones responsible for making that response happen quickly.

Gotcha. And why has it taken so long to get to serverless GPUs vs serverless CPUs?

One of the biggest problems in serverless is what's called the cold boot time. Cold boot, as in you don't have servers running. A request comes in, that request coming in triggers a server scale up, going from zero to one and then one to many. And the time it takes in order to get resources provisioned and ready to handle requests in CPUs can take, you know, a couple seconds on a platform like AWS Lambda could take multiple seconds, maybe 10 seconds for a cold boot. And that's just simply spinning up the environment, spinning up a container or a micro vm, whatever they're running, and getting an HTTP server ready to handle that particular call or set of calls for the user before then shutting down. So coldboot has been a big blocker, and it's primarily the initialization time of the application before handling jobs on GPUs and machine learning exponentially harder. Reason being we're running 20 gigabyte models. Those models can't be taking up RAM before a call comes in because that is not serverless. Then you're just running an always on replic. So the cold boot problem is deeply exaggerated when you get to GPUs because not only do you need to provision the GPUs and the environment or the container, you need to load that model from disk onto CPU onto GPU. That process could take 10 minutes for some models. And it's just been pretty huge blocker for most GPU use cases. So for that reason this product hasn't existed before.

Definitely not trying to delve into the secret sauce if you will, but can you kind of lay the landscape of how you even start to think about that problem? Like what are the some of the different ways that you might address? And maybe different orgs, you know, as you develop competition over time, probably different people will take different approaches. Like how do you even think about that landscape? Because that seems like a daunting task. You know, when you talk about 10 minutes to get it moved over and stuff, that's huge. Like how do you even start to approach the problem?

So this is definitely one of our most prized pieces of ip, our cold boot tech. So can't dive too deep into the details.

No worries, whatever works.

What is publicly known, you got to think about getting. Well firstly, constraint, constraint. You cannot take up GPU RAM if you have a 40 gigabyte, a 100 machine. If you put a model into that RAM, that portion of the RAM or like that machine entirely. If you're not virtualizing it, it's just like it's taken, you are paying for it. It is dead space if you're not using it, that's massive GPU burn without any utilization. So constraint model can't sit in ram.

Okay.

At least GPU ram. So when we go about the cold boot problem, what we're really thinking about is how do we get the model, specifically the weights, as close to RAM as possible without actually occupying resources or more precious compute resources like 40 gigs of limited RAM. That's hard. But if you have a terabyte of storage on the machine, you could at least have local caching the model. So like you could take that up passively between calls without incurring, you know, sacrificing that piece of hardware because you could fit so many more models onto the disk.

Gotcha.

And then you could start thinking about like, you know, how do you start pre caching this on the CPU if the CPU has enough? Not saying that's something we do, but these are like the framework in which you would start thinking about it is how do we get that RAM or that model as close to GPU RAM without actually taking up GPU ram? Because in the end, GPU RAM is, that's where the cost goes. Because once you use that, that machine is tied up, it's not usable for anything else.

In your experience, I mean, I know you've been likely talking to tons of different clients, different use cases that are really kind of thinking about how, how their workflows could adapt to the serverless workflow. I'm just thinking about my own workflows. Like we're running a lot of models, but none of our models on my team are receiving thousands of inferences per second or something like that. It is very much in this zone where we get a burst of activity and then we're kind of down for a bit, not getting that much, and then maybe another burst that we need to process. So in that case, like, I would probably be willing in my own use cases to put up with somewhat like longer of a cold start like response for the model when it comes up and then subsequent ones during that burst being much faster. What have you noticed with clients, like, what is the tolerance there? Like, where are you trying to get and where do you think is reasonable for most workflows?

I guess I don't have a perfect answer for you on this in that ideally cold boots are zero.

Yes, that's true.

I guess on banana on A serverless platform in general. Unfortunately, you do have to start thinking about the servers because you want to avoid cold boots when avoidable. In the case of Banana, if you have a model, it's undergone a cold boot, it's handled the first call, it's ready to go, we have it configured to hang around for 10 seconds just in case more calls come in. And that 10 seconds is completely configurable by the user. So if no calls come in, we consider it. Okay, we've gone through the surge, we could scale down that particular replica scales itself down. If calls start coming in again, cold boots are incurred again only if the existing replication you have can't handle that throughput and start scaling up more. So because we give users the ability to like fine tune their autoscaler in a sense, or fine tune, maybe configure. Yeah, configure. You can configure the autoscaler. So we have some users who choose to run always on replicas with a minimum replica count. So at any given time maybe you have a baseline of two GPUs running, but you could surge to 20 if you need. So we have some users doing that. We have some users who have gone away from the default 10 seconds idle time to go longer because they know like they would rather pay for those GPUs to be up and handle any traffic that may come in, then have more frequent cold boots. Reason I give that context about Banana is I've been really surprised by how few users increase their idle time right now. Or at least the majority of the customers we're serving are more price sensitive than latency sensitive. At least given the general trade off we give them in that they could configure the idle timeout and through that tune how much they pay versus how much they wait. But most users would rather have machines shut down and then incur that cold start time. And that's a great thing for us because that allows us to chip away at this cold start problem and give users an exclusively better experience of the faster cold starts are, the more willing users are to take those cold starts because it's less impactful on their inferences and the less idle time you could run on your GPUs following calls before they start shutting down. Because it's not as risky, you know.

Hello friends, this is Jared here to tell you about Changelog. Over the years, many of our most die hard listeners have asked us for ways they can support our work here at Changelog. We didn't have an answer for them For a long time. But finally we created Changelog, a membership you can join to directly support our work. As a thank you, we save you some time with an ad free feed, sprinkle in bonuses like extended episodes and give you first access to the new stuff we dream up. Learn all about it@changelog.com you'll also find the link in your chapter data and show notes once again, that's changelog.com check it out. We'd love to have you with us.

So as you were kind of describing that, that was really. It's a very interesting mesh of skills. It seems to do what you're doing there because you obviously have to have a pretty good understanding of deep learning in general and kind of the AI space and the performance characteristics around that. But you also have to go very, very deep in terms of network engineering and architectural considerations and such like that. It also kind of brings different cultures together, for instance, in terms of like the choices of languages and stuff. Distinctly like, do you tend to go with one language for everything for simplicity's sake, or do you tend to go with different languages that are catered towards specific use cases by way of example, like Python for deep learning specific things and like Rust, you know, or something. C for infrastructure thing. How do you. Or do you stick with one like Python for everything because that way you have a simpler setup to govern. How do you take that strategy wise?

So the obvious language for hosting ML model inference is Python. It's almost a requisite, as in all of our users are running in it. So therefore the framework that we give users to build off of, which is essentially boilerplate for a server that's written in Python. We don't need to maintain that too much. It's an extremely simple HTTP wrapper and the vast majority of our work on the pipeline and the infrastructure side is all done in Go. So we're probably 95% Go. We have some TypeScript for our web app, some next JS that we're running, and then when you get deep into like the runtime, we work on C and Cuda as well, but that's a small subset of our engineering team works at that level. The majority of us write pipelines and networks within go.

I gotta say, it's kind of funny that you bring that up. Daniel and I love go. We actually met in the GO community because we're both GoPro. We were a like at the time, kind of the two AI oriented people in the GO community. So it's just a little bit ironic.

To hear that that's awesome. I've been so disappointed in Python. I mean, Python's amazing language. It's where I learned my first bit of serious general purpose programming was Python. But I'm saddened to know that the language you chose for GPU programming basically is a language that like has a global interpreter lock like you. It does not have great multiprocessing built in. I wish Go were the choice there. It doesn't seem like it's going to happen, but I'm huge fan of go. I think it's a great language to write in and I could go on for a long time about this. In fact, one of the reasons I learned about the Changelog network was listening to the GOT podcast.

So yeah, for sure.

Shout out, Shout out to that other podcast.

Yeah, definitely, definitely. It's cool to hear about, I guess, the setup of how you thought about this problem and how you even structured the team and that sort of thing. I'm wondering at this point if you could kind of just give us a sense for like if I'm a data scientist, if I'm or even just a software engineer trying to integrate a model into my stack, what does the workflow as of now look like for me with Banana, like what do I do to get a model up and going and maybe just a couple examples of that to give people a sense. It's a bit hard on audio podcasts, but I'm sure you've done similar things in the past, so.

Yeah, well, I'd love to give a visual demo, but going through audio wise, generally the process looks like this. A lot of people are building off of standard models, so say a stable diffusion or a whisper at least for like this current, this hype wave of all these new exciting open source models coming out.

Until next week.

Yeah, until next week. And then the next one comes out.

Yeah.

Thankfully we have these one click templates that you could use on Banana. So in a single click you could go from an open source model that somebody has published on Banana and bring that into your own account and start using it yourself. So within a few seconds you could have a functioning endpoint for popular models that have been put up by the community. And then we see naturally the step beyond that moving from you effectively have an API, you don't really know what's running behind the scenes. You could fork that code, you start working on it yourself and customizing it for your own use case. So if you're doing some fine tuning, if quite honestly you want to go away from the standard or like the big model templates and roll it yourself. Just have whatever deep net that you've built. That's where you start getting into sort of the local dev iteration cycle. And this is where I shout out a previous guest Brev Natter over there. At Brev, we recommend users go and have an interactive GPU environment so that you could load your model, test it against some inference payload, shut it down, iterate. If you're doing something like a stable diffusion, you want to make sure that the image transformations server side are happening correctly. That's where you iterate. You're doing all of this within the Banana framework. We have an HTTP framework you could find open source online. That's generally the building point for most users. So you're modifying a function within that, that is the inference function, takes in some JSON, runs the model, returns some JSON, do that iteratively until you have your customized model that works to the API you're hoping for. And then you push that to GitHub. Then from there you could go into Banana, you could select that repo and we have a CI pipeline built in. So when you select that repo, we build the model, we deploy it, every time you push to main, we rebuild and redeploy. So we generally recommend users to, if they're shipping new fine tuned versions, it's usually them updating say a link to an S3. Then in the build pipeline we bundle that model into the container itself and get that deployed to the GPUs.

So kind of curious, and this is sort of a follow up largely because of the medium we're in, since we're audio only and we don't have the ability to, to show the process that you're describing. Just for clarity, like your typical customer user, what skills would they typically have to productively use Banana? What are those necessary minimum skills for them to be able to really engage productively and move through things?

A lot of our users are quite surprisingly full stack engineers and not deep experienced data people and ML people. So as long as you can wrap your head around using frameworks or abstractions like hugging face, for example, if you could use a pipeline like that, pull it locally, that's something you could deploy into Banana. So some Python expertise in order to write the code in the first place, it's an HTTP server, so you write that, you wrap it around say a hugging face model, you don't need to fine tune it. You could use the standard models and then learn fine tuning later. And ideally you do have some knowledge of Docker. Ultimately what is deployed to Banana is a Docker file. If you build within our template, generally you don't need to do things that are too custom unless you choose to. But a little bit of knowledge of Docker helps. So Python hugging face Docker, that's effectively all you need in order to get something deployed onto Banana.

I'm just on the site now and kind of looking through some of your community templates, which are pretty cool. I mean you have all sorts of things, Codegen, T5, Santa, coder, all sorts of things with a sort of one click deploy button to get them up and going. One question I had just like when I deploy because it looks like based on your docs, I can call it with like the model ID from Python for example. So I could like integrate this directly in a Python app. Can I also call it sort of like as a REST endpoint or something like that? Or is the primary use case a client integration?

We do have public documentation for the REST endpoint, so awesome. It's not officially supported. We try to encourage people to go through our official SDKs, which at this point are Python typescript, go in Rust. That said, anyone who wants to go directly into the REST endpoint, there's documentation to do so. We like being able to boil it down to a simple Banana run function where you just give a model key, you give whatever JSON in you want your server to process, and then you receive the JSON out from that. But our goal is to be able to give people access to levels of extraction that they choose to run in. For example, because we have a public REST endpoint, people have integrated Banana into their Swift applications or into their Ruby applications. So it's an HTTP call. In the end, people could unwrap our APIs and go at it directly, feel free.

Yeah, I guess that leads right into my next question, which is does anything stand out in terms of how people are using this serverless workflo that maybe surprise you based on what you're seeing?

I've been amazed at the quantity of fine tunes that are deployed through Banana. If you look at the analytics of people Deploying from our one click templates versus people deploying from custom repos, 80% are custom repos. And that means that people are coming to serverless because they have a unique API that they need to run somewhere and that they can't simply run with a standard API provider or even an API provider with fine tuning features. Like they want to be able to own the API themselves, own the application logic themselves. To fine tune themselves and just dockerize that up and send it onto Banana. So vast majority of our users are doing custom workloads, which to me was surprisingly little Banana lore. We previously started as an ML, as an API company. The idea of showing up, click the model you want and you get an API for that. And there's a lot of pull there, especially right now with the hype. There's so many people who want to integrate AI into their applications without touching the AI at all. So it has been surprising for us, seeing how many people are running custom code on us. And it's been validating of the idea that the platform approach versus the API approach has been the way to go.

Could you kind of walk us through what a typical one might look like, you know, where someone's doing that kind of custom thing just to give us a sense of how, what it is that you're saying, whether it's fictional but realistic or a real case example, whatever works for you.

So one thing users are doing just as a very basic example of, if latency is an extremely sensitive thing for them and cold boots are particularly painful, what they'll do is they'll engineer a conditional like a boolean in the JSON that they send in. That's called the warmup. So they'll do like warmup equals true and make it so that server side, they actually don't perform any heavy computation, it's just intended as a warm up call. So if architecturally they need servers running like fully warmed up by the time the actual inference starts running, they engineer this into their endpoint. Another thing as well, if people want to run fine tunes or run multiple models side by side and start doing some model chaining, we see people building that into banana as well. And then lastly are just basically state of the art moves so fast right now that the second stable diffusion launched, for example, suddenly there's inpainting, and inpainting is the next thing that came out a week later, and that's some random code people found in a GitHub and they integrated themselves. So customization on that sense allows users to stay as far ahead as they possibly can if it's necessary for their use case.

Could you highlight something you have in your mind as maybe like a workflow that would not be appropriate for the sort of serverless GPU infrastructure? So I think this, like you say, fine tune models inferencing using these state of the art templates, is there something where you would say, hey, maybe that's not fitting for the serverless use case.

Yeah. So in inference land, if you have completely steady traffic all the time, don't use serverless, you'll get unnecessary cold boots and it just slows down your inference and you're paying effectively the same. So that's the inference side, training side. We'd like to think that you could currently train on Banana, though I often find that training is a more interactive experience or at least in like the initial prototyping phase. Once you have pipelines built in to say automatically collect data and batch train. That actually does work on Banana, because you could just fire that data as the payload train the model server side, upload it to S3, return the call and then the replica shuts down. But most training jobs, or most like exploratory training jobs I would not recommend doing on serverless. In part just due to the like the observability that you need to see the tracing setting up things like this is outdated tech. But tensorboard, there's more visualization tools. Also keep in mind I'm not a training expert, so perhaps there's space in the training that people would see value in serverless, but generally I'd recommend avoiding serverless. And then lastly, if you have any jobs that are batched, as in you know exactly when they're going to happen, it's a bit easier to automate your own infrastructure and build it yourself to do that. Ideally we make serverless so good that you don't need to think about that. But I think in the current state of serverless, a lot of batch processing jobs, if you're say running an indexer across an internal database and you don't need to have it running all the time, that's where running on serverless maybe a bit too much lift in order to port it into serverless versus just doing it yourself.

I was looking. I'm also looking through your website while we're talking and I'm in the docs and I kind of hit the SDK area which you kind of talked about a little bit ago with the different SDKs in Python Node go rest. Did you mention Rust earlier or did I mishear that as Rust? I may have misheard something.

I did mention Rust. I actually don't know if we have it documented. We launched it two days ago if I recall.

Gotcha. So the thing that got me thinking here, that's very leading edge. It's very like out there. I'm kind of getting the sense that your, your customers are adopting more, you know, forward leaning languages in general. For what they're doing. And that's why they're, you know, they're leaning forward into this new concept of serverless GPUs. Is that, is that consistent with what you're seeing? Are you really kind of targeting the types of software developers that are kind of early adopters paving the way versus somebody that's maybe in some of the older, more enterprise Y languages, maybe not quite as risk taking and such?

That's very much in line with what we've been seeing. We find that a lot of our users are adamant Vercel users as an example. So they're in next js. They've chosen a relatively modern framework to build their front end apps in and they make the same decisions for their backend. They're often typescript forward. If they want to do systems level, they'll do Rust or go. For these reasons we've chosen to offer these official SDKs.

Yeah, that's really interesting. One of my questions in kind of thinking about this is like the different use cases that you could have, the different industries that are rapidly adopting AI integrating it in their software stacks, like everybody's adopting AI, right? But it's certainly making a lot of strides in certain areas. And certain industries like let's say healthcare or something like that have very unique constraints around even their own inference data leaving to go to some hosted model somewhere that's not in their own infrastructure. But in other words, when I go to Banana, I see all I have to care about is like deploying a model. There's my model id, right? I can think about like the timeout and all of that. It's all very functional, right. And I don't even have to like give a thought for where that's running. I could see like the opposite end of that is like certain industries would probably be a little bit uncomfortable with that. But there's a whole lot of developers that are just wanting to like, you know, bootstrap these like amazing AI powered things like very rapidly. There's so many things coming to market like this that. So I guess that would be fitting in that way. Do you have any plans in the future for like Banana Serverless but like connect my AWS infrastructure or something like that to run in the Banana way or something like that?

Short answer, yes, long answer it's complicated. It's going to be a long time. Yeah, it's very complicated. And one of the things that we see with serverless is the fact that we have economies of scale sharing everyone as tenants within our cloud because that allows us to do more efficient bin packing and make it so that when you're not using a server, like when the server container is shut down, you're not charged. If you're running on your own cloud, you still need to have the underlying resources running. We're a venture scale business. We want to hit that million dollar annual revenue ideally or sorry, not million$100 million annual revenue ideally more. And I think getting into that, we're eventually going to have to start thinking about how to more traditional enterprises integrate this Though choosing our niche right now we see significant pull that could get us to one $10 million annual just from these like new teams who aren't bound by such constraints of needing to run in their own cloud. So long answer restated. We'll get to it eventually and I'm sure it's like it'll be a necessary part of the product, but it loses out on a lot of the magic that we're currently providing. So we'd rather just focus on these new and upcoming startups that are running on us.

Yeah, that makes a lot of sense. I think it does make me wonder like because you are creating so much magic for the users and a lot of that like you're saying, like thinking about like what GPUs are you spinning up? Like how are you bidding on these? Like where are you, like how are you allocating them? Have you learned any sort of like general, like you can get GPUs from a lot of places, there's a lot of different kind of scales of pricing, there's a lot of different ways to run GPUs in the cloud. Have you found any just sort of like good practices or things that you found to be useful just generally in terms of thinking about using GPUs in the cloud that you'd love to pass on to listeners.

So we use this phrase called skate ahead at the puck. It's a phrase from hockey where don't go to where the puck is, go to where it's going. So applying that to auto scaling, auto scaling really has two components. You're auto scaling the underlying nodes. The hardware that's running the GPUs like that's running the Kubernetes cluster, whatever your deployment target is. And then secondly your auto scaling the deployments themselves, going from replication of 0 to 1 to many within the confines of whatever nodes you have set up. So you're effectively auto scaling two things, Kubernetes, pods and the nodes themselves. So my recommendation are if people are building Things like this in house, what they should absolutely do is use a platform that has an automation API for the underlying VMs. Right now, GPU cloud is sort of the wild west. There's a lot of new players, traditional hyperscaler clouds like Google Cloud, aws, Azure, they have the automation. But the GPU prices are not as competitive as you could get on some of these newer clouds. So my biggest recommendation for people building mature systems would be to like choose a provider that you get, ideally guaranteed access to GPUs, which allows you to scale your GPUs up ahead of the demand of whatever workloads you're running within your cluster and then doesn't have to be homogenous the workloads deployed. Just as long as you maintain GPU capacity to handle those, you should be good. But because you're auto scaling like the applications within Kubernetes allows you to have a little more lead time for like super slow scale ups on the GPUs.

This has been a super instructive conversation. I'm learning a lot. I want to extend your analogy one question further, because you're talking about skating ahead of the puck, not skating to the puck, but where it's going to go. You are pioneering this field. You are out there on the front, you are leaning forward and you are supporting other people in other organizations that are trying to lean forward as well. So I'm going to ask you, where is the puck going? You know, short term, middle, middle, long term? How do you see the future for those who are not in your industry but are, are going to be supported by you? Tell us the vision, what's it going to?

Fine tunes are going to be huge. I think there's two camps for where AI is going to be going. There's the one model to rule them all camp, which is there's going to be some mega model that does everything and then there's the other camp, which is what we're leaning into. Which is the best model for you as a user is a model that's trained on data from you, specifically you. And we see customers deploying fine tunes on us, not just for their use case, but for their end user. Imagine you are building a writing assistant app. How do you fine tune for every single one of your end users and deploy that and make it so that that user has a unique model. It's essentially a companion, almost a clone of them. And where the puck is going is where we, every human on earth, just like they have a phone in their pocket, they're going to have a fleet of models fine tuned just on that. And that's one thing we're excited about with Serverless is in order to do that viably, you got to have Serverless. Can't have it running all the time. So very exciting in this sense. If you're not looking into user level fine tunes, I think it's a very interesting space to be in because it gets you so much further than any application level stuff you could do to make the experience better.

That's awesome. Yeah, I think that's a super exciting way to close out the conversation. This is a really exciting time to be in the space, both in terms of what's possible with fine tuning and those sorts of technologies, but also like new infrastructure coming up, like what you're building. So thanks so much for taking time to chat with us, Eric. It's been a real pleasure.

This is awesome. Appreciate it, guys.

Thank you for listening to Practical AI. Your next step is to subscribe now if you haven't already, and if you're a longtime listener of the show. Help us reach more people by sharing Practical AI with your friends and colleagues. Thanks once again to Fastly and Fly for partnering with us to bring you all Changelog podcasts. Check out what they're up to@fastly.com and fly IO. And to our Beat Freakin residents, Breakmaster Cylinder, for continuously cranking out the best beats in the biz. That's all for now. We'll talk to you again next time.