Own Your LLM: Deploy + Build an App on It (Modal Auto Endpoints)

Like all my other videos, this is going to be straight to the point. So everybody currently in AI, or people that are building with AI, this is individuals and also enterprises. What we are all doing is renting all the models we're using essentially through somebody else's API. So if you think about OpenAI, Anthropic, and some of the other model providers out there, we're using the APIs to actually build our products. So today I'll show you how to own your own inference instead. And so this will involve deploying an open source model on your own endpoint and then building an app on top of it. And this will take about maybe 5 minutes, and 5 minutes minus the time of deploying that model to the inference provider that I'm going to be talking about today. And also, by the way, this is not a sponsored video. This is me as a customer of that product. or service that I use, and I find that it's very useful and sharing it out there for other folks that might also find these very useful as well. So let's go ahead and do it. All right. So the service that I'm talking about is Mortal. If you've never heard about Mortal, they're a startup, I think Series C startup at the moment, and they are a serverless compute environment or either provide compute so that way you can train your models there. You can also now inference your models directly through them. And then also I've been fine-tuning some of the large language models for my own personal use cases. And they provide that and they provide $30 credits in the beginning when you start with them, which is amazing. But to go over this blog, which they released yesterday, and this new service they released is called actually Auto Endpoints, which is amazing. And the selling point here that they of course underscored is optimized inference for you actually own. So this is very important. Important, especially right now. So the biggest problem right now, which I talked earlier, is when we build products as enterprises and individuals, we use APIs from these big labs, right? And we are beholden to their rules of engagement in terms of how we use these models. And a very good example is Anthropic recently with Fable-5. But if you're not familiar with it, they released a very strong model Which is a watered-down version of Mythos, which they had promised, but then they yanked it or the government essentially put— essentially the government put restrictions on it. Not other countries cannot use it. So they ended up pulling that model out of production and we had already kind of used it for quite a few days and I was loving it as well. But that creates a weird scenario where if you're a business and you had already kind of built your infrastructure on top of that and your business, that Leaves you kind of in the limbo, right? And you can imagine that is only for the frontier piece. What about the rest of them? And then there's also the aspect where they talked about that model. If you had to ask it a question about improving your, or building an ML or building a model that would exceed maybe what they're already doing in a sense, it would essentially lie to you. So basically they nerfed the model so that it doesn't give you, it doesn't become maximally truth-seeking for you. They're going to downgrade it to 4.8. Or another model, but then they won't tell you that. They, of course, later on retracted that statement and said, we will let you know when we downgrade, which is great, but not amazing either. So this leaves us with a few options. And I think a lot of people have been starting to look at open source models. So yes, you can start looking at open source models and there's companies like ZLabs, which recently just released GLM 5.2, which is pretty strong based on the benchmarks that we've seen and also usage. So it's a very strong open source model. So if you want to use open source models, then the question becomes you got to deploy it somewhere and use it somehow, right? It's not just— you need compute. So this is where Modal comes in. And there's so many other providers out there, of course, for inference. So a quick thing here that I'm seeing. So they kind of basically provide you a way to deploy your own model. So if you pick the GLM-5 or Quen or any other open source, or you fine-tune your own model, for example, or you build your own model, you can also deploy directly to their services in this new Auto Endpoint. And what that does is it gives you an API endpoint that you can call. You can basically plug it into any other place. So here they give an example of where places you can plug into. So you can plug it into any agent harness that you have and it should be able to work. Even regular old writing your code and calling it as an API would work still. So that is the beauty of this. You get an endpoint, you get your own personal API instead of using some other person's API, you get your own API. Directly deployed there, and then you also get all the metrics and everything behind it as well. Yeah, GLM 5.2, and I'm going to show you later. I did deploy this GLM 5.2 FP8, so this just means it's going to use eight different B200 GPUs, which are quite pricey. I think I used it for like ten minutes, and it was like seventeen bucks already. Yeah, for now, maybe not for me, but these are the model options they provide. So I'm going to go actually and do a demo so you can actually see it working in real in real time. We're going to deploy an endpoint. We're going to deploy a model, and then actually. A Streamlit app and actually go ahead and use it so you can see the end-to-end of that. So let's go ahead and do it. So I'm going to make some few assumptions. I'm going to assume you've gone to Mortal, you've created an account, but once you create an account, you should be able to have access to— you pick a space, of course, that you want to work on. For mine, it's just my name. And from there, there's this new section. So there's apps, there's a bunch of tabs here which are amazing. But if you come here to the endpoints, which is of course tagged as new because it's pretty new, this is where you can create the endpoint and actually deploy it here. Instead of using the CLI, if you want to do the UI side of things, this is where you would do it. But you can also just browse all the models that you have, which is another great place to go. So if you come here, you should be able to browse all of them. So you'll see the DeepSeek type models, you'll see Gemma 4, you'll see GLM, GPT Open Source, KIMI, Nimotron, Quen, and then all the other vision ones, of course. So my guess is they'll keep adding them, which is amazing. Now for the sake of this demo, I'm going to deploy a Gemma model. So we're going to pick this Gemma 4 here and we're going to deploy this model. So if you come here, you'll see there's 2 options. So open weights and then custom weights. But let's do this one. So we're going to do that. We're just going to leave the name as it is. I've already generated the tokens here, so I don't need to regenerate them. For you, if this is the first time, it'll ask you to generate them if you want, of course, which is great. So that people don't use your endpoint and burn through your tokens there. And the good thing here is you can just save it somewhere and then you can reference it as well, of course. But then here, that's all you need to do. And then create endpoint and it will start creating the endpoint. So it should kick off to another space here. You can see now the overview. This is where you'll see the progress. So pending. So now it's downloading the model first, and then it's going to deploy the model and then it's going to warm up the model and then it'll be ready for you to actually use. So for now, there's nothing else we can do right now other than just kind of just sit here and wait. So if you go through here, you'll see that we have the Gemma model there. And then endpoint there from the hub at Hugging Face with the Google Gemma 4 B2B. And then required authentication is true because I already generated tokens. And then it's going to be run on A110, A100 up there. And it's just going to be 1. And then there's scaling down window. The good thing with this is also when you use this, when you deploy these models and you have that API, if nobody's using it, it scales to 0. So you don't actually get charged for it, which is great. But when it's in use and warm, then you get charged, of course, for the compute there as well. So that is on that end. And while that is provisioning, I would like to talk to you about the UI that I'm going to use to actually showcase or basically use this model. So I'm going to use Streamlit as the UI. And what I'm going to do is I'm just going to build a chat application with Streamlit. And I'm going to ping this API that we just— we're going to deploy here. And that API, we're going to actually Have it become like a chatbot that you can chat with. But remember what I said, you can deploy that since it's an API now, you can add it into any other agent harness that you have. But in this case, for demo purposes, I'm just going to use Streamlit. So while the model is deploying, I'm just going to show you the Streamlit app real quickly before we actually go into running the demo. This is Streamlit app. I just vibecoded it pretty quickly and it has everything we need to do here. So it It's using the OpenAI SDK, which is great because we can just basically plug into any endpoint and it will work. And so here we're cleaning up the endpoint. So we're going to need a few things here. So this is provided by model, by the way. Once we deploy our model, we'll get this endpoint URL for the model. We'll also get the model key and the model secret, which we'll also save here in the environment. So I'm using the environment example here so you don't see my keys. This is how it looks like basically on my .env file. And of course, without the Xs, with the real keys here, but this is how it's going to look like. And of course, replace the GLM piece here with the one that we will get directly from. So you don't have to worry about this. This will be given to you. All you need to do is just copy paste it to the .env. And then of course, the auth token ID and the token secret there, of course. But that's all it is. This just does the cleaning portion of that to make sure that the model endpoint ends with v1. And then here we load the model, and instead of that, it's not Quen anymore. We should probably change it to Gemma. So if we change that to Gemma 4, that is the correct version that we should be changing into. But that's all I wanted to show you there before a model deploys, and we're going to cut straight to once the model is deployed. It's deployed. As you can see here, it's live on this indicator, and I've run a few just to make sure that it works. And now let's go to our app. This is the Streamlit app I was talking about. So this is a test I ran, and here on the app it saves the conversation so I can come back and resume it or create a new chat. But if I delete that, that just also goes away. So I'm going to up the max tokens here to the max that it can have. I'm going to leave the temperature— maybe actually let's bring it to 5. So that basically just says, you know, I want you to hallucinate half of the time probably, but it's a temperature thing. So I'm So it's not exactly that. It's more like how creative do I want the model to be? So here I'm going to go ahead and ask it a very basic question. Like, like just let's start with hi first. Hi, what model are you? For example. So it's going to take a moment there to send the request. So long as the model there is fresh, it's still live. So let's see if that solves it. Yeah. So I'm a large language model trained by Google. What is your name? And let's see if it gets that right or even if it tries to guess it. So there's this little bit of ghosting there. I don't have a personal name. Great. So let's just tell it to do something interesting here. So can you explain the concept of a drafter as it relates to inference and making models faster as if you're explaining it to a caveman? So let's run that. Let's see. Actually, can you explain speculative decoding to me as if I am a caveman? There you go. So that's much better. So you can tell like it's doing all this pretty fast. The big thing here is we went ahead and deployed a model and we're serving it through an API that is only unique to us and we're controlling everything. So from costs to the metrics and everything we can track. So this is right here. It explained it. That's an interesting concept. So this is basically it in a nutshell. I don't want to go too long, but the idea here was to Deploy our own endpoint to an API that we personally control, and then actually build something with that endpoint. And you can think of this as this is purely yours. Nobody saves the data for you. So a lot of enterprises are trying to go this route. So this is kind of the future that I'm thinking of with open source. And open source is very important for this specific reasons that we've already talked about. So if you enjoyed this, feel free to subscribe and yeah, goodbye for now.