Modal Office Hours

What's in Eli5 about Modal?

Yeah, it's a great question. So the Eli5 with Modal is that it is, let me start with EliPhD. It's remote procedure calling in Python that has, that doesn't suck. So, there's code running on another, there's like code or an environment on another machine and you invoke it from some other Python script. This is an idea that goes back to the '90s with like Java Beans, Enterprise Java Beans, where like cloud remote procedure calling way before it was cool. But the LiFi version of that is that it's like, rather than managing servers on the cloud, you just hand over like code, like scripts, like things that are closer to actually the work that you wanna do to a cloud provider and they run it for you. So that's what Modal does. We do that for Python functions and we're focused on data-intensive Python workloads right now. So that's like things that require a lot of computation or have like a really large data input, data output, or environment definition. So like they have 80 gigabytes of model weights in them, or you're sending in an HD video or you're producing an HD video. Yeah, and a lot of the existing technologies for doing this, AWS Lambda, Google Cloud Functions, don't fit that particular type of workload well. They're designed for web services. So that's what Modal is. So that's maybe ELI, I dropped it down from ELI PhD to ELI 15. Maybe ELI 5 is, Yeah, Python, Modal lets you run your Python in the cloud.

And I guess I'll add just a little bit, because I think there's a lot of magical stuff here going on, like that, the magic in a good way. So basically, like, the thing that, there's like two things that are really magical to me. Um, and there's a lot, you know, is like one is like you develop locally and then remotely your stuff just appears.

Yeah.

It's hard to like describe it. You have to like show, show it to get it. Like there's no, like what you have to experience it basically. Like one of my favorite onboarding tutorials for Modal is to make a web application where you just locally change the code. And then in real time, it's like hot reloading the actual production thing without, without going through like a deployment process or anything. You just like keep iterating on it. And like, this is really fun. For example, when you're like, okay, if you have like a webhook, if you have like some web server that's receiving a payload, if you've ever done that before, it's very painful, right? Like you have to constantly like debug that. You're like, oh, the payload actually contains like this schema has this extra field here. Oh shit, like I need to, I need to resend that payload and then like redeploy my application every single time. And it takes like hours to do something simple. With Modal, it's like, okay, I can just have a live running endpoint and I'm getting these payloads. I'm like, oh, it's a bug. Oh, let me just switch my code. Oh, it's a bug. Let me just fix the code. I don't have to like redeploy anything, do anything. This is hot fixing it right there. It sounds like— now the way I described it, it sounds like that would be a bad idea, but actually the way they've made it is like solid. It's like, yeah, like you have to— and, um, maybe, uh, we can show something today. Uh, we will be scheduling a workshop also with, uh, with Charles so you can show more. I haven't scheduled that yet. We're gonna do that today. But might get into it today. I just also wanted to see what the questions were so we could like form the workshop more.

I think I've used Modal less than Hamel and obviously less than Charles. Let me, maybe I'm in the midpoint where it's easy-ish for me to give the Eli Five and then you guys correct me if I've said something wrong. The, the most basic way of running Modal is you write a function, you write it as if you were going to run it locally. You can add a decorator and say, oh, actually I want this to run on 4 A100s on Modal. And even though you're just running it locally, actually the thing your local computer does is just send a request to Modal and Modal does the running. Um, and that, uh, is a different model from most ways that you run stuff in the cloud where there's an explicit step outside your code where you've written your code and then separately you say like, actually I need to move this code to my EC2 instance, then on the EC2 instance I'm gonna run it. And so it's sort of, it's just a way of having your code specify where you want it to run.

Yeah, yeah, it's like infrastructure alongside code. One more thing, one more thing.

One more thing that I— the second favorite thing is like the scale of it. Like, so there's things like this, really simple things to understand. Like, there's like a mapper, for example. You can take like any arbitrary thing and just map it out in the cloud to like arbitrary amounts of like GPUs to like parallel things and like massive, like just massively in parallel, like so easy with like barely any adding any lines of code. Uh, so yeah, anyway, sorry, I didn't mean to cut you off. I just want to get my favorite parts in.

Yeah. Yeah. So, but I, like, yeah, the high-level thing here is that it's like, um, yeah, the co-founders, Eric Bernhardson was at Spotify, built Discover Weekly, and then like worked at fintech company Better, and like kept seeing the same infrastructure problems being solved over and over again, and kind of poorly by internal teams, and was like, what would we need to do to like solve that in a more robust way that's like productizable and sort of like get all these infrastructure problems out of the way of like data scientists, ML researchers, model builders, and like, and do it in a way where they get back that like super fast, tight development loop that they want. So yeah, there's, yeah, some good writing from Eric on like the importance of tight development loops and tight feedback loops that it's like relevant even outside of development. It's like if you're building anything, you want these like super fast feedback cycles.

Ankur Singh asks, which I think is a natural follow-up and something that you guys do especially well. I am going to give you a hard question at some point, but this is just like Lambda, only one concern, startup time. Any comments? So I think that you guys have worked a lot on this. What do you have to say?

Yeah, so, right, so depending on what you're trying to do, the like startup times on Modal can be very fast. So the like core, like Docker run kind of experience, like just starting up a container, just that part is like, that can, that's like tens of seconds to minutes with something, with, if you were running like Docker on a, a normal cloud is like 1 to 2 seconds with Modal. That's the like P50, we aim for our P50, our 50th percentile startup time to be in the like 1 to 2 seconds. So where things can get slow is if there's a bunch of work that needs to be done to set up your environment that can't be written to disk or that like writing it to disk doesn't speed that up. So because what that means is like in order to get your like function ready to run, in order to set up the environment for your function, You need to have stuff like living in memory. So the most important example of that for this class is like language model weights. If you want to be able to like start something up in a hun— like somebody sends a web request and immediately a language model, like, you know, 10 milliseconds later has sent them a token, you have to have model weights in VRAM, like already ready to start like processing those inputs. You can't like take them from disk to the CPU RAM and then to VRAM, which is what happens when you like start up a VLLM server or you do your like auto causal LLM in Hugging Face, whatever. So the solution with Modal is that you would end up, if you needed that, you would like turn on keep warm and so leave the application running all the time ready to go. So then you would have to like, with GPUs, like there's no way to kind of slice up a GPU, so you get the whole thing all the time. And so you end up like having to pay to have a GPU server up all the time if you wanna be able to do those like fast, like super fast, you know, millisecond type startup times. So that like, that increases cost. We're working on CUDA checkpointing. So this is like GPU checkpointing feature from NVIDIA just came out about a month ago and we're like integrating it into our stack that will help speed up like subsequent invocations in a lot of cases. The other, the other side of this is that if you have CPU-intensive work, it's like your data preprocessing, your ETL workloads, communicating with the database, for that, it's like pretty easy to slice and dice CPU RAM and CPU time. So if you leave that on and warm on Modal, so long as you aren't handling a ton of requests, like you're maybe not requesting that much time on the CPU, like your process is running and every like, what, like 100 milliseconds, the operating system's gonna be like, hey, do you have any work to do? And your process will say no and we'll say cool and we won't charge you anything 'cause you won't use the CPU. And same thing with memory. Like if you aren't handling a ton of requests with a FastAPI server, then the amount of space you take up in RAM is very small. So I actually run a lot of little web servers on Modal in keep warm mode that are like all just CPU work and they cost me like 30 cents a month, but they have like very fast like responsiveness. And then finally, I will say the like LLM fine-tuning repo has not been tuned to try and speed up those boots. We've designed it to try and be like a little bit easier to adapt, adjust, and like understand. If you were to go in and sort of like really work on speeding up a bunch of the cold boot times, you could cut that down substantially. Like, I think there's some low-hanging fruit for like a factor of 2 at least.

Well, do you want to take this? The— I don't know if you're looking at the questions. I can read one to you.

Yeah, go ahead.

Yeah, top one right now. Curious, what is the recommended approach to locally develop a full fastai application with auth database integrate interaction, et cetera. So you'd have VS Code with debugging, and then you have a local database, et cetera. And then you want to be able to deploy it to Modal.

Yeah, that's a great question. So if you look at a lot of the Modal examples, so anything, any of the guides in our documentation or the LLM fine-tuning repo, we usually take what I call a thin client approach to using Modal. The client that talks to Modal, the code that talks to Modal, all it is is like which is like bare Python standard library with Modal installed. So that brings along FastAPI. And so like you can do some FastAPI development with it, but it doesn't bring any of the additional dependencies that you might have if you start adding extensions to FastAPI. It doesn't have, you know, TensorFlow or PyTorch or whatever you've installed. And so that thin client, you can't do much local development inside of that thin client. And so that means like it's hard to get like autocomplete, type stubs, et cetera. And so the like flip the, I think we've tried to make it as easy as possible to still get a really good development workflow with a like in a thin client architecture where all of the environment definition is in Modal. I like that a lot because it's like I can install a Python library just for talking to the New York Times API. So that I don't have to read any of their kind of bad documentation and can just use that library, keep it isolated in a single container, and then like it doesn't interfere with any of the other components of my application. And that, yeah, so we have things like you can shell into a container, you can like execute commands inside of a container, like you can't, and like poke around and live debug the way you could if it was a like running server. So like if, you know, in that thin client architecture, that's what you have. It's like get inside of the server and develop it, develop inside it. You can also kind of like launch a VS Code instance or a JupyterLab, like inside of the same environment that you're running on Modal. We don't have as many good, like solid examples of how to do that in a complex application. So that's a non-answer to Wade's question, which is telling you how you could get a good development experience in the cloud. The answer for local development is that you could do, like, you could build instead a thick client architecture where you do have, like, the local Python environment that's the same as the environment in the, like, service that you're running on Modal. And that in that local Python environment, you're also sort of, like, managing dependencies you have Poetry or Conda or whatever, and you're installing all of these things, like you're installing your connection, your like, PsychoPG, or your like, ORM, and you're doing, and like, you install that locally, you like, share the same definition of environment between Modal and your local environment. You use a Dockerfile, you use like, requirements.txt files, you point at those, and then, you have your local development and you, and you have, you have actual true local development where you, and, and you set that up and then Modal is your like next tier of development outside of that tightest, fastest loop for debugging, like, you know, like very raw stuff. We don't have a ton of examples of how to do that just 'cause like it requires a lot of more effort to set up that kind of, and like environment management to set up that kind of thick client. We do have an awesome modal GitHub repo just made like a week ago that has actual, a lot more like actual production examples. And the, and so some of those have more of a thick client architecture and you can look at that for, like for inspiration of how to build that. So that's awesome-modal on GitHub on the Modal Labs organization.

Also, you all should really try Modal because, uh, you know, I'll say like Charles is like crazy responsive, like probably one of the most responsive people in the whole Discord that I've seen. I don't know how he's doing it, but he's like basically responding to every question that is there. So it's like a really missed opportunity if you don't.

Yeah, I try to answer every question that comes in. If it's more complicated, I'll probably direct folks to the Modal Slack where there are a bunch of people who are all extremely responsive. I think it's like part of the culture of the organization coming down from Eric and Akshat to be like extremely responsive to user and customer questions. And I think like overall, like I'm very excited about a future of open source models with customization via fine tuning. Like that's one of the reasons why I got into Modal in the first place. It was like, this is exactly the tool that is required to allow the sort of like broad open source developer community to kind of like, you know, keep up with proprietary model development and like create useful applications. It was like what unlocked actual— like I was suffering through trying to make these kinds of applications myself and like Modal massively accelerated the pace at which I could do this work. And I want to accelerate that work for all of you. Um, so that's why, like, that's why I'm like in the Discord all the time answering questions, and, uh, that's why Modal as an organization is like very excited about this class.

Pretty awesome. Uh, I got another question from Wade, uh, very similar. Okay, so before I was running an app, here he asked, ideally I'd like to fine-tune locally with just a small dataset and a small model. Do debugging there, do it all working in VS Code, and then once I'm satisfied, I want to run this on Modal with a full dataset and bigger models. Do you have any recommended, he says, setup or even like best practices for that iterative?

Yeah, check out, I mean, it's like the, the thing that you're going to run into is where you are now get like with Modal, maybe you're running into this issue where it's like the boots are slower, it's harder to do state management the same way that you can do locally. and so you feel that it's slowing you down, would you switch over to having like a local environment that's different from your, like different from your production environment? You'll find that like, um, like test production skew, like difference between your testing and development environment and your production environment is now the pain point and the problem. So I would take a look at the, uh, the image, the modal.image class, which is how we define container environments. And the documentation on storing model weights in Modal. We have a guide on that for the tips and tricks that help with that. And in particular with the images, you're gonna wanna look at sort of the, a lot of our examples show the Pythonic SDK for defining environments where it's all right there in the same Python file. If you're gonna build a local environment, you wanna make sure those things match as tightly as possible. So you're gonna want to use things more like from requirements file, you might want to use like pip freeze and poetry to get like a tighter connection between your local environment and your remote environment. I think the biggest one is like, I doubt that your local GPU is the same as the GPU that you're gonna run on Modal servers. Like we have A100s and H100s, and if you're running a like bigger model and a large dataset, you're gonna want that. And that's gonna be your biggest, like, your biggest skew. So it's gonna be like, watch carefully about how you are defining and building that environment for differences between development and production.

I just wanna say something also on this about the Axolotl question. I think generally speaking, I would suggest people actually do some work working with Axolotl, not in modal, just on a VM, just to see the bare metal of like Axolotl, just to get some intuition about how it works. And then jumping into Modal, like then you can really like wield the power of Modal. It'll be less, you won't be as confused if something happens, you'll know where to look basically. I think it can be useful for a lot of people. Like you can automate it the way you want. Like you will be able to say, hey, like, you know, I always want this option on. And I always want to do things this way or that way. And you can kind of wrap it even more. You can kind of like wrap around Axolotl in ways that work for you.

Um, so is there— I don't know if there— PyTorch Lightning had like a CPU-only like flag that you could turn on, and I remember really liking that for being able to like, you know, knock around on my local machine. I feel like that's like, um, You won't get everything. You can't run DeepSpeed fake on your CPU, but you can do a lot with a small BERT model on a small dataset so that you don't have to wait all this time and you can use your actual local machine. That would be my recommendation here.

Oh yeah. Yeah, I mean, I haven't tried running TinyLLaMA locally, but it is one of those small models in the examples, like the really like really small models.

Yeah.

They might want to be even smaller than that. But yeah, just know the plumbing works and whatever. It's like a good idea to do that first.

Yeah, I want to answer a question from Alex from ZenML. So is Modal mainly for GPU-accelerated workloads? Or if there's a bunch of parallel compute that's needed, is that also like something you can use Modal for? Yes, you should absolutely use Modal for CPU, right? CPU intensive and especially parallelizable CPU intensive tasks. Basically, it's very cheap to use CPUs on Modal right now. There's a couple of reasons for that. One is like, as I said earlier, it's like easy for us to like slice and dice CPUs. They come from this like time-sharing mainframe world, unlike GPUs, which come from the workstation world. And so they're like designed. For like multiple people to be using them at the same time. And there's like a thick stack from the hardware to the high, to like the high-level software to make that possible. And then on top of that, when we get ahold of a GPU instance to run work, the GPU instances come with really beefy CPUs. They're kind of configured for something like a training workload and also even like a computer vision training workload because that's just like the heritage of the instances available from the cloud providers. What that means is there's like a huge pile of like Threadrippers or really, really nice like Intel server-grade CPUs with a ton of cores and a lot of RAM attached to the GPUs. But for inference workloads, for the kinds, like for what a lot of people are doing with inference these days, they actually don't need that much CPU power. They don't, and like they aren't like processing one batch with a bunch of like, cropping and rotations and random transformations on the CPU while the GPU is crunching through one the way people used to when training computer vision models. Because of that, we have a bunch of like CPUs, like kind of just vibing, waiting for work to land on them. And so like we love to like get a hold of additional CPU work. So we like, we're paying for those. We'd love to have people use them. And so there's, it's actually, there's like a lot of stuff that's actually just easier with CPUs because there isn't as much hardware complexity. And there isn't, and like the tooling is all so much better for CPUs than GPUs. So like, yeah, we have some examples that like, that show parallelization of just CPU tasks. In fact, even the Blender rendering example shows that you can get very high throughput by going with more CPUs instead of fewer GPUs. And so it ends up being like a throughput latency cost kind of like complex trade-off rather than just GPUs go brrr, it's fast. Um, so yeah, highly recommend trying out Modal for your like CPU compute workloads as well.

Uh, this one from Shubham, I think, is also a good question because I actually had an experience this morning, uh, on another platform that weighs in favor of Modal. So Shubham says, it seems that fine-tuning a model with the same, same model, same dataset is more expensive on Modal compared to say Jarvis Labs, can you talk more about costs? And after you, I might add something as well, but yeah, talk about that.

Yeah, sure. So yeah, so Modal, like we don't run our own cloud just yet, you know, like we don't have our own data centers, but so we're grabbing compute from all of the, like from multiple clouds, AWS, OCI, GCP. And so like that sets like the floor of what we can charge you, you know, like there are some people in Silicon Valley who will, you know, sell you a dollar for 50 cents, but like, that's not the way that we run things. So that sets a floor on what we can charge you. So if somebody, if like, you know, it would be cheaper to run, like, the floor of price is gonna be like what the cloud providers give you. There's some caveats to that, but that's a good starting intuition. So the win with Modal is when— is a couple of things. One, if you— to actually materialize the decreased cost of something like, you know, of a cloud provider that gives you VMs or servers, you have to spin up the server, do your work on it, and then spin the server back down. And I don't know about you, but I have certainly left an EC2 instance or two running slightly longer than I needed it a couple of times. And it's like pretty easy for those bills to rack up. So the, like, the fundamental thing is like, what is your utilization rate of what you are paying for? So if you are just like, you, you like spin a server up and you're right there actively working on it on a, a server-based cloud provider, and then as soon as it's done, you spin it right back down, you got like 100% utilization, right? You were using the hardware you were paying for the entire time. And like a serverless compute provider cannot, if you can achieve 100% utilization on servers, like go for it. Don't use a serverless provider. We are not the right workload. We are not designed to like to run your workloads. What Modal is designed for is where the like, is two things. One, the operational and engineering time required to like spin something up, run it, spin it back down, like if that cost starts to outweigh the actual computing cost, then like you might wanna switch to something like a serverless provider that like abstracts away that infrastructural complexity. So that's one argument. The other argument in favor is like, you take that fine-tuning job, great, you ran it, you got good results. Now you wanna do hyperparameter tuning, right? You wanna run like 1,000 copies of that. You aren't gonna be able to spin, automatically spin up and spin back down 1,000 instances without incurring a significant additional engineering complexity. And if you are like doing that, you know, if you aren't in control of the workload, like when it comes to inference time, then you maybe even can't predict what your utilization is gonna be. And then finally, getting ahold of GPUs is not just a matter of like swiping a credit card. So if you want 1,000 A10Gs for 10 minutes, like no one's gonna take your call. And it's— and so Modal, because we have a relationship with cloud providers, we have some pre-allocations, etc., like we have them ready and you can, you can have them for like 10 minutes, an hour, up to 24 hours. And so like you can get access to a maximum possible scale that is both operationally and like maybe like literally for economic reasons, like, like more difficult to access. Um, and so those are the, like, those are the places where you start to have a win with Modal. Um, yeah. And then I think the last thing I would say is like, I would, like, I've, uh, somebody on Twitter said I would still use Modal if it was 10 times more expensive than AWS because of the developer experience. Um, and I think there's an argument for that, but I prefer to rely on the like more solid, like, uh, kind of like economics and, um, infrastructural arguments.

Yeah, and just to add on, uh, this experience is fresh for me this morning. So we had a consulting client, um, we were running something on another— on RunPod, and they did a fine-tuning job and they wanted to know like how much it costs. And so they had like a little spreadsheet and it was like the 3 parts of the job were downloading the data, training the model, and then like there was just some dead time where people walked away from their computer in between. And so, uh, and like, so the waiting time, it doesn't need to be this way. If you were sitting at your computer every minute waiting for something to finish running, um, you wouldn't have this time. But in our case, you know, roughly a third or a little bit, you know, 40% of the time was just dead time in between steps where there's actual computing going on. And right, the model's a little bit more per hour, but then you end up using fewer hours because you're only paying for when the compute is actually happening.

Yeah. And actually there's a State of AI Infrastructure Alliance report— or sorry, State of AI Infrastructure Report from the AI Infrastructure Alliance that says that large organizations tend to achieve only about 50 or 60% utilization of their, of their GPU allocation, even at peak usage. So even at peak usage, if you do the math, it's like roughly even on cost to use Modal versus get your own stuff. And with Modal, like, you aren't doing commitments to cloud providers and you don't have to build all the Kubernetes infrastructure yourself. So yeah, that's the, the State of AI Infrastructure report. Um, I also would like to answer this, uh, question from Christopher Lee. Um, could you go over again how you have a FastAPI app running with KeepWarm at 30 cents a month? I'd love to host my FastAPI app there, but unless I'm missing something, keeping a CPU core warm at, uh, 20 cents an hour amounts to $138 a month. So here's the thing, CPUs are designed to be time sliced, right? Like, this is why when you run like time.sleep, if you look at the documentation, it's like Oh, it'll last about a second. And you're like, why can't I get like 1 second out of my computer? I thought computers were precise. The operating systems that almost everybody uses, the like, uh, Unix or Windows, are, are time-slicing operating systems. Like the ticking of the clock, um, at like the ticking of the CPU clock and the like time that your process experiences are separated from each other. The operating system hands out cycles to different processes. And so what Modal charges you for is actually using time slices, actually using the CPU. And so you aren't gonna be like, you aren't gonna be using an entire core all the time. If you've ever looked at the utilization of a web application, it's like very low a lot of the time, even when there's traffic. And when there's no traffic, it drops very low and you end up only paying for RAM. Um, and so the, like, that's what pushes this cost, um, like far down. It's like, oh, you're, you're, you're actually paying for like 1/100th of a CPU core, which is like $1.38 based off of this math.

It's amazing to think about, um, Yeah, apparently how cheap polling to figure out whether there's work is.

Yeah, uh, like when I use AWS, there isn't a good— I mean, there's not good tools about like having automatic spin-down stuff, you know, like you have to manage all that yourself. Like you have to, you have to have like tools that spin down your EC2, but you know, because they— there's no incentive for them to do that. It drives me insane.

Yeah, it's the same reason why they ban crypto mining, you know, like it's like if everybody could just mine crypto on AWS, then it would be like, oh yeah, anytime I'm not running other stuff, just like go ahead and mine me some Dogecoin, you know? And then like AWS actually like, like, you know, they got to power the whole data center and they have expected utilization rates and that sets how much power they need for that data center. Um, and like, they have engineering tolerance tolerances, but if everybody was running at 100% CPU all the time, like, that would actually substantially increase their costs and possibly even like basically brick their data centers. Um, and so they're like substantially disincentivized from, uh, like getting 100% utilization.

Maybe their margins are really just like substantially tied to underutilization.

Yeah, there's also some good, uh, posts from Eric about this, like the economics of serverless versus the economics of, uh, VM-oriented clouds. And there's also a paper from folks at Berkeley, the, uh, "Cloud Programming Simplified: The Berkeley View of Serverless." Um, and that paper also has some good stuff about, like, that incentive structure where, like, yeah, we're encouraged to try and, like, uh, like, make good use of resources. We being Modal.

Wade is asking, okay, is there a good, uh, example that shows how to stream output from LLM versus a FastAPI app?

I think via a FastAPI app.

As well? You might show something like that in the serving lesson?

Yeah, that's, yeah, I'll talk about that then. We, yeah, so if you go to our docs, there are a bunch of LLM examples. Yeah, though, yeah, we have, like, with examples, you have this last mile problem of, like, you have, it's like, I want to run, like, like, spin the wheel. I want to run Llama 3 code version on VLLM in OpenAI-compatible mode on a SANic web server. And it's like, okay, we don't have all of those things together in one example. But yeah, so we do have a good number of streaming examples. The Mixtral example should show streaming. I guess there is also streaming in the inference example in the LLM fine-tuning. It's not with FastAPI, but if you look around and you look at some of our FastAPI examples, especially if you poke around the modal examples GitHub, you'll see some examples of doing streaming responses with FastAPI. So you gotta kinda like glue those two pieces of code together, but I think the bones are all there. And Modal does make async programming in Python a little bit easier 'cause we like bring our own event loop. So you can just like async on all your, like you can just put async in front of def on all your functions that are Modal decorated and it should like just work. So that should make it slightly easier to do this kind of streaming. But yeah.

Uh, all right, I got a, a question. I'm gonna— it's not in the question list. I've always had like a— but I'm— I think of everyone at Modal as being much better software engineers than I am, so I trust your judgment on this, but it always felt to me like my code, like without Modal, I can treat compute as some commodity and I can just move my code from one place to the next, to the next without changing it. And it's almost like a separation of concerns of where does it run versus what does it do. And then the thing that's always made me a little uneasy about using Modal is that I have to break that separation. And now if I ever wanted to run it somewhere else, maybe removing decorators isn't so hard, but like I have to make some changes. Like, does this feel like a reasonable thing to be concerned about, or like, how do you feel about that abstract— changing that abstraction?

No, I totally, I totally understand that. I think, I think it is possible to write modal, like, code on modal that separates those concerns. I think it becomes challenging to write code that is both like super high performance, low cost, and separates those concerns with Modal. But that's also kind of true about just like software in general. It's very difficult to write extremely portable code that satisfies like high performance at low cost and low resource utilization. So in some ways, that's like, that's like maybe an unsolvable, like pick-two kind of triangle situation. But yeah, I would definitely agree, like if you just jump into it at the start, it is a little bit surprising to, like, be writing code that's so tightly tied to its hardware and its deployment model. That said, Modal Code is just Python code, so I think, like, you, like, you can just ignore the decorators and call the functions, I'm pretty sure, or, like, yeah, like, programmatically reach underneath them and call the wrapped function. So yeah, I think that there are a good number of escape hatches.

Yeah.

Here's one I hope the answer is easy.

Yeah.

What is the privacy policy of Model Labs? Will you use your data to train any models?

Yeah, that's a great question. So we are SOC 2 compliant. Like, that's just a common, That's like a baseline sort of like, are you good stewards of data kind of certification. We're working on SOC 2 Type 2, hopefully by the end of the year. The, yeah, so then, but not like HIPAA compliance is substantially harder. And yeah, there's like a bunch of others, ISO, yeah. Yeah, but then for the like narrower question about like, are we using your data to train models, like your application data, that's like sacrosanct. We ask permission before we review your application data. You'll see that if you ask for help on our Slack, you may have seen me asking for permission to review logs. So like that application data is privileged. There's no intent to train models on user data. I think the point, like ironically for what workloads are run on Modal, Modal is not that much of like an ML or AI company internally, like, um, it's like an infrastructure company. So it's not like we're, we're sitting around being like, let's build a docs chatbot. One of our users built a docs chatbot for us and we use it in our Slack. Um, but like, we don't do a ton of that kind of work, like, um, for internal development. So I would be, I, you know, I don't want to like commit the company to like, we would never train a model because who knows what 10 years from now, maybe that literally every software company is training bespoke models on user data. That would be a great outcome for the members of the LLM fine-tuning class who would all be experts at that particular activity. But yeah, but the short answer is we do not train on user data. And we take the privacy and security of your applications extremely seriously.

This is a really good question that I'm interested in. It says, so Alexis is asking, is looking to spin up a key-value store on Modal for development purposes. And he wants to like spin up a web endpoint to maintain a hosted LevelDB to read and update the keys. Is that a good idea on Modal? I know that, you know, should you have an external service? What do you recommend?

We have built-in key-value store stuff. We're sort of like, like I think of it as like we're kind of like slowly exposing features of Redis to people as like as like Pythonic objects in the Modal SDK. Um, so like modal.dict is a key value store that all your modal functions have access to. Um, and then, so like that would be, that would be the like blessed way to do this. And then actually modal.queue is actually— Is that persistent?

Does that last like when the run ends and how does that—

Oh yeah, yeah. There are ephemeral versions if you just need communication, like, like within some, like narrow run, just like there are like ephemeral modal apps, um, which is like what you're getting when you do modal run or modal serve, the thing Hamel was talking about with the fast reloading. Um, but the, um, yeah, but the, by default, the dicts are persistent. There are some limits and stuff there, and like right now it's free, and like if people abuse that and start storing like many gigabytes of stuff in it, like we would have to you know, turn the party music off. Um, but like, there is a distributed key-value store for like many purposes. It's like pretty performant. Like, I want to say I was trying to do some like, um, like a Hadoop-y type thing with the Dicks as queues, uh, as like communication between, um, modal functions. And I want to say like 50 milliseconds to like, like, uh, like a step or something. Uh, yeah, yeah, like, um, you remember that, uh, Unix shell tools can be 2,000 times faster than your Hadoop pipeline blog post? This is like a famous blog post that showed that like running MapReduce at anything other than the like terabyte or petabyte scale was a bad idea in like 2007, 2010. Um, and they showed how you could do it with Linux pipe tools. Anyway, I was using Linux piping tools where instead of using a unit like a Linux buffer to pipe between them, I was using ModalDicts. And the answer is it's way slower than doing it locally on a, like Unix buffers are like in memory and yeah, this was like, so it was closer to the speed of the Hadoop example there. But the, but like that's not what the dicts are for, they're for being like a distributed key-value store. And there's also queues, which is a distributed key-value queue system, also a like Redis feature. For like, Modal's not designed for hosting databases, it's designed for hosting things that talk to databases. So I would recommend for like the, like if you're using like LevelsDB, RocksDB, one of these analytic databases, I would recommend like, yeah, if you need a hosted instance of that, storing it elsewhere. And then also I think with a lot of those, I feel like you can get away with using like Parquet or Arrow files stored in S3 that then in Modal, you just like pull those down, you like mount a cloud bucket to a, to a Modal function. And then you just like rip through your analytic queries on the modal worker itself. So we do have an example of doing that in the documentation. Look for, I think it's using the New York taxi dataset. So there's like, yeah, examples of showing that as, yeah, mounting S3 files and using, I think that's DuckDB, we use DuckDB, but like kind of same lineage, maybe even literal actual software lineage. To like Monadb there. But yeah, um, like, yeah, analytic queries, um, that, yeah, and that would be the architecture I would probably recommend as opposed to running a hosted instance. But yeah, all right. Uh, anyway, so Modal's for a lot more than just LLM inference and fine-tuning. I think we've got a great product for, for fine-tuning and inference, but yeah, if you want to run if you wanna run your own private Snowflake, you know, give us a try.

It's like really good for cron jobs too. Like all these little cron jobs that you have in your life, like you wanna scrape some website or check something or whatever. I have all kinds of stuff like that.

Yeah. I don't know that everybody has a bunch of little cron jobs in their life, Hamel, but like I get the spirit. Yeah, it's fun. It's fun.

Uh, we got one from Boris. What's a good balance? And I think you've sort of alluded to this. What's a good balance in terms of cost where if I need 100% uptime, um, and I want to do GPU inference, um, GPU inference could be expensive to keep that GPU warm, but if I need maybe 20% uptime with a few seconds of keep warm, then it might be cheaper. So like, is there some middle ground there?

Yeah. Yeah. I would say, yeah, if you're, you, the like rough barometer that I use, if your utilization is like 60% or lower, then, um, like Modal Labs is gonna save you money. Um, so, uh, then like you, you, depending on the application, you might need to do some more tuning to make sure that cold starts do not cause a bad, like, user experience, like if you have to wait, you know, 30 seconds the first time a request comes in, that's like generally gonna cause users to churn. But if you're, it's a long-running job, that's not the case. So like batch jobs, you have much, if you have a cron job that like, you know, fine-tunes an embedding model on Wikipedia every night, like that's a very low overall total uptime, like total utilization. And a few seconds delay isn't so bad. But yeah, so like a 2:1 ratio is where it starts to go from like modal is cheaper to modal is more expensive. Yeah, if you can achieve 100% utilization, then yeah, go for VMs. You're gonna be spending a lot of engineering effort to achieve that, but if that's where your differentiated value is, if that's where your skills are, then VMs are right for you.

I interpreted the question potentially incorrectly, but if you imagine I'm gonna call some OpenWeights model, one thing I could do is store nothing. And every time I want to run inference with that model, I'd actually pull the weights from Hugging Face and then that would take however long it takes. And then after I pull them onto that machine, now I put them on the GPU. Another thing that would be cheap but slow. Another thing I could do is have them the way it's stored in the GPU, and now I'm just paying for that GPU all the time. Like, yes, how does Modal handle all that middle ground where I want the data like not— I don't want to totally reserve the GPU, but I want it to be able to boot up semi-quickly?

Yeah, so, um, so there's like kind of 3 levels here. One is definitely like You don't need to download the weights every single time. You want to move, like, you want to move weights either into a modal volume like we do in the LM fine-tuning repo, or you want to, um, like, put the weights directly into the image build step. So then when you deploy for the first time, the weights have to be downloaded and that takes like a couple minutes, but then you like never need to down— like, if you're just using the same model every single time, you would never need to like download the weights again. So like that definitely, that cuts like several minutes off of boot times. Then there's like, yeah, there's latency to get weights onto the machine that actually uses them. So like they're stored in our like cloud storage where we keep all of our images and then they need to get onto the machine that uses them. That's like, that's gonna give you maybe for a 70 billion parameter model, I feel like I've seen that in the like 20s of seconds. Where like 20, 30 seconds latency. Yeah, you can, yeah, for, and this is so for pure cold starts, like we, you have nothing running, we're not like booting up extra workers and like scaling up on top of your existing like pool of workers. Yeah, you have that and then you need to like get things into RAM like get the weights from disk into CPU RAM and then GPU RAM. So that's gonna add another couple seconds of latency. There are, there's like a CPU checkpointing feature that can speed up some of that, especially if you have like, like Axolotl does a lot of, well, and like, yeah, importing Torch does a lot of work. That's like a second of latency that you can shave off by using CPU memory checkpointing. So look up the memory snapshot feature. That doesn't yet work for GPU snapshotting, but that's, yeah, that speeds up, like, after you've cold booted many times, we start to figure out what your application looks like in memory right before it starts processing inputs. And we, like, keep that where it's, like, faster to just, like, flash that into RAM. Than to like redo the computations that put it there. So that can speed up like getting the weights into RAM and yeah, but you will still have like some latency there.

This is a really good question. Okay, Kit's asking, during development, not prod, assuming you're not totally GPU poor, say you have 4090 or something like that, How do you intuitively split workloads with Modal? Like, how do you find that sweet spot between cost and time? I'm always tempted to run everything locally and only use the cloud whenever it's too big, but I feel— I'm wondering if that's the wrong approach. Um, you know, when should I split things up? Should I split things up more different— like, differently? What are your intuitions, etc.?

Yeah. Yeah, 24GB of VRAM is, is like great for inference with the current state of things. It's pretty rough for fine-tunes and it's kind of rough for evals. So let me break that down. Like with inference, your batch size, especially if you're like doing your own personal inference for your own local Llama, like your batch sizes are pretty small. So like you can't take advantage of big VRAM to improve throughput by pushing a large batch through. A small model. Um, so like for inference, it makes a ton of sense to do local development. I like, like to do some, some stuff on my MacBook because it's a pretty solid local Llama machine. Um, evals are in a little bit in between because when you do an eval, ideally your eval set should be nice and beefy. Like your eval set should be getting bigger over time with more and more examples that help you build confidence that you're like that your change to your flow engineering or your change to your model or your new fine-tune is like better. And so then you actually do have batches and now you can substantially increase throughput and so decrease the time it takes for you to find the answer for your eval by running on bigger, beefier hardware. So with, yeah, with 8GB, and like you could even, you know, run several eval sets at the same time even if it fit, even if all of them fit within the 24GB of VRAM that you have on your local 4090, or 4090s have more than 24GB, but if I'm remembering correctly, but anyway, if it, you know, whatever it fits on your local machine, you could just like split many eval sets at the same time on like multiple cloud GPUs. And so, Yeah, and like we have features like map, like modal.map to just like transparently take a function that runs on like one input and run it on many inputs. And then finally, with fine-tunes, as you may have experienced, like you incur a ton of additional RAM pressure when you are doing backpropagation, when you're training models. You have, you like, you need the gradients, which are the same size as the weights. They're actually larger than the weights these days 'cause they require higher precision in order to avoid floating point nonsense. There's a good discussion in the Discord about this. And then you also need, like, if you're using Adam or one of these, like, other approximate second-order optimizers, you actually need, like, basically multiple copies of the data that is in, that is the size of the gradient. So you need your, like, running average gradient and your running, like, average gradient variance. And so that's a huge, like, massive additional multiplier. And then of course, fine-tunes are batch-oriented. Like, you probably shouldn't be doing gradient descent with batch size 1 on a fine-tune. And so you have this huge extra VRAM pressure. So there are all these tricks and techniques out there that can squish things down into fitting inside of a 3090. And it's awesome that there are people like Tim Detmers and Jeremy Howard and team working on squishing those things down, but it's gonna be harder to run those workloads, the software. It's like, it's just hard to run things in 4-bit and to do on-the-fly quantization and all these things, to do sharding. And so you're always going to have an easier time if you take those workloads and run them on something a little bit comfier, like an 8xa100 instance. And so like, it's a matter of like, if you feel like getting cracked at like running QLoRa at the, like, at the absolute lowest VRAM possible with CPU offloading, et cetera, et cetera, is something that like you enjoy and will be like, you want to become an expert at that for the use cases so that you can get hired to a job where those, that's the use cases, then go for it. But I think like outside of that narrow skill development and like climbing Mount Everest because it's hard kind of attitude, I think it's With Finetunes, it's like pretty clear case for, uh, for data center cloud workloads.

I want to say one less technical thing about this. Um, so Google BigQuery used to be available for— or there's a version of it that was available for internal use. And there was a period of time when, when you ran a query, if you were a Google employee internally, they didn't want people to overuse it. And so it said the cost of this query to Google is, and it said, and right before you ran it, it would say like, and then people would be like, ah, maybe I just saved the company money and like not run this query 'cause it's gonna cost $4. And so they changed the way that that works and said that running this query will cost the same amount as, and then they denoted it in how much time of a software engineer's salary it would take. And then you'd be like, oh, I could save this query or like, maybe I could do something clever on the backend if I like spend an hour or two on it. And before you're like, maybe I'll save $4, but now you're like, and it's going to save Google the equivalent of like 2 minutes of my salary. And when they made that change, they made that change from denominating in dollars to denominating it in your time because people were being inefficient at optimizing things instead of just using the compute. Um, it is my, like, quite strongly held view that when it comes to using cheaper GPUs, people are undervaluing their time. And if you can get something done in 5 minutes instead of 50 minutes— instead of 45 minutes— and it costs an extra dollar or two, like, for most of you, your time is so valuable. And hopefully soon someone will be like if you want to get paid a bunch for your time, hopefully you will be compensated well enough that pretty soon you're like, oh, I can avoid the cost of an A100. It's actually like economically terrible for your employer if it means you work more slowly.

Definitely, yeah.

Um, we're close enough on time. You want to take a couple others?

Uh, yeah, just let me, let me answer some quick ones, just like rip through these. So, Nahil Jain, are there some good cookie cutters for modal projects? You can search modal cookie cutter. I think there's a couple floating on GitHub. Would love a cookie cutter for modal projects. Would love to have, like, the, like, npm— npx create-app, you know, experience for modal. So yeah, please DM me if you're interested in working on that project. I'd love that. I just don't have the bandwidth to work on that. There's a bunch of, like, quicker reps that I need to work on with examples and guides. Does Modal support a form of auto scaling? I keep warm one CPU, then scale up as requests come in. Absolutely, yes. So you like, keep warm is just an optimization to reduce like your tail latencies, your like 90th percentile latencies or whatever when the first request comes in. We do auto scaling. You have some control over that auto scaling behavior. You can check out the docs. For details on that. You can set how many inputs can be like queued on one worker at a time. Wei Jie Li, can I download the Docker image built when running Modal? No, that's not an option right now. You can build a Docker image and send it to us. So if you want control over the Docker image, that's the direction. But really kind of Modal's position is like, we want to abstract that away from you. So we don't offer it as like a release. Um, would you recommend to serve inferences on modal in production for your clients? TensorRT, TGI, VLLM. To Dan's point, uh, I think software engineer time is, is for a good reason more expensive than GPUs. So VLLM, very fast to get up and running, like, and rapid development, and they keep getting faster. Every time I download a new version of VLLM, there's like a performance increase. TensorRT does still beat them out on benchmarks. I see varied numbers for myself and for others from 10% to 2x. So you might have to benchmark it and decide, but really VLM, just like you get up and going so much faster. I feel like it's a problem to replace that with TensorRT once you're further along. TGI, I haven't necessarily seen a great use case for it, but between those two, it doesn't have anything that differentiates it from either. And the open source licensing dust-up that they had, I think has reduced the community development on TGI, so it's just Hugging Face. So that's, I don't know, that's my opinion. I have less informed opinions about TGI. What would be the best examples to work through beforehand to get the most out of your demo? Try the Hello World example on Modal, and then, just because that's the raw ideas, and then also the trt-lm example. Just because that like really good— there's like some fun neat details about LLMs and GPUs. Um, okay. Uh, then, all right, I actually gotta run. I have a, uh, meeting after this one. But yeah, I think we're getting a hold of these questions and, um, I can, uh, answer them offline in, uh, Discord.

I'll extract them and put it— drop a CSV with the questions. This was awesome. Thanks.

Yeah, yeah. Thanks a lot for having me, Dan.

All right.

You got it. Bye. Bye.