Modal: Simple Scalable Serverless Services

Yeah, thanks for coming, everyone. I wanted to walk through a deeper dive on Modal. So, and to talk about sort of like the broad vision of what Modal can be used for, focusing less on like fine-tuning LLMs in particular. And the sort of vision with Modal is to allow people to deploy scalable services and in a cost-efficient, serverless manner, and to do all that in a way that is simple and easy. If you want to check out, by the way, the slides for this are available at the link in that QR code. Great. I'm going to break down each of the pieces of that alliterative title. And what they mean to me and what I think they can mean to you. The goal is to create scalable services. What do computing services need to do? There's 3 things that you're going to need to do with a service. First, maybe is what's all the way on the right here, input/output, connecting your service to the world and the world to information. This means that rather than having a Jupyter Notebook that just runs on your machine, or a script that runs if somebody happens to, is able to download the Docker container image and run it. You have something where people can connect via the internet or some other network to your service and put information into it or get information out of it. You need that input/output. That's maybe the most important part. Uh, but what's on the other side of that input/output is the what happens to that information inside of the system. So one of the most important things that happens to information is just storing it, holding on to it from the place it was created and then using it somewhere else. So you need to store or preserve information using something like a database or storing files somewhere. Uh, and then finally, you want to do something usually with that information, even if it's a simple as make the storage of that information easy, speedy and fast. So it's stored on files, but it's accessible almost as fast as if it were in memory. So you need to compute and manipulate that information. So the collection of these things together are what we use to define the services that we make available to people. So I've been talking about services from more of a databases perspective as I've been describing these things. Which is like the way, you know, something like Amazon works. You click a button to indicate your interest in purchasing an item that's stored somewhere. The related items are computed and then displayed back to the user. But the, like, that service could also be LLM inference. You put in a description of the code that you want, you get out the code that you want. So it's like a fairly generic set of problems that need to be solved. And the— what, like, one of the big challenges with doing, like, something like this is doing it in a scalable manner, is setting it up so that those connections that are coming in can come from all around the world or can come from, like, hundreds or thousands or millions of people at once, that that storage can scale up to gigabytes, petabytes, or maybe even more in size, and that that compute can scale, that you aren't limited to just what can be done by a single machine, but what can be done by many machines. And so scalability, like when defining a service, is increasingly becoming table stakes. People expect to be able to hit your service from all around the world on day one, or people expect —people expect to be able to share something you've posted publicly on a social media network and draw 100,000 of their closest friends to your service for a day. Going down when that happens is something of a rite of passage, but it's also a bit embarrassing. Defining this in a way that it can scale is critical. The way that people figured out to make these things scale is to distribute them, to spread them across many machines, possibly in many different data centers. And so this is like an important, one of the most important ways that applications and services are developed as like distributed services in the cloud. The problem with that is that it's really, really hard. And so one of the goals of Modal is to give you the ability to define these scalable services in a simple way, in the comfy velour chair that is Python development. Modal has Pythonic versions of all of these things that you need in order to build your scalable service. We have a very simple and Pythonic way to define web endpoints or web servers, so to just wrap any arbitrary web server that you want and turn it into something that is like, you know, that can be distributed, scaled up with, you know, no YAML files and only the, you know, the configuration you need inside of your code for defining that service. For storage, we have, I think, in the end, distributed databases, like managed database services are kind of the way a lot of people build these things. We don't yet really have a database primitive, but there's other storage that you need to do. One is caching or communication between— once you scale something out, now all of a sudden you've got hundreds of computers involved in delivering your service. You can't just store stuff on a disk or store stuff in memory and share it between processes. You need a distributed version of storage. We have dictionaries and queues that are you know, on our service and natively distributed. And then you also can't like write things to the file system at one point and expect them to be available at another one. It's a little bit naughty to do that when you're building a web server anyway, but that used to kind of fly and work, but that's increasingly untenable. And so there's like, Modal has volumes for like creating something that looks like a file system but can be accessed from many different many different workers. And then lastly, and maybe most prominently, Modal has a bunch of facilities for defining compute. So the scalable units of work or things you might want to do, so a Python function is the unit of work that we offer and that you can say, okay, this is the Python function that I wanna run. When somebody hits this endpoint or when I'm running my ETL job, here's a Python function that I wanna run. And yeah, so let me dive in a little bit deeper on each one of these as we go. So gonna have some screenshots from the Modal dashboard and kind of walk you through them. So for example, you might have a model inference function. So this sort of encapsulates all the work that needs to be done to do inference. Then as people hit your service and consume your service, that function gets called. That's this function call graph up here at the top of this diagram. Then as that happens, you'll scale up and down and consume different amounts of resources. I think this isn't maybe a great example for scaling up and down. You can see that it's consuming different amounts of CPU resources at different times. When there's lots of requests at once, the CPU works harder, the amount of memory being used increases. This one seems like it fit on a single GPU and we never needed to scale up from one GPU to more than one. But we have all those resources available for this function when it's called and we can bring more to bear. If a bunch of calls come in at once and the function can't service them as quickly as they're coming in, then another function gets scaled up. This one, I think with a lot of model inference functions, you define them such a way that they could take in lots of inputs, more than one input at once. That's why this one has many inputs coming in, but only one function available. Then on the bottom here, another way that functions get used is as these really easy cron jobs. Something that runs regularly on a period, In this case, this is for a little tiny retro display called a tidbit in the modal office. I was showing a poem written by Mixdrill every hour. So 45 minutes past the hour, so 15 minutes before the hour, I would spin up Mixdrill, write a poem, save it somewhere, and then display it on the display for the next hour. And both pushing the poem to the display and writing the poem was done via these cron jobs, these like regularly running periodic functions. So those are great for like pulling information from a production database into a data warehouse or running a regular analysis on data that's in a data warehouse or regularly fine-tuning and retraining a model on a, on a certain cadence as more data comes in from production or rerunning your evals with live new user data to check for or Rift in production. All these things are well supported by some kind of like regular execution cadence. I'm gonna quickly check the Q&A section to see if there's some questions. Oh yeah, Wade had an interesting question from a couple of minutes ago. Is a database service coming? Would love to deploy a serverless Postgres alongside a serverless API app. I would say that's not something that we have planned immediately. In particular, serverless Postgres is more challenging. There's two types of databases that people use these days, OLTP transaction processing databases and OLAP analytics processing databases. It's relatively straightforward and we have some examples of how to run analytic workloads. On modals. So that's like your database is on a— is in a big collection of Parquet files somewhere. You run analytics on it regularly. Maybe you do it with one of these cron jobs to pull down these Parquet files that you're writing to S3, run some analysis on it. So that is like— it's— there's not like a modal primitive for that, but we do have some examples of how to use your favorite, like, DuckDB or maybe we don't have a Polar's example, but like how to use those. I think we have a DuckDB pulling Parquet files or Arrow files from S3 example that shows you how to do that. For transaction processing, the workloads look very different and it's much more challenging to scale out transaction processing. It's much harder to build a distributed database that operates on this like row-by-row level where you're like, join 7 rows, like 7 tables together to pick out this particular row that represents the user whose shopping cart I'm updating and then update that row. That is, it's like doable to do distributed databases, but it's much more challenging. And so for that, we very much lean on other services for hosting. So that would be something like Neon. Would be a great serverless Postgres option. There's another one and I can't believe I'm blanking on it. What's the other really good serverless Postgres? Supabase, right? I've actually used Supabase more than Neon, but Supabase and Neon both have— I think Supabase has scale-to-zero semantics and pricing, which we'll talk about in a second for what serverless means. But in both cases, they work as a great external service for giving you these Postgres, the most popular trend or the most popular open source longstanding transactional database as something you can talk to from your serverless API app. I've deployed that on Modal many times and had a great time with it. Great. Yeah, great comment from Prasad as well. Okay, I'll answer some of the other questions are bigger and I'll answer them like, you know, as we go through the talk, but please keep them coming and I'll make sure that they all get answered live. Great. So in like computing is fun and good, but computing is not super useful without storage. Like data is what ends up defining the value of a lot of like applications these days. And so Modal has lots of features for storing data. So I'm going to focus more on the long-term storage side rather than the dictionary and queue stuff. If there are questions or interest in job queues and dictionaries in Modal, please post in the Q&A and I'll review it. But the part that's more interesting or more important with people who are doing LLM fine-tuning and ML inference and ML training workloads are these like file system abstractions like volumes, which you can use to store weights and datasets. So these are just like some of the volumes that I have stored on Modal. Like I have those model weights are I think from Axolotl fine-tuning runs, but there's some kind of training runs that I've run. Yeah, you can see I've got CIFAR-10 data. I think that's actually CIFAR-10 data and models in a single little volume or file system. Wikipedia, like a raw Wikipedia dataset in, in from Hugging Face datasets as Arrow files, if I remember correctly for that one. So like reasonably sized dataset, maybe in the low gigabytes. And we have some examples to show you how to store really, really big terabyte scale maybe even low petabyte scale datasets on Modal. Those are in our examples recently added, actually since the course started, some examples of how to store really large datasets. So these are— the important thing about this is that it's a form of storage that looks to your Python code like it's local files, but is in fact distributed, you know, like a a distributed file system, like a nice robust cloud-native file system. The primary catch or the design choice that we made with the volumes is that they're designed for writing a very small number of times but reading a very large number of times, or WORM, write once read many workloads. That's pretty common for datasets. You don't like repeatedly overwrite the same dataset element over and over and over again. And with model weights, like you don't overwrite the same model weight version over and over again. You might like repeatedly write new weights, but you aren't overwriting the same weights. And the nice thing about that is it's much easier to scale out many readers than it is to scale many writers. And so these volumes work really well for those kinds of workloads. That's one reason why it can be easier to run these like analytical, analytic databases on, with Modal as the like fully the backing like cloud platform for it because they frequently have write-once-read-many workloads where you write like, you dump like a big file and then you read it from like many workers to run in parallel as opposed to like writing from many different places. If you're running like distributed Redis or distributed Postgres, you've got like writes coming in from multiple write replicas. Let me check Q&A, see if anybody was— if people were thirsting. People were thirsting for discussion of dictionaries and queues. They're not, but I do see a good relevant question from Philip Thomas. Will you guys charge for storage? I can't see pricing on it now. We will eventually have to charge for storage, I think. And the goal is to charge for storage at like roughly the price of S3, which is the price that we're basically getting it at. So we do things like look around to find like storage primitives that are available that are like S3 compatible that allow us to like, you know, run this service more cheaply. But there is like a limit and storage is not free. So I think, yeah, but for now we aren't charging for it and you can store lots of stuff there. I think if you were to start sending us petabytes of data every day, we might have to have a conversation about what your plans are with that data. But yeah, great question. Other good questions, but not anything that I see on storage. Keep going forward. And then, so last but certainly not least, there's input and output to your compute and storage, the ability to interface with this stuff. So I think the most blessed way to use that in Modal is with FastAPI apps. So I have the Swagger docs for a little FastAPI app that's like a fake version of Twitter there on the right-hand side. So with the, like, FastAPI is really nice. You get asynchronous concurrent Python without as, you know, as much of the agonizing pain of writing, like, async/await. And it's got great documentation. And it's really great to do that with Modal in particular because async, like, sort of plays more nicely often with kind of scaling out and running lots of different workers and gets you more juice out of any individual worker. And anything else with async that's important on our platform? Oh yeah, and it's also like Modal's designed to be able to run sort of like weird compositions of synchronous and asynchronous functions without throwing some of the weird error messages you might be used to seeing if you run async. So like you write just FastAPI code, which is like, like very Pythonic and clean. And then you don't have to worry about running your own event loop. We've got like, you know, we run your own event loop that we keep any of our code sort of like out of. So you get good performance with it and then you can, but if you don't care about event loops or you're not thinking about it, you just run synchronous functions and nothing goes poorly and you can just pretty much just start flapping asyncs around in your modal code, at least your modal decorated code without needing to refactor your app. At least you might need to refactor it to get performance, but that's the way async works. You won't get— yeah, we have some nice async features for avoiding them coroutines not being called, if you're a concurrent Python person, that sounds exciting to you. We have web endpoints. FastAPI is an example of an asynchronous server gateway interface, ASGI, web framework. We have some nice features that are FastAPI-centric, FastAPI is a dependency of Modal, so it comes along for the ride. And like when you install it, you don't need to think about managing another dependency. And then we have web endpoints, which just take a Python function and turn it into a URL you can hit. And that's for now based off of FastAPI. But really what makes FastAPI like kind of tick underneath is this asynchronous server gateway interface protocol for defining a web service as something that interacts with other things. And so you can— any ASGI app can be run on Modal using the ASGI app decorator. And so that's one level up of additional flexibility and compli— but with some complexity. Parallel to that, the other framework for serving for like web Python is WSGI, W-S-G-I. The most famous or popular WSGI Python framework is Flask. If you've ever made a Flask app, you've made a WSGI app. WSGI Flask, it's a joke that almost nobody gets, I think. WSGI is based off of no asynchrony, no concurrent Python, and that the tooling there is older and more mature. I think Django is all sync and has a WSGI component. I haven't looked at that part of Django in a minute. You might find more batteries-included projects and you have this thick stack that covers everything. With WSGI, but, and you don't have to worry about all the stuff I was just talking about, about async and await and goroutines, but you might leave some performance on the table and you aren't, and some of the hottest newest libraries with cool new features are sort of more focused on the ASGI stuff. But you aren't, so ASGI, WSGI, these are like server gateway interfaces. It's like a way to define a web service generically. We also just let you run any web server you want. And just like, it can be— it doesn't even have to be Python. There were people running like Go servers and stuff. And that is our like final sort of exit point of like, oh, I just wanna run any old web server. I'm gonna call it with subprocess so it's as though I'm like calling it from the command line with Bash. And I just want Modal to handle like scaling this thing up and down and keeping it alive and HTTPS and all that stuff. Yeah, some comments on this one. Let me go through these. Yeah, for web endpoints, is there any mechanism to prevent something like DDoS attack mobile.com by default? We expect something like that in the future. Exposing WebPoints on public website and giving you H100 GPUs, you can get people coming in and trying to use them without bringing you any benefit. We don't have any protection like that built in. I think that is a good idea and something we could offer in the future. I'll definitely bring that up as a potential product feature. I think right now it's like you can wrap authentication if you're using FastAPI or Flask, there's middleware for authentication. Really in the end, preventing DDoS attacks, like there are ways to prevent like people, like suspicious traffic patterns that can help with this. But in the end, it like, it ends up being, you know, putting things behind a login. Like YouTube and Twitter are contemplating Twitter's already made it that you have to log in to read stuff. YouTube's contemplating it, and that's the only way to prevent people from accessing your API without you about it if it's out there. Yeah, but something like Cloudflare DDoS protection is a cool idea and definitely worth exploring. There's a question about WebSocket interactions with the max execution time. I haven't used the WebSocket stuff very much, so I'd recommend bringing that up in the modal Slack. You can probably get some good answers on that. Let's see. Point of clarification from Hugh Brown, Django has support for writing async views along with an entirely async-enabled request stack if you're running under ASGI. Interesting. Thanks for the callout, Hugh, on that. I haven't tried async Django. And then there was a point of clarification on storage stuff. Some other questions came in. Petabyte-sized dataset living within your store? I actually, I don't, I think that one might be backed by S3, like raw S3 rather than modals like volumes that are on top of cloud storage. And yeah, another question about petabytes is datasets. Does current pricing or lack of it also apply to data transport costs? Yeah, we currently aren't charging on ingress/egress. If that starts to be a large cost for us because people are doing big, like, you know, doing huge jobs for it, we'll have to implement pricing. But again, the goal is to stay, you know, just above the raw storage that we're being charged for. How does a mount differ from a volume? Is mount meant for local dirs? Yeah, I kind of skimmed, just breezed right past this one. Mount is take something that you have on your local computer and make it available to the stuff running on Modal. That's like we mount your Python file in order to be able to run it, for example, every time. But you can also mount your own stuff like, oh, I need this file for my static site, I need to mount my assets. That's what mounts are for. Great. Okay. So there's some cool stuff that I wanna make sure to get to and yeah, but great questions. And if I don't answer your question here, please do like post it in the Discord or in the Modal Slack and I'll make sure it gets answered. Okay. And the last piece about these scalable services that we've kind of dancing around a little bit is that there's serverless, simple scalable serverless services. I can't say it 5 times fast. This is an important part of what makes Modal, even though it has a higher sticker price, often an economical choice and not just developer experience and ergonomics-based choice for teams. The key thing here is that if you run any service, you'll find that your resource utilization is variable over time. You might have slow traffic patterns of like, oh, people in particular time zones use my service in particular ways and cause resource utilization to increase. You might have big spikes like when things end up on the front page of Hacker News or whatever. The classic way to handle this is to provision ahead of time for whatever you think your peak usage is going to need. The pink bar there is how much resources you provision. If you're like not doing cloud development and you're doing like on-prem, um, or like, yeah, co-located or running your own data centers, you like have no choice but to provision for peaks, um, uh, ahead of time. And the bad news is that resources generally cost money as a function of time, uh, and so if you've like provisioned resources up for the peak and then you aren't using it at peak the whole time, you're going to be paying for it. You have to get really clever with resource utilization like Amazon did. We're like, wow, we need all these computers for Christmas. We don't need them the rest of the time. Might as well invent cloud computing to make use of that. But not everybody has that ability to or has something else they could do with their resources off peak. Hamel Husain, I saw you palm off camera. You got a hot take to drop?

No, I was just getting ready for questions, so I thought this is a good time to— Yeah. All right.

Don't get distracted by me, sorry. Great. You never know when Hamel's about to drop a truth banger on you. Then the other thing people do, if you are not buying resources, you can manually provision things on the cloud. When you start to see your resource utilization go up, You say, oh, well, let's spin up some more stuff. Then when you see it go down, you spin things down. And then you have an oh shit moment when you hit the front page of Hacker News and you suddenly need to spin up like 100 copies of your service. And then you wait for a while to like see whether the traffic spike is really gone before spinning back down. So you can like, this works well to reduce your like excess cost. As you can see, the like area under the curve area between the two curves has gone way down when we switch over to manual provisioning. But we're also starting to get a lot more area above the curve here where we're missing some stuff, like we didn't have enough resources and that usually leads to poor user experience. This is also very stressful on teams. You can do automatic provisioning and auto-scaling and one of the primary tools for doing this is like Kubernetes has auto scaling in it and will auto scale up containerized services on clouds. That can get your resource utilization to match, often slightly lagging, just like with the manual provision, there's a lag. You can see this time lag of the pink curve relative to the green one. But computers are generally faster than people and can be programmed to be nearly as good as them at resource utilization, and so you tend to track it more closely. But with autoscaling, it will be easier for you to match this curve, the smaller your units of autoscaling are. When you run the— yeah, what's achieved by this is that your costs match your actual resource utilization. If your resource utilization that you are— what's the way to put this? If your resources scale to match your needs and they can also scale all the way down to zero, that combination is what people call serverless. The usual way that that's achieved is by packaging things up as something much smaller than a whole virtual machine, but just as a function. So it's also called Functions as a Service, though that term has kind of fallen out of favor in favor of serverless. And so the nice thing about this is you are like generally, like you're paying a lot less for the resources that you need and delivering a superior user experience. So what makes this possible. Like, why? Like, you can do this yourself. There's a lot of engineering effort required. So that's one reason. It's like, does it make sense to expend this engineering effort? But infrastructure is important. And like, it's like, why would I go to somebody like Modal to do this auto scaling for me and run these resources for me? This is like, like, infrastructure is an important part of my business. Why, why would I let them do it? Um, the key idea is that, oops, This thing should not be there. The, like, if you look at the resource utilization of a serverless platform like Modal, we have many people's, like, resource utilization profiles superimposed on top of each other. So our baseline is way higher. We don't have to worry about going all the way down to zero, which can be especially painful. And those fluctuations that are big for any individual, like, consumer of the platform are small relative to the entire platform. Those fluctuations that were really challenging and speedy for an underlying user become much less. The engineering complexity is much less and is amortized over a much larger workload. There are things that can be engineered at that scale and with those timescales that can't be done at the lower level. That's the argument put forward about the economics of serverless computing in this Berkeley View paper from Eric Jonas and others. It's a great paper, changed a lot of the way that I think about cloud computing and computing in general. All these Berkeley View papers that have the little Campanile on the front are bangers. Ian Stoica and David Patterson are on most of them. And they're all really incredible. Great. Okay. So this is set to end at 10:15, right? Hamel?

Yeah, it's at the end, 10:15. So we have about 7 minutes or so. Got it.

Okay. I do want to show one thing. I will stick around for questions. For another 15 minutes at least after the set time. But there is one more point I wanted to get to, which is the core idea with what's going on with Functions as a Service, I think in general, but certainly on Modal, is an old idea being revisited, remote procedure calling. The general idea with remote procedure calling is that your local code calls a function, and instead of that function being binary code that lives somewhere else on your machine or some other process or something, it lives on a totally different machine. The goal of remote procedure calling is trying to make that as transparent as possible, but no more transparent. That's the idea with Modal and that's basically like we use gRPC as our framework for doing that. It can feel very confusing and weird because Modal otherwise makes it feel so much like local Python coding when you run into this piece of the system. So it's worth walking through in this deeper dive. The idea is your code is on your machine, that's the script you're writing, and then you might ask to run a modal function. If that modal function is defined inside your script, when you do modal run, we're grabbing that function, pushing it up to our machine so that it's ready to be run from your local code dynamically when it's needed. That's the source of why does Modal have all this stuff? Why do I need to be careful about the global scope? Why do I need to put imports inside functions? It's so that we can do this magical remote procedure calling of a function that's defined inside the same file. But the core idea is just to get that code onto our machines where we're running the Python environment that you asked us to run, and have that code available so we can run it on a GPU even though your local machine doesn't have a GPU. We run your code on our machines for that. I wanted to show this quick demo I built of this in only a couple of hundred lines of code, but I don't think we'll have time to do that. Minimodal shows that core piece of it as 200 lines of code that runs just in a separate Python process locally, but shows this, the basic structure of having an app and using decorators to send code somewhere else to run. So if you're interested in, some people have asked some questions about the guts, how does this work? That's all at minimodal on my GitHub. If you're interested in that and want to dive deeper, please hit me up. Okay.

[Speaker] So you created your own Modal, like kind of to kind of understand how it works in a way? [Speaker] Yeah, yeah.

Local only, local only, very dumb version of Modal. And like the only thing it does is sort of like separate out virtual environments. So that's like almost useful to like be able to call code from multiple Python virtual environments.

Yeah. What is this symbol here?

Oh, that's my dumb version of the modal logo that I made. I asked ChatGPT to make an SVG of two cubes. And I knew it would get it wrong. And it made this delightful bad modal logo. Yeah. All right. So, I want to make sure to answer some of the questions that came in because there were a lot of, like, Really good ones. Yeah, okay, great question from Kurian at a high level. Yeah, let me get to the end there. All right, so how is Modal hosting Suno.ai? And it's like, Kurian says my favorite GenAI application for the last 3 to 4 months. Yeah, so there's a blog post, where Suno talks a little bit about why they chose to use Modal. The details are kind of interesting. Like, they're making heavy use of Modal's platform. Like, there's things like we have all these remote storage things like dictionaries and queues. So that's used to, like, manage work. And yeah, but fundamentally, like, yeah, they're using functions, prongs, volumes for storage, yeah, setting things up as web endpoints, yeah, using all of the stuff that we talked about.

We need to leave at least a minute or so for the transition between, we've learned, so in between different sessions, 'cause like when it's back to back.

Oh, right. Yeah, there's another session immediately after this one.

Yeah, yeah, there's another session from your favorite company immediately after.

Right. Oh yeah, Replicate has their office hours. Okay, so I can't stick around here in the Zoom. Let me jump into the Discord voice chat. Yeah, totally.

You can totally do that. You're an admin, so you can effectively— I can like create one.

Yeah. Cool. All right, I'm gonna like, quickly screenshot everybody's questions, lo-fi style, and drop into the voice chat. Great. All right. Well, yeah. Thanks, everyone. Thanks for great questions. Hope to be able to answer them. If I didn't answer it and you can't make it to the voice chat, please, like, post it in the channel on Modal. Wasn't able to watch the Discord, so I'll also answer any questions that came in live there. Hope you enjoy the platform and you know where to find me on Discord and on Twitter, Charles_Irl. Look forward to seeing what you build. Use your $1,000. Yeah, use your $1,000. The other $500, if you didn't get it last week, will be coming out today, midnight UTC, más o menos. So make sure to use Modal by then. All right. Take care, y'all. All right.

Thanks.