End-to-end cloud compute for AI/ML

Foreign.

Welcome to Practical AI. If you work in artificial intelligence, aspire to, or are curious how AI related technologies are changing the world, this is the show for you. Thank you to our partners at Fastly for shipping all of our pods super fast to wherever you listen. Check them out@fastly.com and to our friends at Fly. Deploy your app servers and database close to your users. No ops required. Learn more@fly IO.

Welcome to another episode of Practical AI. This is Daniel Whitenack. I'm a data scientist with SIL International and I'm joined as always by my co host Chris Benson, who is a tech strategist at Lockheed Martin. How you doing, Chris?

I'm doing very well. How are you today, Daniel?

I'm actually doing amazing. So I'm not in my normal location. I'm down in Orlando, Florida. So one thing is it's sunny outside and I can be outside without suffering. But also while I'm in person, meetings here with some of our collaborators and partners and they wanted me to do a demo today. So I got up early this morning at 6am before hotel breakfast and I threw together a quick demo and I used Modal for that. And there's literally someone that stood up out of their seat and clapped after the demo. So our guest today is Eric Bernardson with Modal. And so basically Eric is making me look good in all respects and I'm pretty excited to talk more about Modal and share it with everyone today. Welcome, Eric.

Hi. Hi. Thanks for having me. I'm excited to talk about Moodle or anything else.

Yeah, yeah, I think. Chris, do you remember like quite a while ago, I don't remember when this was. Maybe Eric, you remember. I think you wrote a blog post about building data teams or something like that. I forget exactly what it was, but I remember Chris and I talking about it on the podcast. I'll have to see if I can find it back in your blog, but.

Yeah, that was in the summer of 2021. I wrote the blog post.

Yeah, yeah. So we should have had you on the show then, but I'm glad that we get to have you on the show now. You describe Modal as an end to end stack for Cloud Compute. So I guess one big question maybe to start things out is Cloud Compute isn't new, but it definitely can be complicated depending on what you're trying to do. Like what got you starting to think about the set of problems that you're addressing with Modal? Like what, what got you going down this path?

Yeah, it's kind Of a longer story maybe, but I've been working with data for 15 years or maybe more most of my career. Initially I was at Spotify for seven years. I built the music recommendation system there and open sourced a vector database called Anoi and workflow scheduler called Luigi but did like kind of everything from like deep learning to like business intelligence to large scale big data type like Hadoop stuff. And then I was just at a company called Better for six years. Managed data teams but also managed other teams. And so as I was thinking about starting a company I kept coming back to data and my starting point was really just like it's hard to work with data and I feel like data teams don't have the tools they need. And initially I was super agnostic as to like what to build. I kind of frankly wanted to rebuild everything, which is not particularly realistic, maybe in a lifetime aspirational megalomaniac maybe. But what I realized was that at least if you want to rethink a lot of the data stack, a good place to start is at the bottom. So I almost sometimes joke that I'm kind of grudgingly had to start spite startup and doing this work now at the most lowest level which is to you know, solve the compute problem of like I have code and I want to deploy it in the cloud and like why is that so hard? I don't know. But you know, it's a big problem for many data teams is you know, just a problem of like you know, taking code and scaling it out or scheduling it or running it on GPUs or you know, setting up web endpoints or whatever it is and really focusing on that problem as like, you know, building this core foundational layer that's very abstract and very general purpose. So that's also why our website, I think it's a little bit confusing the first time but in particular what I would say we'd be focusing on the last six months is online inference. So it's a lot of machine learning AI models focusing on that use case as a sort of initial starting point. But modal always to me had this promise of running almost anything. It's almost like a kubernetes in the cloud.

Yeah. And one of the interesting things to me I think where like maybe it took me a second for this to like sink in, but once it did it like was a really encouraging thing for me was like I have my code locally and I know how to run it locally but then like you have this sort of concept of like these decorators within Python code that kind of take your code and like, you run it, like Python, Python, main py, whatever. But actually something like the moment I realized, okay, well, now this function is actually not running locally. Like, I just did some sort of like, batch inference or something with this script and didn't, like, you know, my fans aren't going on my laptop because this is actually running somewhere else. Could you describe, like, I mean, there's a lot of ways that you could have gone about this sort of lower level of the problems that data teams face. There's a really fundamental piece of this, which is like the local to cloud or local to deployment cycle. And with modal, that seems very, very quick. How did you zero in on that kind of workflow?

We built something that architecture looks something like AWS lambda, right? It's like function as a service. We take code and execute in a serverless way in the cloud. The starting point, the reason why I ended up going down this rabbit hole of doing that really, this whole serverless runtime is really kind of thinking about developer productivity and developer happiness. My philosophical observation as CTO for many years is that developer productivity, I think, is very often well understood in terms of feedback loops. And so in particular, there's a sort of. As you write code, there's almost like a nested set of. For loops. It is the innermost loop of you write some code, then syntax error, then you fix it, and then you run it. Maybe you have some unit tests, but then there's these outer loops that are often like, okay, let's deploy this to the cloud, or let's run this on a massive data set. And that's when the iteration speed gets very, very slow. And so you look at data teams, they're often particularly exposed to these feedback loops because they have to run on large data sets or they always have to run things in production. You can't really run things on synthetic data as a data team. You have to kind of deploy it into production or run it on a real data set. And so it really frustrated a lot of data teams. It's like that sort of like very slow iteration speed. It's like I write some code, now I have to create a container, push it to the cloud, then go and click on that interface, or like, merge some pull requests or whatever. Then my container fails. Now I have to go look at logs or whatever. So I started to think about, what if we bring the infrastructure into that innermost loop? Like the loop of like, okay, you just write code and then you immediately run it, but it actually runs in the cloud. And in order to do that, we realized we can't do this with Kubernetes, we can't do this using Lambda. We basically have to build our own infrastructure that takes code and can launch containers, maybe hundreds of containers in the cloud in a few seconds. So we went very deep down that rabbit hole and built basically our own container runtime, our own file system, our own container builder. Luckily, I'm not afraid to go deep and solve tricky container problems in dealing with Linux and file systems. But that's like a lot of what we had to build in the last two years is that foundational level, like that runtime. But the benefit is now we have this super nice developer experience. We can just take code locally. You can spawn 100 containers in the cloud in a few seconds, running the latest code in the latest container.

Sounds fascinating. I'm really interested in it, but I want to ask you to step back for a second with a follow up and bridge a gap of understanding for me. You were saying, can't do it with Kubernetes, can't do it with AWS Lambda, and I believe you, but I don't know why, and I'm imagining that maybe a few of our listeners don't know why either. Could you kind of tell us what it is like? Because a lot of them, their companies are in one of those big three providers and to kind of show them, you kind of demonstrated with the user experience quite well a moment ago. But could you talk a little bit about like what was falling down in those kind of more mainstream big three kind of approach, Google, AWS and Azure, so that we can understand that because you made a statement, I'm with you on that, but just bridging it.

Yeah. First of all, I'm like the world's biggest AWS fan, right? We run everything on aws. I love it for the capabilities it brings me as a developer to run things at scale. Developer experience in AWS has never been particularly good and so I've been banging my head for years against AWS documentation and in the end I usually figure it out. But it's a pretty jarring experience. I think in particular the problem with both Kubernetes and AWS or like Lambda EC2, et cetera, that we saw either for users to use it directly or for us to build on top of that is just the iteration speed. For instance, in Kubernetes, let's say you want to run something in Kubernetes in production, going from Code locally. Well, now you have to first build a container, then you have to do some sort of docker push. There's a registry, right? Then you have to kick off a Kubernetes job. Then you have to go and look at the logs of that Kubernetes job. And by the way, kicking off a Kubernetes job that often entails the Kubelet worker pulling down the Docker image. And so we were looking under the hood and trying to understand how Docker works. And Docker, it's an amazing piece of technology for the new way of thinking that it brings to the table around insulated containers. But it's quite inefficient in starting containers. Like, you know, most containers end up having lots of data that's never actually read. Like there's like, you know, thousands of like time zone files of like, you know, local information about, you know, time zones in Uzbekistan or whatever. Like you're never going to read those, right?

Unless you're in Uzbekistan. Sorry, just getting that in there.

Yeah, on this, you know, whatever, you know, or uninhabited islands, like there's like time zone information about uninhabited island in like the standard Linux distribution. Like, okay, like great, but like get them out of my Docker container. But the other thing is also Docker is quite inefficient and like it has this like layer thing. But like other than that, you know, it doesn't really deduplicate information. And so what we realized is that what if we like rethink how those containers get pushed and pulled and we end up building our own file system. We do de duplicate the content by computing a checksum of every file. That's actually sort of similar to how Lambda works. But Lambda is also not fast enough in the sense that like if you publish a new Lambda, it still takes about a minute for you to be able to run it. Lambda also has other limitations. It doesn't support GPU, doesn't support long running jobs, etc. So those are all the reasons like, well, we ended up deciding we can't build this on top of Kubernetes or Lambda or any existing solution. Also not Docker, we ended up using lower level primitives instead and building a lot of it ourselves.

And are there specific things about. And I sort of like in my own experience and using modal have experienced this, but from your perspective, I would be interested to hear like you talked about kind of moving towards this use case. Like the use cases around machine learning around AI as being kind of like very well suited to this workflow of those types of workflows have any sort of like added benefits and or challenges that may be like running web scraper or something like some other sort of use case which is related to data, but maybe not involving sort of like serialized model files and inference and GPUs. What are those things about these machine learning or AI workflows where you think either there's specific challenges that people have that are solved by this kind of quick cycle workflow versus just kind of like other data related workflows?

Yeah, I think we focused a lot on online inference recently. So basically let's say you have a model, could either be some off the shelf model from hugging face or some fine tuned model that you have yourself and you want to deploy that. And in particular if that model uses gpu, the set of vendors that support that is somewhat limited. And the other reason why is also cost. You know, traditionally if you go through the Kubernetes or EC2 route, if you want to deploy model inference endpoint, you have to spin up an instance that sits idle for most of the time. You can set up auto scaling, but auto scaling is pretty slow. So moving to serverless makes a lot of sense from a cost perspective. And so I think that's the other reason why we've seen a lot of, you know, it's not just us, there's been, I saw the banana was in the previous episode for instance, like there's a couple of other vendors that also focused on this. I think cost is driving a lot of that, the demand for serverless vendors for GPU compute specifically. I also think it's something that just came up in the last few months where a lot of people ended up realizing we're very good at training models, building custom stuff. We don't want to deal with infrastructure and running this in production. And so there's been a lot of demand for vendors like Modal where they can just take a model and publish it to modal and run it in production and not have to think about auto scaling policies and have to think about setting up web endpoints and dealing with security groups and and all that stuff. That being said, I mean modal kind of going back to its roots like we did. You know, it's not just online inference. Like we started out focusing a lot on what I think of as like embarrassingly parallel problem. Like this idea that like you have something you want to fan out and do a lot of stuff in parallel. So besides online inference, Modal also does a fair amount of batch Inference or sort of parallelizable things. Like a lot of people actually use this for web scraping. Other people also use this for things like computational biotech, large scale trance coding. You can also use this for various types of simulations or back testing, that kind of stuff. So this is a pretty wide range of things that model does well. But I think right now the user experience of online inference is like 9 out of 10. I would say at modal the user experience for batch inference and large scale Parallelism is like 8 out of 10. We're working on a lot of the other stuff like data pipelines, building more complex support for scheduling and that kind of stuff where right now it's good, but it's not quite yet where we want we think the long term potential is.

So Eric, I mentioned full disclosure to everyone in the world. I'm a huge fan of Modal and have been using it a lot and building things in it, including the side project I'm working on, Prediction Guard. And I think I just counted. I'm in the interface Now. I have 129 modal apps deployed right now.

Wow.

So I want to try to like describe from my end like it's hard because like this is an audio podcast and like talking about how things work without like showing something visual is a bit tough. But I want to do my best at trying to describe like how I would describe it and then I'd love like you to fill in the gaps or correct me if I'm wrong at any point. So if you think about running something in modal, you can write a python script like let's say app py or whatever, you can have functions in that script. And then actually one of the things I love is like dependencies is a really annoying part of particularly AI and ML workflows. So you can decorate certain functions in your code with like stub function and then define a modal stub in your code, which is essentially like referencing a container with certain dependencies in it. And then when you execute your code you say python app py and when it gets to executing that function in your code, which is decorated with the stub, it actually doesn't run it locally. It spins up a container in modal and runs that in the cloud. So you can do this either by just calling that function, or you can actually deploy then your app and have that function be accessible as like a serverless function or a web endpoint for your other applications or your other APIs to access. So I don't know if I did a Great job at describing that, Eric. That was my initial attempt. Feel free to make that more coherent.

No, I think that's exactly right. I think you touched on a couple of points of modal where we maybe think different about infrastructure than other. In particular, this guy Swix wrote a great blog post about it. It's called self provisioning Runtime. And I think that's like, to me, it's been kind of putting words to an idea that I always had around. It's sort of similar like if you have used like a service, like Pulumi for instance, or like Terraform or something like that, this sort of idea that like infrastructure as code, but like modal has always gone further than that. It's also like infrastructure and the app code. Like just put it together in the same code and have the app itself define the infrastructure it needs to run. And so with modal in code, you define the containers you need, including Python dependencies or any other binary dependencies you need. You can have different functions using different containers, calling each other just like Python functions, right? And it just provisions itself. Like you can say, this function should run on a GPU. This function should have 16 CPUs available. This other function needs 128 gigabytes of RAM. Like in code there's 0 config. In modal, there's like not a YAML file. Like there's nothing you can configure modal. Like everything is in code. And to me it all goes back to this idea of how do you make developers productive in having the fast feedback loop? I think traditionally we've had to give that up and basically make engineers run things locally in order to get the fast feedback loops they need. But then the problem is later they then still need to deploy it to the cloud, and then you have a whole set of issues that then break because the cloud is running in a different environment. And so this goes back to what I said like maybe 20 minutes ago. Like, what if you can take the infrastructure and bring it into the innermost loop of how you iterate, Then you solve this problem of having different environments because it's always running in the cloud. And it's fast enough that like, it feels, you know, some people even say modal is faster than running things locally, even though it's running in the cloud. You never ever have to think about these environment conflicts because it's always running in the exact same container at any time. And it's fast enough that you don't have to like this like, frustrating thing where you have to build containers and push them around and like, you sort of get the best of two worlds. You get the developer productivity of running things locally, but you have the full power of the cloud and all the power of containers and GPUs and whatever. Right. I don't know if that makes any sense.

Yeah. Chris and I have talked about this at certain points in the podcast. I have always really had this disdain for maintaining a whole bunch of local environments as well. I'm not a condo user. I have very minimal setup locally on my machine. And one of the things I think I kind of grasped onto is like, oh, well, I can develop now locally with modal and just import OS and import JSON and kind of normal ish things and import modal, but when I need to access transformers or Pytorch or some random other package that normalizes index scripts or something, I actually have zero concern about like setting that up locally to test, because I can just add that as a dependency in the modal function and that runs in the cloud in its own container. So I actually never even have to install that locally. Now, I could do that maybe before, like using like a local build of a Docker image or something, but that again, like you're talking about Eric has another cycle associated with it, which is also annoying.

Yeah, yeah, it's kind of annoying.

So, yeah, I love that this, like, I can just think through, like, my imports are minimal. Like, I can even run like a PI test. Right. And it's just testing as it's going to run in production. Right. Because it's running in a container in the cloud already, it's running that function. Yeah. So I. That's kind of like a lot of my love. I feel like that I've. I've enjoyed about it. What are the surprising ways that you've seen people use modal that maybe have been unlocked for users that were really either difficult for them before or like, oh, I didn't expect people to do this with modal. Have you encountered any of those things that stand out?

I mean, model inference in itself kind of was a little bit of a serendipitous thing for us. We didn't expect that people would do that in general. We thought of modal primarily initially as more of a batch workhorse, like, you know, something that helps you scale out. But we've seen a lot of traction on online inference and model deployments. And so for that reason, we're focusing a lot on improving startup performance right now. Because when you're doing online inference you have to spin up containers very quickly. You also have to load models very quickly. And especially when you're dealing with GPUs, there's a lot of overhead of copying models to GPUs, et cetera. So getting that down has been a big focus of us or ours for the last few months, I guess. Another thing I've been sort of surprised by is we enable the functionality to set up webhooks pretty easily. So in modal you can define, oh, make this function exposed to the web and give it its own URL and now you can call this URL and it triggers something in modal triggers some Python code. People started leveraging that for building full blown web apps on mobile, which I was kind of surprised by graphical UIs and all kinds of stuff like hosting whole UIs because I never anticipated that being a use case. I always thought of like, well people are going to use whatever like Vercel or Heroku maybe for something like that. But that's been sort of interesting to see that a lot of people are using that and so it's pretty promising. Maybe there's something more to be done there. I tend to think like our bread and butter is like machine learning and AI and data pipeline. So I don't want to go all in on building more like a web hosting platform. But I think there's something interesting and sort of similar along the same lines. A lot of people have been using us more for sort of job queues type things more like almost like I said, replacement to salary. The idea that they can create a modal function and then they can enqueue work for it and they never have to think about scaling or sort of deployment and productionization of salary like job queues. And that was also something we didn't really think about but a bunch of people have been telling us to actually do so that's kind of cool.

So I got a follow up question here and you've both sort of covered it to some degree already. But as the person who has not yet had a chance chance to use it, I'm really curious and I'm imagining there are a few people listening as well that are wondering could you take us Eric, like through kind of a classic workflow with modal. We've done that with, with other technologies that you may have heard on other episodes and stuff. But I'm trying to get in my mind Daniel is, is doing it all the time. But I'm, I've been left behind a little bit on this kind of just Take us through a typical AI ML workflow on modal, just verbally like what the steps are, just to kind of show us that simplicity. People probably going to be thinking about whatever they're on previously if they're on some other platform. Just as a point of comparison about how you're doing that. I'm just kind of curious if you can. Any example is fine.

Yeah, I mean we've optimized a lot for making it possible to deploy things and run things in the cloud in a few minutes. So it's actually pretty straightforward. Like in modal, you basically take any Python function. So let's say you have a Python function that maybe uses hugging face just as an example and it uses some off the shelf model for maybe stable diffusion. And so let's say you have a Python function, existing Python function that uses hugging face, it takes a prompt and it returns an image. Now you can decorate that Python function in modal with a special decorator and then annotate it and say use this image and then define an image in code using a special modal syntax. You can also give us a Docker file. But it's actually we support. Almost everyone just does it in Python internally. So in code you can say basically use Debian Slim and then install these Python packages like Transformers and Accelerate and Diffusers and a few other things, and then annotate the function to say use that image. And then that's pretty much it. Now you can run on the command line, you can do modal deploy or modal run and then it just takes that code, builds the container if it doesn't exist and runs it in the cloud. And that could typically take less than if the image is already built. It typically takes about a second to take the code locally, spawn a container in the cloud, running that code. It works for any Python function.

I mean that's dead simple right there.

Yeah, and it works for any Python function and you can run pretty much any code you want because we support FAT containers, meaning you can install Python packages, you can install ffmpeg if you want to transcode some video, you can install whatever thing you want. We have a lot of functionality for manipulating images and building dependencies and doing pretty advanced stuff as a part of that too. Pre baking models into images is something people want to do sometimes to optimize cold start performance. But yeah, getting started with modal, like we really, we really optimized for having that sort of like magic experience. The first time you try modal, like making it easy to just like you Know, install the Python package, set up a token and run code immediately in the cloud. We want that first experience to be magic and sort of set a tone for what modal is and the fact that modal, we think, is a better way to work with infrastructure in the cloud.

So one of the things I was wondering about, which I guess it was a surprise to me, I didn't really think about it when I was first using it. So I don't know, everybody has their different setup, but usually I've got my code editor over here and I've got my terminal over here on maybe another monitor or something, something. So I've got both up and I was like writing my. It was a webhook in Modal and I had it like, you know, Python app, whatever. And when it's a web hook, then like the code runs and then modal gives you this link where you can like ping, like a development webhook. And of course, like, I never get my code right the first time around. So like I bring up Postman or something and I like try to hit my web, like that link. And of course it like I get whatever, error. And kind of without realizing I just like went over to my code and I fixed it and I just saved the file and I saw over here in my terminal, like it just redeployed and gave me like the link again. I think that was like a really cool like surprise for me, I guess, is like, oh, I don't even have to like, I can just like keep this up over here in the terminal. How does that work exactly? And like, was that something that you stumbled upon? Because I found that a really satisfying way to develop because where it's like, oh, I just keep this up, I keep modifying the file and trying it until it works and then I can just control C and say modal deploy and then I'm done, right?

Yeah, for sure. I know I'm harping on it, but kind of thinking about feedback loops and the sort of iteration of speed. As a cto, I manage a lot of different teams. I manage data teams and front end teams and back end teams. And it's sort of interesting like how like different disciplines of software engineering have figured out their own iteration cycles, like the ability to get feedback loops very quickly backend. Engineers tend to build a lot of, write a lot of unit tests. Like that's like their way, like they write some code and then they run all the unit tests or maybe they run a specific unit test that you know is going to break and they have that like sort of way to go to get the fast feedback loop. You go to front end engineers, they have kind of a setup like you just described. They have like one monitor with the website and then one monitor where they write code and when they save, it's like hot realistic code. So I feel like sometimes data and backend people don't give enough credit to front engineers. They have really figured out a lot of stuff around software engineering for fast feedback loops. And actually if you look at the modern tool chain for front engineering, I actually think in many ways it's more advanced than any other part of software engineering. And so that is the sort of feedback loop that I wanted to have with Modal. And what I think makes engineers happy is that super snappy feedback. You just have to save code and then it's live in the cloud. Yeah. So we built that specifically for the web serving part of Modal because that's something you kind of want to have is that ability to. We don't, you know, it's maybe less like you sort of visual feedback, but it's like the ability to like deploy something in the cloud and then you can like, you know, hit it with Postman or Curl or whatever immediately. Yeah, I mean under the hood, it's, it's not super complex. Actually refactored yesterday. It's kind of funny. We just monitored the file system and then when we see that any file was updated, we just reload the entire app in the sub process and live patch the app running in the cloud. So it's pretty straightforward. We had a lot of that already built.

I think the problems you've been solving for the past two years are probably really complicated for you to loop that into the category of really simple problems. I think that would probably be quite complicated for many, many people.

Yeah, for sure. I guess it's like simple in the sense that it's sort of, you know, we already built so much of the like underlying complexity to like make that relatively easy to support the hot reloading. Like the fact that we already built like so much complexity around like take code and deploy to the cloud and do that very quickly. You know, that's a very nice foundation.

To then that's your sort of bread and butter.

Yeah, yeah. Like building that, you know, fast container, fast file system stuff. There's a lot of cool stuff that that unlocks.

So this is a particularly interesting episode, I would argue for me and probably for quite a few of our listeners that listen regularly, because we're talking about something and Eric we have the privilege of you as the person who's created this, but we also have Daniel, whom I, you know, I've been working closely with. And our listeners have been listening to and hearing Daniel's passion and him building his own business on your plat platform. And we talked to lots of different companies. And so it definitely has intrigued me in a way that not every different company owner, if you would, has. I'm kind of curious. I'm thinking about it from a slightly different perspective from Daniel, but you've really got me wondering, like, how to make this happen. I work at a big company, as you know. We have big investments in kind of the, the big cloud providers, as all large companies do. What are good strategies for companies to say, okay, we have so much in these other big names and stuff that are out there. How do we start using modal effectively? What are the kinds of things you've seen your larger customers do in terms of migration over or things that you might recommend that enable something of a migration to be more seamless, less painful? Because normally when you think of large company migrations, they are almost always fraught with pain and misery and challenges for the IT crews. So how do people get to this thing that we're hearing about today and mitigate all of those problems?

Yeah, yeah. I mean, first of all, like, admittedly, like we're fairly early and you know, so a lot of our customer base is early stage companies like starting from a clean slate, who have absolutely zero infrastructure. And that makes it a little bit easier. It does, in part because there's like nothing legacy that they have to port over, in part also because they're just desperate for tools and so the sales process is a little bit easier for us. I find that like, the conversation when we talk to bigger customers is obviously quite different. First of all, there's often an existing data platform that's already built in house. There's of course also security compliance question, and that's something we're working on. I think long term there's a lot of really cool stuff you can do around VPC peering and other things to enable big companies to have the security guarantees that they need. But I also think it's a separate conversation where at a bigger company it's like this one person who's a decision maker who has the credit card. There's another person who built a data platform who now we're saying, oh, actually we shouldn't use that, we should use modal instead. And it's a tougher conversation. And then there's maybe a data scientist and a third person is a data scientist who really want to deploy models. They don't really care about the infrastructure, but they heard good things about Modal. I tend to think in those conversations it's about finding a niche use case that's low risk, that doesn't sit in some sort of critical path of. The whole business relies on this. So it could be some sort of greenfield, something new, deploying a model or a very simple pipeline, something that maybe doesn't touch super sensitive data or have super critical guarantees. So some researchy project that's typically where I tend to start is and often trying to find people, data scientists or machine learning engineers who feel like the platform team doesn't really have time for them. They want something that lets them iterate quickly without having to bother the ops team. Those are probably the easiest conversations to have with slightly bigger companies.

That's good guidance, I appreciate that.

I think it's very fitting that the platform is at least right now and please correct me if I'm wrong, it is very Python centric in terms of like the development workflow and what's supported. Do you see this being sort of like, like you said that you can support so many different types of jobs and apps in Modal. So you know, on one side you could say well this could become like very general purpose in some ways or it could like fill a really niche gap that obviously it is starting to fill and just do that really well and continue to kind of go deeper there. What do you see as kind of the path forward? Or maybe it's a both. And with some things coming sooner than.

Later, I think of modal as my 20 year project. I'm finally building a tool I always wanted to have and I want to spend the rest of my career doing that ideally. So my end goal is to build very general purpose set of tools that help data teams be more productive. That being said, kind of like what I said at the start of this show, I realized that's almost like megalomaniac vision. I think it all comes down to, in practice, to finding something that resonates with customers and drives growth and validates demand and then sort of sequencing and kind of layering all sort of adjacent product over time. We tend to think right now we have one, one sort of use case and one sort of target Persona that works really well right now, which is deploying online machine learning inference. That I think is an area where we see enormous amounts of demand and traction. So kind of how that fits into sequencing. I think an obvious next step for us is to make fine tuning and training easier to do in modal but also thinking about pre processing, scheduling, retraining so it happens on a loop on a regular basis. Maybe thinking about how do you move your data sets into Moodle to some extent too. Hosting more stateful applications. I think there's a long list of layering on step by step, more and more advanced features and gradually expand to take over because I think the demand is there. No one wants to have this 35 different point solutions that they have to integrate themselves and a lot of the data landscape today I think is very fragmented and as a result a lot of data teams have to integrate so many different vendors and kind of duct tape them together. I think there's a big case to be made for either some sort of consolidation or some sort of defragmentation of the space where fewer vendors do more. So long term that's absolutely my vision is to. We're starting with this place right now. Similarly in terms of languages, right like you mentioned, Python versus other languages, we think Python right now is and great place to start because that's 90% plus of data teams use Python. But definitely think long term. A lot of the infrastructure that we built is low level and it's written in Rust. It doesn't really care about what stuff it's running. We think it could be great to add support for TypeScript or R or Go or Rust or whatever. So there's many different axes of this in terms of how we think about sequencing, expansion.

I'm just saying, you saw me raise my hand. I love Rust. It's my current favorite languages. Go and Rust are on the back end and so. But let me ask you a question that came to mind as you're going through that, as you're kind of exploring the world and and you have certain areas of focus, but there's also some kind of able to stretch out depending on different parts of the strategy you have. How do you see, kind of, to use a very generic open term, the Edge out there, things that are not in the cloud. Do you see you doing anything in the future that would be kind of Edge based or do you see yourself more as the cloud partner for things that might be out on the edge and you have APIs and such available to those. How do you conceive either working with or including the Edge in your overall strategy?

I think Edge is primarily useful for like very, very latency sensitive applications. And that's probably a segment of the market that we just feel like that's not what model is going to be good at. Because if you do things in like WASM or V8 isolates in that case you can make it kind of fast enough. But the way we focus on serverless right now is sort of fat traditional Linux distributions in containers or VMs. And that just has. It's always going to have some non trivial overhead. Maybe a second, maybe eventually we can get it to few hundred milliseconds, I think to sort of edge workloads that people talk about. That's when you really need one millisecond, right. And you're really trying to. Either you're doing some IoT type controlling devices for manufacturing or you're doing high performance CDN SEO type stuff where you want your website to be absurdly fast. Those are the types of workloads I don't really think Modal is suited super well for. And I'm more than happy to let other vendors dominate that space. We tend to think on the timescale of a few hundred milliseconds and up. That's where we focus right now.

No, that's a great answer and definitely, I mean trying to address every problem, you know, out there in the larger space isn't a successful approach. So hearing when I talk to people and I hear no, we're not going to go there, I usually take that as a very good thing in terms of focus and good strategy. So. Good to hear that.

Cool.

Yeah, kind of. As we wrap up here, I'd be curious to hear like obviously you're very passionate about this project. You want to work on it for 20 years. Like this is your life's work. It sounds like, what are the things that are on your mind right now in terms of the things that you're excited about seeing happen in Modal and over the next year, what are you most excited about seeing come to pass as you continue working on the project?

The thing that I personally spend the most time on is probably figuring out the ergonomics of the SDK itself in code. How do you express programs that execute in a distributed way in the cloud and still making it feel intuitive and easy to the user without having to think about the fact that this function runs in a different container than this function. We've made that work reasonably well for online inference. But I think when you go to training and start dealing with file systems, there's certain things that are still a little bit gnarly and I'm working a lot on that right now. So Making that user experience good and sort of intuitive. I think it's really important. On a similar note, like modal right now is somewhat janky when you run it inside notebooks for some particular reasons. I'm not going to get into it, but it's something I definitely want to make. The user experience running modal inside a notebook I think should obviously be. We need to fix that too. It's fine, it's not terrible, but I definitely don't think it's quite yet where it is. If you run modal in the script, there's a lot of backend stuff. We definitely need to scale this up 10x to 100x to scale. We are, we see a lot of demand. Modal does not have a publicly available sign up right now. You can sign up and you go on a wait list. And part of it is that just that we want to have a little bit more control over the scale. There's a lot of work we need to do on the backend to build a foundational architecture running all of this stuff, which is a very hard problem. Essentially our own lambda, our own kubernetes. There's a lot of work we need to do on GPU support and in particular cold start with GPU models and fast loading of GPU models. So those are some. There's a lot of cool work we're spending a lot of time on there, especially when it comes to containers and in general isolation and VMS. It turns out that supporting GPUs in a secure way in a multi tenant environment is quite hard. So we're going very deep and like, you know, I'm reading about Linux device drivers and CUDA and like trying to understand all of those things. Yeah, I mean those are all the things we're working on. I think in a year's time like I think modal, you know, we'll see a lot more traction, you know, modal for like other things than just online inference. Like we're going to see a lot of people using modal for training. We're going to see a lot of people using modal for like parallelization. I think we're going to have you know, much more like sort of, you know, customers on the enterprise side right now. Like we're focusing very much on the startups but we're doing, laying a lot of the security compliance work to be able to go up market. Yeah, those are some of the things where I'm pretty excited about.

Yeah, yeah, there's a lot to be excited about and yeah, please pass on my personal thanks again to the modal team for making me look good today and recently. And yeah, really excited about what you're doing and appreciate you taking time to chat with us.

Yeah, of course. I'm also very excited about this, so always happy to talk about it.

Thank you for listening to Practical AI. Your next step is to subscribe now, if you haven't already, and if you're a longtime listener of the show. Help us reach more people by sharing Practical AI with your friends and colleagues. Thanks once again to Fastly and Fly for partnering with us to bring you all Changelog podcasts. Check out what they're up to@fastly.com and fly IO. And to our Beat Freakin residents, Breakmaster Cylinder, for continuously cranking out the best beats in the biz. That's all for now. We'll talk to you again next time.