Multi-GPU training is hard (without PyTorch Lightning)

Today's models are actually not models. Like we need a new name because there's something that doesn't exist. Like what do you call an encoder and a decoder working together to make an autoencoder or variational encoder.

Right.

They're not models. It's collections of models interacting together. Same for Transformers. Right. So that's really what the lighting module is about. You pass in these models into it and then how they interact together is abstracted by that.

Right.

And I think that's a missing abstraction that was not there. So it's important to decouple that because now I have this single file that's completely self contained that I can now share with my team across in a different division. And their problem might be completely different with a different data set. And they don't have to ever change the code on that model. All they have to do is change what hardware they're using and then what the dataset is. So it makes code extremely interoperable. Right. So I think people come to lightning because they want to train on multiple GPUs and so on. But that's only like a very small part of it. I think once you get into it, you see that the rest of it is the ability to collaborate with peers and be able to have reproducible and scalable code.

Big thanks to our partners, Linode, Fastly and launchdarkly. We love Leno. They keep it fast and simple. Check them out@lino.com changelog our bandwidth is provided by Fastly. Learn more at fastly.com and get your feature flags powered by LaunchDarkly. Get a demo at launchdarkly.com this episode is brought to you by our friends at O'Reilly. Many of you know O'Reilly for their animal tech books and their conferences. But you may not know they have an online learning platform as well. The platform has all their their books, all their videos and all their conference talks. Plus you can learn by doing with live online training courses and virtual conferences, certification practice exams and interactive sandboxes and scenarios to practice coding alongside what you're learning. They cover a ton of technology topics. Machine learning, AI, programming languages, DevOps, data science, cloud containers, security, and even soft skills like business management and presentation skills. You name it, it is all in there. If you need to keep your team yourself up to speed on their tech skills, Then check out O'Reilly's online learning platform. Learn more and keep your team skills sharp@o'reilly.com changelog again, o'reilly.com changelog.

Welcome to Practical AI, a weekly podcast that makes artificial intelligence practical, productive and accessible to everyone. This is where conversations around AI, machine learning and data science happen. Join the community and Slack with us around various topics of the show@changeog.com community and follow us on Twitter. We're racticalaifm.

Welcome to another episode of Practical AI. This is Daniel Whitenack. I am a data scientist with SIL International and I'm joined as always, by my co host, Chris Benson, who is a tech strategist at Lockheed Martin. And this week we have a really exciting show. I'm pumped to talk. We have William Falcon with us, who is creator of Pytorch Lightning and CEO of Grid AI. Welcome, William.

Well, thank you guys for having me. Really excited to chat with you.

Yeah, yeah, we are as well. And I think I might have even mentioned this to Chris on our Slack channel, but I saw you like on Twitter when Grid AI was launched, there was like a screencast of like, this is some things that you can do with Grid AI and it was one of those moments. I don't know if you've ever seen like a Kelsey Hightower demo in like the Kubernetes world or something like that, but it's one of those moments where I was like, things just sort of snowballed and then all of a sudden you were running like you were running models on all of these GPUs in the cloud with very little effort and it was pretty cool. So I'm excited to dive into that at some point.

Yeah, I'll sort of share it.

Yeah, cool. So maybe before we get to there, let's maybe start at Pytorch Lightning. People might have heard of Pytorch, they might have heard of Lightning. I know Lightning kind of shows up in my Twitter feed quite a bit. Could you just give us a little bit of context for what Pytorch Lightning is and how people can use it, where it might fit into people's workflow?

Yeah. So I think, you know, I'll talk a little of my experience to understand the motivation behind it. Right. Because I think my sense from speaking to people in the community is that we've all had very similar problems and thought about very similar approaches.

Right.

And the difference is, you know, we open sourced this and a lot of people started contributing to it. So I started out, you know, I was a software engineer and I was working in finance and before that, I guess I was an undergrad and I was starting, you know, to do Research. And I'd been working as a software engineer, and when I got into AI research, it was in neuroscience, right? So computational neuroscience. And we were trying to take a neural activity from the brain and trying to reconstruct what generated that. Right. And that was in the context of, you know, eyesight, basically. And so what happened there is none of us were like really, really big engineers in deep learning. Like, we weren't experts, right? And so I started training models. And, you know, back then I was using Theano, right? Which is like a very old framework. And I remember the first time we got something wrong on gpu and I was like, magical, because suddenly my time went from months to like a few days and I was like, great. And the research continued. And what I found myself doing over and over back then was I'd have an idea about something that wasn't quite that, right? So in neural decoding, it's basically like translating a sequence of signals into something, an image or another signal, right? So it's a translation problem in essence. And so you could do things like Gans autoencoders, you could do things like regression. So many ideas. And I would want to try a few different approaches with my teammates. And we would have to copy the code over and over again, right? You would either fork the project and then kind of like copy that code over. And then if something new came out, like multi GPU training, you have to then write it into all the code that you did. And so suddenly you're maintaining like 10 different files that are all doing the same thing. And I started abstracting that into like a kind of joint class. I think we all, honestly, I think all of us kind of do this at some point. And I think at that point I had been using Sklearn for a while, so I loved their fit and all those methods. And so I was like, okay, well, whatever, let's just call it fit and do that. And then I transitioned to TensorFlow because we needed to get into multiple GPUs, and it was really hard to do in Theano. So that took our training time dramatically down and then continue working on it for a while. But the problem is it just continued. Every time I wanted to do something new, you had to copy that code over. And then new things came out all the time. There was just a different way of training and it was really hard to go back and copy and paste all that stuff. I left that project for a bit and then kind of went into the startup world, right? And I spent a few years putting NLP models into production. And there it was less about focus on training and more about deploying models. And so I was just like, cool, quick baseline and then just like put that thing in there and see what happens, right? I was less concerned about solving like a very unique problem and more about, hey, I have the data here. I just want, I don't care what the model is, I just want to see some results. Right? So we got that working and ended up, you know, scaling that to a company that got acquired and that was basically using NLP to help, you know, low income first generation students figure out how to pay for college or a text message, which was really cool. And from there I started my PhD and kind of like started that research flow again. And then coming from the startup world, I was like, how do I bring that speed and agility to research? Right? Because we all know this and I think our pathy talks about this. I mean, we all know this firsthand. But the outcome of doing anything with AI nowadays is honestly a function of how fast you iterate through ideas, right? Because like 90% of your ideas are going to fail and then one or two are going to work and then you're good to go. So literally just how fast can you power through those ideas is probably the single biggest predictor of if that thing's going to work or not. So I knew that and I wanted to bring that ability to my PhD research. I was like, hey, maybe I can finish this thing in like three years, right? As opposed to six or whatever.

Ambitious.

Yeah, looking back now, it's not a good idea, but yeah, that was, that was the goal, right?

And so I know the feeling.

Yeah. And so I took my code from my undergrad days and, you know, kind of brushed it off. And then at that point I had already switched to Pytorch. So I was like, okay, well let me just rewrite this thing in Pytorch and see how it goes. So I started working with again NLP at that point and then we moved into like audio research, right, to do speech synthesis and so on and all of that using the same code, right? So it was interesting because like the first code was for nlp and then I modified it to work for audio and then vision and so on. And then eventually I don't think it was quite there at that abstraction level yet because I was still having to do a lot of bespoke code. But then I don't know what happened. Like, I guess over the window something clicked and then the trainer got factored out and then it just became obvious that at that point you need to separate the model from the hardware. And so that's what Lightning became. Then I open sourced it, and then I joined Facebook Research that summer as an intern affair and continue my PhD research. And there you have a giant cluster. Right. And I was like, okay, well, if I have Facebook resources, what can I do? And very ambitious in terms of trying to do research ideas. So we were trying to scale up massive data sets on the cluster as much as we could. Right. So I was consistently training 500 GPU models, that kind of stuff all the time at fair with this framework. And then people noticed because the cluster. There was a handful of teams across Facebook that was using the cluster that efficiently, but the rest of the teams weren't because it takes a lot to do training at scale. And so I started working with those people because they're experts at this. And so we embedded a lot of those practices into Lightning and then ended up with a framework now that can do really scalable training. And then at that point there was some adoption internally, then adoption externally, and then it just kind of took off after that. But I came at it from, how do I move really fast through research, knowing what I know about putting models into production as well and knowing what I know about doing research as well.

Right.

So it's just kind of like having both requirements made it really interesting. And what's really cool now is that it's evolved into. My vision really was you and I, all three of us are going to code the exact same thing in our own projects, right? We're going to code half precision, we're going to code stochastic weight averaging. We're going to code whatever new thing comes up. But why waste that effort? That's not the job. The job is to, if you're Lockheed Martin, I don't know, predict metal, whatever, Right. Find deficiency in materials. I don't know what you guys do there. Right. But that'll work.

That'll work.

I think that's exactly what Chris does, I assume.

So that's the goal. The goal is not to figure out how to implement stochastic weight averages. So what's cool now is that, I mean, I think we're approaching 500 contributors, but these are all like top researchers and PhDs all over the world who implement these things and put them into papers and then within a few hours it's ready and available for everyone. So do you have to know how half precision works with, I don't know, on GPUs you don't.

Right.

But you just know that it's going to save you memory. And so it's been basically turned into a community project. And my vision was really like, can we build like the world's research lab, basically? Right. Can we have all access to top researchers and resources? And that's what's happened so far.

So I noticed as you're kind of going through the story, it seems like as you progressed over those years through the different aspects of your own life, and you're kind of looking at the same problem through multiple lenses as you're going from, you know, software development and then you're doing research, and then you're at Facebook doing research, and the scales are changing. It seems very much like, like were scratching your own itch, but having the benefit of taking into account multiple perceptions of that problem so that you ended up having a very rich understanding of what was needed and how it could satisfy multiple user groups. Do you think that's a fair assessment or am I missing the boat? It seems like it was a really smart way of building a robust project from different perspectives all rolled into one.

Yeah, I mean, I think that's right. I think like I said, you know, none of this was ever because I was trying to build anything for anyone else. Right. I was trying to make myself move fast through research.

Right.

I think, like, once other people started using it, they gave me the perspective there. Right. And they put those constraints and I mean, lightning is not where it is today because of me. It's there because of the community. Right. Like, there's no way I could have ever created this by myself.

Right.

I think, like, I could see the idea and see the templates, but a lot of my job has been to guide the community and maintain standards, maintain usability. Right. So I care a lot about user experience. So. And I don't want to remember a lot of stuff, so there's just been a lot of guidance there as well. But at the end of the day, it's a community that's done a lot of this.

Right.

But I think holistically, having to focus on a lot of domains has made it super general.

Right.

Because doing NLP is very different from vision, and it's very different from reinforcement learning and meta learning and so on. And it's not obvious to know where they overlap. So it's been kind of a research project really in the long run. Right. How do you factor out deep learning code and make it interoperable? Yeah. So that's been an interesting journey.

So Far, you mentioned when you were introducing the motivation behind lightning, the idea of decoupling models from hardware. And I noticed like even just on if I look at the repository for lightning, you talk about, you know, Pytorch, Lightning is just organized Pytorch and it's organized to sort of decouple science from engineering. And so you've got this model side and the hardware side. Could you dive into that a little bit more in and talk about the specifics of what does it mean if I'm using lightning? What does it mean that my model is disentangled or decoupled from the hardware? Both practically in terms of how I write the code and what happens once I hit fit, like you're talking about.

Yeah, so when we're working, I mean, look, I think if you're working at a company or any team really, even research, if you're working with multiple people, you need the ability to share code. And if you're at a company or even university lab, you want to share code across teams. Right. And that's really hard to do without something like lightning because what happens is people tend to intermingle a lot of stuff like data model and hardware into the same files. Right? Well, you know, one team may not have GPUs, or may have different types of GPUs, or may only be using CPUs or your production requirements mean that you can only use CPUs for inference. Right. So there are a lot of constraints there. And I guess if you're not thinking about it how we are from the abstract level, you won't really realize that a lot of the reasons why a lot of that code doesn't operate together is because you're mixing the hardware with the model code.

Right.

And that's something that took us four years probably to get there, to see this. Right? To have these insights. And what that means is that we can factor out deep learning code into three major areas. Well, at least four, I guess, maybe. And we'll find more, right? I mean, it's ongoing research. So one is training code. Right. So this is anything that has to do with linking your model to the machine specifically. So how do you do the backward pass? You know, backward pass on the street is very different from just on CPUs, right. At least technically speaking. What happens if you have half precision there? What happens if you're using stochastic with averaging? What happens if you have truncated back steps? Right. So there are a lot of details that go into it. So all of that is handled by the trainer. And this is the stuff that you're going to do over and over again, right? It doesn't matter if you're doing audio or speech or vision, you're always going to have a backward pass, you're always going to have a training loop and so on. The model is the thing that changes. The model is not just. I like to think about models, I guess, Lightning. We have this concept of a Lightning module. And to me a lightning module is more of a system. Right? So we can think about a model like, I don't know, like a convolutional network or a linear regression model.

Right.

Just like a self contained module. Today's models are actually not models. Like we need a new name because there's something that doesn't exist and I think the Lightning module, which is a system, right? Because models now interact with each other. Like what do you call an encoder and a decoder working together to make an autoencoder or variational encoder.

Right.

They're not models, it's collections of models interacting together. Same for Transformers. Right. So that's really what the lighting module is about. You pass in these models into it and then how they interact together is abstracted by that.

Right.

And I think that's a missing abstraction that was not there and which is why people were jumping through so many hoops, right? To be like, oh well how do you do Gans, how do you do this other stuff? So it's important to decouple that because now I have this single file that's completely self contained that I can now share with my team across in a different division and their problem might be completely different with a different data set. And they don't have to ever change the code on that model. All they have to do is change what hardware they're using and then what the data set is. As long as it conforms to the API that the model is expecting, it works. So it makes code extremely interoperable. Right. So I think people come to Lightning because they want to train on multiple GPUs and so on. And under the hood we have this API called Accelerators that lets you do that. But that's only like a very small part of it. I think once you get into it, you see that the rest of it is the ability to collaborate with peers and be able to have reproducible and scalable code.

Foreign.

This episode is brought to you by Snowplow Analytics. Snowplow is the behavioral data management platform for data teams. Maximize the value of Your behavioral data using Snowplow Insights, a managed data platform that's built on leading open source tech leveraged by tens of thousands of users. Capture and process high quality behavioral data from all your platforms and your products and deliver that data to your cloud destination of choice. When marketing needs to make data informed decisions, when product needs next level understanding, and when analytics needs rich and accurate data, Snowplow is a solution for data teams who want to manage the collection, processing and warehousing of data across all their platforms and products. Get started and experience no plow data for yourself@snowplow analytics.com again snowplow analytics.com.

Thank you for the great introduction to what lightning is and how to think about some of the abstractions that you're working with. I'm wondering if you could maybe share a little bit. I've seen some different stories online, but but I was wondering from your experience with the community that's working with this, could you provide any sort of stories around how people have been able to scale things up with lightning? Maybe in your own work or maybe stories that you like to highlight?

I mean, there are a lot of companies and labs using Lightning today, right? So you can get on GitHub and see that for yourself. I think. I don't know the exact numbers, but it's definitely like, you know, the thousands, like few thousands of them.

Right.

And they go from pharma to retail to anything you can think of. Right. And I think today what's interesting is that, you know, when I run into these people because they're, you know, we're coming to work with them on grid on some of them, it's interesting to hear the use cases, right. Like stuff that I would never imagine. Right. Because I'm not at a company doing this kind of stuff. So that's why I made the joke about Lockheed Martin. But I'm sure you guys are doing much more advanced stuff. Unless I'm building, you know, planes, there's no way that I'd know to do that.

Right.

So what's cool is just like it's been super flexible. I think there are public cases that we can talk about. I mean, there are blog posts by big companies like Nvidia, Facebook and so on about how they use lightning. Right. So you can read that. And I think something that we do specifically in the community is like we really like to kind of protect our partners because like this is a community and we want to keep people's work fairly private as well. So I won't get into too many Details. So I'm just pointing you to open sources that you can look at and how they use it. Right. But these are big projects as well and there are probably about 3,000 projects now that you're sliding that you can literally just go to see them. So the companies that have open sourced their work, you can see what projects they're working on. So it's everything from like video prediction to segmentation to nlp. Right. To summarization to classification. We integrate really well with basically most frameworks out there. So if you use anything that's Pytorch based is very likely going to work with lightning off the bat. Now, in terms of scaling, I mean I personally we've done it internally. Right. But we've also heard from the corporate partners that they're training things on. Yeah, I mean, I guess the number, I don't know, there's no real limit so far. I guess it's whatever Pytorch supports, however.

Many GPUs you can get your hands on.

Yeah, and like, you know that's a big part of Grid now, right? It's like with create enlightening, you can literally type in, I don't know, a thousand GPUs and if you have the Amazon quota, great. And we can give you as many as we can as well. But there's no limitation, you just have to run it. And I know it sounds crazy, but you literally just have to run it and then it'll just work. Right. So it's just a function of the compute there. I mean a few weeks ago. No, it was like a month ago at this point we did a collaboration with Microsoft. So Microsoft has this library called deepspeed which is really cool. Facebook has one also with the Fairscale team. And basically it lets you scale up models dramatically by helping you use CPU memory efficiently. And you know, the way you shard gradients and the way you shard parameters across GPUs really helps. So we were able to train a GPT model. I think it was like, I remember it was like 20 billion parameters or something like that. So we have a case study for that. So just for context, like the original GPT3 was, I don't remember, it was like, let me see here, 160 billion parameters or something like that. So I don't want to misquote you numbers, but basically whatever the original GPT3 was, I think it was like one third of that. Well, only eight GPUs. So that's crazy. Just I don't think anyone in industry needs that much. I haven't seen people use that much. So I'm just saying, like that's a pretty good lower bound.

175 billion at least. That's what Google is telling me on a search.

Yeah.

So you were very close and you said you were running that on what, eight GPUs.

Yeah, on a one hundreds. Only eight of them. Oh, wow. And I mean it's a 100, so they're much bigger than V one hundreds. But I mean we'll be doing more tests. That was with these just literally. And what's cool about it is if you're just using lightning on your trainer, you just say, I think it's like plugin equals deepspeed, like a string called deepspeed. Just by doing that, you get that out of the box. Right. So that's the kind of stuff that we embed into training. So, you know, do you have to know how to do that? You don't, but now you get that benefit.

So I wanted to real quick pop in one thing before we kind of start moving on on this. There are some people that are listening that may not. They may even be not Pytorch users. They might be TensorFlow users, but they're thinking about switching. You know, we always get into conversations. How does a workflow look like when you're integrating Pytorch lightning into your workflow, you're using the rest of the ecosystem. Could you at a high level just for those who haven't used it and maybe not have the something directly that they're going, oh yeah, I've done similar to that. I can just add lightning into that. What that looks like what that savings. Why is it called lightning for them? They're kind of going, oh, there's this thing that may really help me. Can you kind of just top off a little bit of a workflow on how I go from the beginning to getting something productively deployed and what that looks like for somebody who hasn't seen it before?

Yeah, absolutely. Wait, so I found the blog post. So it was actually 45 billion parameters that we scaled it up on 81 hundreds and you can look it up, but it's called accessible multibillion parameter model training tuning with Pytorch lightning and deepspeed.

We'll link it in the show notes.

Yeah, sounds good. Okay, so basically it's how do you adopt lightning into your workflow? Right. So I mean, obviously if you're coming from not Pytorch, then you would just, you know, start With Lightning, there's a very simple readme there. Like I would say, you know, copy, paste, readme, there's an MNIST example on there and you just run it. You'll notice though, as people will say, well, but where are the advanced examples? And my point is that that is the advanced example. Like, all you have to do is change the data and it'll still work for imagenet, right? So that's the beauty of it. There's no different example for that. I mean, we put it if you want, but at the end of the day, just change your Data and set GPUs to 64 and you're good to go. So that's the easy part, right? So if you're coming outside of Pytorch, then you can do that. If you're coming from within Pytorch, then what TPPL tend to do is when they start a new project, they'll either start it on Lightning directly or they'll convert their existing project into Lightning. So it is really a refactor on your Pytorch project, Right? So you basically take your main loop code, which usually looks something like you initialize a model, you set a bunch of flags, you set some sort of arc parse arguments, and then you download some data or link it somehow, and then it's all boilerplate. And then there's two loops in there, which is like four epochs and epochs, and then you have in there four, four batch in your data loader and then you start training. So literally everything up to that 4 batch in your data loader is deleted, Right? So it's gone.

Yeah.

So then the only thing that you need to track is what's in there, right? Which is like the. We call that the training step, which is the meat of what you want. I mean, think about when you're doing work, like that's what you spend your time on. So that goes into this function called the training step. And then the training step goes all the way from taking your badge into generating a loss that you return with that gradient attached, right? So some graph. So, you know, it could be a few lines. Usually it's only a few lines because that's most of what you're doing. Now, the model that you left at the top, that one, you can keep it separate and just pass it into Lightning module and just use it, you know, soft model equals model. Or you can define that model within the Lightning module, right? So you can literally copy paste the layers and all that into the Lightning module. If you Want because the lighting module is an NN module at the end of the day. So that gets you basically most of it. Then you need to find your optimizer and bring it into a function called configure optimizers and then you just return it there, right? So then you're going to link up the parameters through that as well. So that's three methods, right? That's your init, that's your training step, and then that's your configure optimizer. And then the rest of that is optional after that, right? So forward, we don't actually need it. We use the forward method for inference, right? So if you train a model model and you for example, an autoencoder, right? So an autoencoder has two sides, an encoder and a decoder. The encoder maps some input into some space and embedding and then the decoder maps that embedding back into some space. So an autoencoder can be used in two ways. You can use it as embedder, basically. So you can take an image and get an embedding for it and then do similarity search and so on. So if you're building like a visual engine or something, you would do that. Or you can use a decoder for sampling. You can give it a random some vector and it'll give you an image, for example, or text or whatever you want. So depending on what your use case is, that's how you're going to implement the forward, because the forward is what's going to be called in production, right? You're going to call the model with the input to it. So we actually allow the model to be torch scripted and put into onnx as well. On nx, I guess for production use cases. It's literally a function called torchscript to onnx and then you're good to go and it does all the things for you, you. And then, you know, you just have to get the inputs, transform it, pass it through and then do the return. So it's very simple. Now there's other stuff left, so that's literally it. So you just have to copy that stuff and then anything else that's left is usually around data or maybe validation or testing the validation. We have a validation step and a test step as well where you can just copy, paste that code in there. If you want a validation loop or test loop for the data, you can leave it as this. You can just pass in the data loaders directly to Lightning or you can use something called the data module, which is a completely optional abstraction, but it basically captures your training, validation and test data loader into one class and couples the transforms as well. Because what usually happens in big companies is that I'm working on, I don't know, let's say I'm doing, I guess maybe selling something, right? And so I'm selling clothing. And so I have the data set of our inventory with images and stuff on. And then when I give it to you, you're going to be like, hey, how did you transform the images? Did you crop it? Did you random flip? What did you do?

Right?

So unless I give you that code, then it's going to be a little bit hard and we could mess it up. So the data module embeds all of that. So I just have to say, here's a data module for the clothing data set and you just run it and you know it's going to be consistent across the board no matter how you run it. So that's an optional, I mean, highly encourage abstraction, but it's optional. Yeah, that's basically it. So if you do it, I would just recommend like, like don't delete your project, just do the refactor first, put it into Lightning, run it once, right? You can run it on cpu. When you do it with Lightning, you're going to be able to run it on your local machine with CPUs or GPUs, take a batch of data from your data set or a single example and overfit both models like your original code and this one with the same seed and everything, and make sure you get the same result. And then once you get that, then you're good to go. You know, you didn't mess it up. At that point you can go ahead and say GPU is equals 128 and then off you go.

So it sounds like that if I'm a Pytorch developer and I'm already using that API, I'm creating the layers of my model. I don't have to throw out the way that I created that model. In some ways I get to sort of delete a bunch of my code having to do with the hardware stuff and some of the other training related things. And I can keep my model and sort of refactor it into this Pytorch module, the Lightning module, and then call the trainer and essentially then I now have less code. But my code is also more robust in that I can run that training on a whole variety of hardware and that sort of thing. Am I Basically summarizing that. Correct. Or anything you would change about that.

And it's more readable. Right. You can literally give it to your colleagues and then they know to go to training step to see what's happening. Otherwise, what do you do today? You're like, hey, here's this, like, seven lines on GitHub.

That's crazy.

You know, you can actually. They're like, wait, where? Where is it what you're doing? Because most of it, it's like boilerplate training stuff right now you can be like, hey, here's exactly what I'm doing. They're like, oh, you're sampling delay in space before doing this thing. Oh, interesting, right? It's not mingled with all this other stuff. So it's very easy to read as well. You know, I joke, but it is kind of like cleaning. Yeah, like cleaning your house, I guess. Like, imagine, I guess, roses, right? So maybe this is a good example. So a rose, you have to cut it from a bush and trim all the stuff. And then, you know, you get this, like, bulb at the end, which is what you care about. It feels like that. It's like, no one's adding these other leaves because they want to. It's because they have to. Right. So when you refactor your code, it's the sense of like, okay, it's a lot cleaner now. Like, I just removed a lot of unnecessary stuff and also stuff that you're likely to mess up. Right. Like, we test very, very thoroughly, and we have thousands of people testing this stuff. So did we mess up the backward pass? Definitely not. Right? Did you mess it up? Hopefully not.

Changelog is the best way for you to directly support practical AI. Join today and unlock access to a private feed that makes the ads disappear, gets you closer to the metal, and help sustain our production of practical AI into the future. Simply follow the Changelog link in your show notes or point your favorite Web browser to changelog.com once again. That's changelog.com.

It's better.

Okay, I want to kind of circle all the way back to where our conversation started, because I want to get back to that cool demo that I saw on Twitter about grid AI. So maybe you could just give us a little bit of. Of sense of what grid AI is, kind of how it came about, how it's maybe connected to the lightning community, if at all, and then we can get into some of the details about what it enables.

So, I mean, as you saw from my story, like, I care a lot about reproducibility and speed. Of iteration and something that I thought a lot about as we were doing research. And you know, building Lightning was in a corporate setting. You would want to scale this stuff up on a lot of compute and you have cloud resources and all these different things. So the requirements for training at scale in a company are very different, right, Than just like on a Google Colab or a Kaggle. It's just like a very different world, right? So, you know, it's funny because, you know, deployment also goes into that. People are like, oh, here you go. You deployed on this thing. It's like, well, yeah, but like most APIs, most real machine learning systems are not just an API, right? So we know that. I mean, a lot of us build these models. We've all been at companies before as well, at scale, so we know exactly the pain points there. So the thing that kept coming up is like, cool, Lightning is letting me do all this, but how do I, you know, I'm still having to do all of this cloud stuff. Like why, you know, if I ask for 32 GPUs on lightning, yeah, Lightning will do the thing. But like, you need to give me the 32 GPUs and giving you the 32 GPUs, that's a lot of work to do it consistently and at scale and cheaply so that you don't have to burn resources, right? So what people end up doing generally is they build these ad hoc internal solutions. They're like kind of put together bash scripts or things that they string together a semblance of a platform and they're great. And yeah, you will get things running, but you won't be able to just scale them down immediately. You won't be able to have really fast build times because they're highly optimized. You want to have real time logs, you want to have real time metrics, you want to have time integrations. So, so all of these bells and whistles, when these things happen internally, they usually get pushed away because they're not a company priority because they shouldn't be. You're building airplanes, you're not building machine learning platforms, right? So you're normally not going to put the effort into making all the things that we care about as researchers and data scientists and machine learning engineers in there. So it's just going to make your life a lot harder. So it's about how do we bring that whole experience and encompass that model development cycle in a scalable way for the needs of like, companies and even big labs, right? Because like most serious AI Labs, they're training things on very, very large scales as well. Because training is a big part of the picture. It's not just a deployment. I think the deployment is interesting, but it's a lot easier because we've been deploying websites and things forever.

Right.

But we haven't been training for that long. It's kind of a newer thing. So that's really the focus of grit is to just completely eliminate the pain point that was left from using lightning by not even having to deal with it. You just type in 32 GPUs and it just happens. Right.

So I'm wondering. There's still a lot of people, I think, and maybe I have a misconception about this, that they think like maybe Training models on GPUs in the cloud is always going to be more expensive than training on a sort of like you're going to buy a on prem server and do it in house based on sort of your experience with that and like the current sort of state of cloud providers and all of that. Is that perception mostly driven by the fact that, you know, and I feel very seen by the comment about like you have all these bash scripts strung together. That's like my life. Maybe, but is it because, like that way of doing things is a bit inefficient and you waste a lot of resources and that sort of thing or where do you think that perception's coming from and do you think it's accurate? I guess is my question.

Yeah, I think you hit it right on the nail. If your system is inefficient, then it's more efficient to have your own machines.

Right?

Because running on grid means that we install your dependencies. Everything you do link up your data in a matter of minutes, if not seconds. People don't generally optimize their stuff in the backend to do that. So what they end up doing is they want to run in the local machines because they don't have to install their environment. Environments. I don't have to do all this stuff again, right. It's just there and it's repeatable and things start immediately. So it's a lot cheaper. I'm not going to say that running on your local stuff is not generally cheaper if you're doing things 24, 7, but you're limited by bursting capabilities. Right? So you're never going to have, you know, I don't know how many GPUs AWS has, but it's got to be hundreds of thousands, right? So if you have to hit a deadline or do something really quick and even go through ideas fast. If you're buying your own GPUs, you're going to be limited by how many you have there, right? So it's going to be more like sequential model building as opposed to asynchronous building. So with grid, you can go spin up 200 GPUs, run for five minutes and shut them down, and you just got a lot done, right? Whereas on your own machines, even if you were to do it yourself on the cloud, you would probably not even get the models running for 20 minutes and 30 while you spin up the machine, set up all that stuff, right? So. So I can take $100 on grid and get more GPU minutes out of it than you would normally without optimal systems, right? So it's just very, very optimized. Now, I do think that people need to know about things. I mean, we do a lot to lower the cost, right? And I think one of those things is spot instances, right? So spot instances are machines that can be killed at any time by aws, right? Or whatever cloud provider you're using. And then at that point you're kind of done. Right? But the nice thing about Spot is that it will be like, I don't know, 50 to 80% the discount, right? So if a GPU costs $3 an hour, it could be like 30 cents an hour to maybe a dollar an hour, right? So it really depends. So I think what you're saying is true because I did the calculus myself and I. In fact, I have like, blog posts on how to build your own GPUs for this reason. But that was only four 1080 TVs, right? And it cost me maybe six grand to build that machine, which is great. Now, $6,000, if I'm paying full GPU prices, I'll burn through that in like two weeks for sure.

Right?

But if I'm paying spot prices, then that's good. That changes the game. And then, not only that, but if I'm getting more training minutes out of that, that's a lot better. And then you factor in depreciation and this other stuff, plus maintenance, then it actually becomes a little bit competitive, right?

It does. I'm curious, you know, and we're talking a lot about the training. Could you talk a little bit about Grid AI's deployment story and what that is? And, and in my mind, one of the things, like speaking for myself, I'll be training centrally in the cloud and stuff, but at the end of the day, I got to get my model or my system of models out there into something, often some sort of edge device, not cloud based, something that's a physical thing out in the real world. Can you talk about how you work with Grid AI to affect that?

Yeah. So today Grid doesn't support deployments.

Right.

So the thing that we like to focus on is making sure that we really nail certain experiences before moving on to other things. Right. So we will support deployment at some point, probably very soon. Right. But the thing is like, I don't think that we're fully, fully optimal on the training side yet. Like, I think we want to provide a really world class experience there. So for our users today, like, you know, you cannot access artifacts, you can get model checkpoints and all that stuff. So the deployment, most users have a deployment system in house already, so they can just take the artifacts and do their thing. Right. So we're not blocking any of that. And all of these things are like URL based. And if it's lightning, that's very easy to do. Now we're going to make it a lot easier for sure, kind of the way that we do things. But today we are laser focused on training. But I will say I think working with Grid at this stage is great because I think companies will be able to help us influence that roadmap and help us build something that they really care about as well. Because as soon as we start getting to deployment, we're going to do it our way and we have a very special way of doing things. So we hope that we have the feedback from the community and users to make sure that we're doing it in a really useful way.

And how as a user of Grid AI, because this is really fascinating to me because I've even been struggling to get some in house GPUs just with supply chain issues and all of those things. So running things on the cloud is something that we're actively thinking a lot about and doing it in an optimized way. Now we kind of talked before about going say from Pytorch to Pytorch Lightning, let's say I've got my Python code, I'm using Lightning, it works great. And now I want to run it with grid AI on, you know, 100 GPUs in the cloud. What does that look like? Do I need to set up my cloud account, set up billing on that side and then set up my Grid account and then use a grid tool to connect them both? How does that whole flow work from that point?

So I think yeah, That's a good question. Okay, so generally I like to think about what we're trying to do, like that leap between, I don't know, Windows machines to Mac machines, where things just work. So what is that Apple experience for machine learning? And I think, to answer your question, it's very, very easy. It's not as easy as I want it to be today, but it will be. So basically there are a few ways. We have three tiers of usage on Grid. We have the community tier, which is free. Literally you're just paying the AWS compute.

Right.

There's nothing in there. Like we're just orchestrating stuff for you. But it doesn't really work for teams and like big companies. Right. Because there's a lot of stuff that needs to happen. So then we have the teams and enterprise tiers that let you do those kind of things. On the community tier, you literally have to do nothing. You just copy paste the link to a GitHub file, you paste it into the UI or use the CLI and you select how many GPUs you want and you press Enter and you're done. Right. Like it's that easy. Dependencies are automatically pulled for you, their inference from the code that you have, your requirements, all that stuff. So we try to do as much as possible. Yes, there will be times when that fails and we will work with you to figure out what happened and make sure that we get it done. But dependency management is a big deal for everyone and it's a really hard problem to solve. So it's going to take us a while to fully solve that problem. But if you are at a company or at a big lab, usually we call that community tier. That's going to work great for side projects and public data and stuff like that. Kaggle, you know, prototyping things. Sure, if your data is not secret, then it's fine. So great for academics as well. But if you have corporate data, then you're going to be on the teams or enterprise tier right there. What you end up doing is we basically link up your cloud accounts, Right. So you just set it up through Grid, you pass in credentials through there, and then those keys let us control resources on your behalf. Only as much as you allow us to. Right. To make sure that we orchestrate everything in your cloud. So it's kind of this hybrid on prem versus not on prem. We also offer on prem if people want it. So once you do that, you basically put in your cloud credentials in there, then you're good to go. When you Run stuff on grid. Instead of running on the grid cloud, which is a community cloud, you just select your cloud, whatever you named it, and then you just run on it. Right? And that means you can link up as many of these as you want as well.

As we kind of wind up here, one of the things that's really struck me through the conversation is that you are a man of substantial vision. And as we kind of of wind up, I'm really curious if you would kind of look out a little bit beyond just, you know, the next product cycle and that kind of thing into where you want to go, both with grid AI and where you see the larger industry going in general in terms of trying to make this work a little bit better for people and take the struggle out of it that you clearly have been working on for a while in various capacities. Could you tell us a little bit about what future you think we're going toward and what you would like, like to shape it and how you would like to shape it?

You know, when I started in research, I was really disappointed that I had to do so much work over and over again that other people were doing and that I had to learn so much just to, you know, decode a little bit of neural activity. Right. And the world that I would love to kind of help bring to the table is a world where the person, the scientist, the researcher, the machine learning engineer, the person that has the knowledge of whatever they're building, right? The doctor, the biologist, the mechanical engineer, you know, you name it, the person who really knows their domain can basically focus on that and not like have machine learning and all of this cloud stuff just kind of fade into the background, right? And just be like WI fi, just like your cell phone signal, like you don't think about it, right? You're just working on your problem. So how do we take that leap? And I think that's what we're trying to solve. Are we there yet? No, but we're definitely on track. And I think that working with a lot of amazing companies and getting to make sure that we support their use cases is what's going to help us get there. So the person who builds the models, who has the ideas that. Dr. They can be the ones to actually train and deploy this stuff. Because at the end of the day, I think that deployment is literally just another training cycle, except the data is live and you're not back propagating into your model. Model.

Right.

That's awesome. Thank you so much for talking to us about grid AI and lightning. It's been really wonderful. And like I say, we'll put show notes and everything, the relevant links that we've talked about in terms of lightning and grid AI, definitely check it out. And yeah, thank you so much for joining us, William. It's been a pleasure.

Well, thank you guys. This is really fun conversation. Thank you.

Thank you for listening to Practical AI. We appreciate your time and your attention. If you enjoyed this episode, help us out by spreading the word. Think of a friend, think of a colleague, somebody who would benefit from listening to it and send them a link. We'd really appreciate it. Practical AI is hosted by Chris Benson and Daniel Whitenack. It's produced by Jared Santo with music by Breakmaster Cylinder. Thanks again to our sponsors, Fastly Linode and launchdarkly. That's our show. We hope you enjoyed it and we'll talk to you again next week.