Large models on CPUs

Foreign.

Welcome to Practical AI. If you work in artificial intelligence, aspire to or are curious how AI related technologies are changing the world, this is the show for you. Thank you to our partners at Fastly for shipping all of our pods super fast to wherever you listen. Check them out@fastly.com and to our friends at Fly. Deploy your app servers and database close to your users. No ops required. Learn more@flyio.

Welcome to another episode of Practical AI. This is Daniel Whitenack. I'm a data scientist building a tool called Prediction Guard. And I am not joined today by my co host Chris, but I am joined by by an amazing guest who is an expert in all things model optimization and efficiency and running on CPUs, which is super exciting. I've got Mark Kurtz, who's director of machine learning at Neural Magic. Welcome Mark.

Thank you, Daniel. Thanks for having me on.

Yeah, yeah, of course. So let's maybe just start out with a kind of like state of model optimization right now. So first off, could you kind of describe like when you're talking about model optimization or that set of tooling, what do you mean by that and how does that fit within maybe the things that a data scientist or an AI person would want to do?

So whenever we're looking at model optimization, we're usually focused on a few different techniques, but the ultimate goal is to make the overall model smaller and faster. Neural networks, known to be very large models, especially compared to more traditional machine learning. And it turns out the size of those models is the important part in terms of exploring a large dimensionality of a space. But it actually doesn't use all of those pathways at inference time. So what we specialize in is specifically pruning, where we're going to remove connections within that network. Quantization, where we're going to reduce the precision of those connections of the network. So going from, you know, the typical FP32 down to intake and then additionally distillation, where we're taking larger models and trying to teach a smaller model to mimic the capability and the functionality of that larger model. So it's kind of, you know, overall a high level. And yeah, it's, it's a very exciting space right now. It's kind of exponential in terms of the number of research papers that are constantly coming out on a topic. Everybody's very excited about sparsity specifically mainly because you can turn these large models and get rid of up to 95, even 97% of the weights are actually useless in these. Obviously you can use that for a lot of efficiencies around performance and energy. And that's specifically where we've been focusing in at neural magic and what I've been focusing in on my work. Awesome.

Yeah, that's. I definitely have felt this problem. So I'm sort of asking this question maybe for others out there that, that maybe haven't felt this problem as much. Why is it important to make models smaller or make them more efficient? How does that fit within what enterprises or even users running smaller applications? Why is that important for people I guess is the question.

Generally there's going to be two cases that we're looking at in terms of deployment. One would be an embedded space where we're running on the edge and trying to work there. So generally you want real time latency and optimizing the accuracy as best as possible. So if you're using an object detection model, you want to make sure that you know, for example you're on a security camera trying to draw object detection and make sure that you know when a person walks into frame and whether or not that's alarming or not versus a dog or something like that. So in general on that edge application what you can do is, is use a larger model, remove a lot of the pieces from that larger model so you can keep the accuracy of the larger model but take up the space of the smaller model. So significant improvement in terms of accuracy on that edge device while still maintaining, you know, the constraints that were set for you in terms of memory and latency that you need to request back with. And then the second one would be on the server side. And that's generally where we're looking at, you know, more throughput based applications and potentially also latency if they're shipping the data up to some server to be processed either on NLP or computer vision. But overall there what we're looking at is, especially whenever people get into larger deployments on ML and neural networks, the cost significantly shifts not from training but from, but to deployment. You're for a lot of larger enterprises that are actively deploying 80, 90% of their costs is purely in deployment on these machines. So what you can do is take the exact same model that you have, reduce again the amount of compute that you need to run it so that one will run faster. But two, ultimately what that means is that it's going to run significantly cheaper. Right. And we have cost savings on the order of, you know, know 10x20x even larger if you're really trying to specialize and optimize. So it can be a significant reduction once you're at that scale. I would say definitely. If you know you don't have anything deployed yet, don't worry about optimizing the model, worry about getting a use case that works and something that you can prove out as soon as you go into deployment. Model optimization is a great thing to start because it's essentially just free performance that's left on the table that can significantly affect your bottom line.

We've mostly been talking about kind of like model size and optimizations and I do want to get sort of down and get into the nerdy stuff around like how some of this works. But before we do that, I'm also curious about this element of deployment on GPUs versus CPUs. It seems like some of what's indicated, at least in like the tooling that you're building is like the potential to take a large model might require a GPU at inference time and potentially run that on cheaper commodity hardware that only has a cpu, maybe doesn't have a gpu. Like what is the state of that now and how far can you push that? Or maybe also how could people best think about that in terms of when and when that might not be possible?

I guess as you said, we specialize almost entirely on CPU performance and in that actually our latest ML inference results on MLPerf has have come out. So in that we show that we're running faster than T4s and A40s and things like that on just commodity CPUs, so server based CPUs, stuff that you have in your laptop, desktop, things like that. And it's very surprising that you know these, what's thought of as these little CPUs can outperform the GPU. And we see this generally across every domain that we've tackled and that's been across image classification, object detection, excuse me, segmentation. And now we're working in the NLP and NLG space and actively coming out with that. But overall we're seeing the same use case where these models are over parameterized. We can take away a lot of that compute. And what that means is that you can actually get the CPU and the GPU about equivalent in terms of compute throughput because with the sparsity and the dynamic setup of CPUs we can run and skip all those zero multiplications, right? So significant reduction in compute, they're about even. But then the CPU has a unique cache hierarchy, which means that we can reuse that cache more often than what you can get on a GPU L1 and L2 being extremely quick, faster than GPU's main memory, and L3 being about equivalent. So overall, what we do on our performance optimization is skip all the compute to get even, and then use that cache hierarchy as efficiently as possible on the CPUs, so we can get faster memory access than even you can get on a gpu. And we pay a little bit more by doing a little bit more compute by doing that. But overall, it works out that you can actually beat the GPUs in that setup with just pure software.

I anticipate this is maybe a question that you get sometimes, but it's like you hear so much about the necessity of GPUs for running these large models. Do you find generally, practitioners are just unaware of this possibility of running these large models on CPUs? And how has that been for you and those that you work with that are actually doing this amazing work and have these awesome tools like, how has that been in terms of overcoming that barrier of perception that people have?

It was a barrier that we hit especially early on a couple years back, where it took a lot of convincing to even get in the door to talk to anyone because they just didn't believe what we were saying. Now it's gotten to be quite a bit better, especially with the newer software that's been pushed out and the newer chipsets on CPUs, they're getting a little bit more even in terms of gpu. So that's, you know, within a stone's throw to try and match gpu. So people are a little bit more accepting of it. But, yeah, whenever we show them the numbers, generally the first reaction is, well, let me try that on my hardware, because you guys gotta be doing something weird. Let me. Let me try to do that. Let me replicate that. And as soon as I do, then the next question is, okay, well, how do I do this to my model? And that's usually where the tricky part comes in, is model optimization hasn't always been the easiest thing to do. It can take a lot of research to enable new architectures and things like that. But that's what we've been also specializing on at Neural Magic is making all the research that we're doing, being able to put that into open source and also building out a SaaS platform on top of it so everyone can easily play with hyperparameters and get something that is consumable. But I would say that's probably been the biggest gap in terms of trying to get people off of GPUs. Onto CPUs is the model optimization that needs to take place first to be able to run faster than the GPUs.

You talked a little bit about sparsity, which I want to dive into over time and I want to get to those tools and the open source stuff you mentioned like these large models and people are probably used to hearing these numbers, you know, 3 billion, 7 billion, 13 billion. However, up from there, these models have. In terms of numbers of parameters, could you describe a little bit what you mean when you say 90 to 95% of these connections or maybe less than that, but a high percentage in some models have no impact on the actual forward pass or inference in the model. Could you describe a little bit more by what you mean by that?

Yeah, definitely. And I'll. I'll take two steps to doing that. One is just covering kind of the 90, 95% class, at least where we've been able to get to on those. And the second is looking specifically at large language models. So for the first one, whenever we're looking at getting rid of 95% of the weights, let's take ResNet50 as an example. This is our toy benchmark model. This is essentially what we prove out all of our technology on, because it's a common feature in MLPerf and for most performance tests. So what we can do coming in is looking at those convolutional layers it has. I forget how many million parameters within it, but it's definitely not the 3 billion, 7 billion or up on top of that. But within that we can actually zero out. So what we're doing is taking all, imagine taking all those parameters, dumping them into a giant array and we're just going to zero out the ones that are not important and figuring out the ones that are not important as part of the research. The easiest assumption is just saying that the weights that are the largest are the ones that you want to keep. So the ones that are furthest from zero are the ones that you want to keep. Generally you can think of this in two ways. One is that as the model is training and being regularized, the weights that don't matter are going to move towards zero. And then the other thing is during that forwards pass, the weights that are higher magnitude have more an effect on the output, right? And everything else is going to be noise in between. So we're able to essentially get rid of just our. What Whenever I say get rid of, I mean setting those parameters to zero within 95% of them. So you're left with 5% of your weights that are non zero. That's actually all that you need to preserve the accuracy on Imagenet for Resnet 50 for example, and some quick kind of intuition in terms of how I've been able to think about this and why it works and things like that. You can see as we increase the size of our dimensionality in our optimization space, what we're doing is, and there's few research papers out on it, that we're able to connect more of the local mins, right? So the optimization process will slowly converge further and further down because more of the local mins are connected. Generally though, there's only a few of those pathways that you actually need to connect those local mins. So all that we're doing is we're following down that most optimized pathway and removing everything else around us in terms of that dimensionality. So it's kind of one of those things that as you're training, it's slowly selecting the weights that matter that gets you down to that local men. And there's very few. So the important part was that large dimensionality of the optimization space, but not every direction mattered, right? So then we can get rid of it. And then diving in on the LLM side and large language models, we actually have a recent paper that came out from one of our principal research scientists, Dan Alastair, called Sparse GPT. And that's where we're looking at taking Opt and Bloom models all the way up to 175 billion parameters and being able to optimize those and remove as many weights as possible, all in this case in one shot. So just using the model without any retraining, we're able to get rid of around 60% of the weights without doing anything. And there's a new paper out of Cerebras, actually that was looking at the LLM story, and they're able now to get to 80% sparsity on these LLMs with retraining. So that's kind of the research direction that we're headed down now is proving out how optimized we can make these models. Because there's also a lot of interesting stuff that happens with the large language models specifically because it's generating one token at a time, very latency bound. And that means that it's a lot of memory access to load those weights. So if you can quantize those and then get rid of half of them, you're already at, you know, anywhere from a 4 to 6x speed up just on your Inference times. And that's generally where we're, where we're focused and looking at currently to try and get those LLMs to run faster. The other thing to call out for those two is, you know, 7 billion parameters and 175 billion parameters. Those don't fit in a single GPU. So now you have, you know, clusters of GPUs to serve one model. And a lot of that compute is just completely wasted because all that it's going to is trying to maximize the memory on the GPUs. For CPUs you can throw a few terabytes on there and it works out fine. So that's the other thing to call out with the LLMs. In terms of GPU versus cpu.

This is really interesting. Mark, I want to follow up on what you were just talking about, which I think is a really. It's sort of a subtle point, but it's really interesting in that I think if I understood you right in what you're saying, like let's say that I have one of these large models, 175 billion parameters or whatever, and even for inference I have the necessity to have multiple GPUs just to load that model into the memory of the, of the cards. Whereas on a CPU you can have terabytes of memory. What I'm assuming is like you could load that in as long as you're able to execute it quickly, which I guess is the other piece. So am I right? You sort of have to have both like the ability to load it into memory and you have a bit more space in that on the CPU side, but then you also have to be able to execute it very quickly, which I guess is like why you would think about both space and sparsity. Is that an accurate way to put it?

As you said, you know, you have a total spacing you take up and then a minimum latency that you want to respond to the user out, right? And that's going to set the constraints for your hardware and for CPUs currently, at least for the smaller models, you can get to a usable speed on those. If you've seen like llama CPP, they're doing N4 and things like that on smaller models and they're usable, but they're less accurate. So we're trying to do, and what we're actually working on right now is making sure that we can get that GPU class speed while maintaining the large memory advantage of CPUs so you can deploy this 175 billion parameter model on something Local and you don't have to worry about data privacy, anything like that. It's just there working and available and highly accurate for you.

I definitely heard people that I've talked to who have like tried various optimization techniques and have maybe been dissatisfied with the the performance hit that they're getting, not in terms of compute, but in terms of like actual model performance or accuracy or whatever. Does that performance hit often come about because of the quantization that's maybe part of the optimization techniques? Or are there multiple sources of that like performance hit? How should people think about that?

I think the biggest thing there is is honestly just the amount of choices that people have to apply and not knowing when to apply them. Because generally for quantization, for example, you can apply quantization to pretty much anything at int8 for both activations and weights and have it recover. But there are definitely cases where, for example, we've been quantizing efficient nets quite a bit on our image classification side. There's one or two layers in some of these that are extremely sensitive for whatever reason to quantization, that you can't quantize those. So removing those, then you get 100% recovery. Right. So it's a lot of these kind of little things that researchers know intuitively in terms of having used this constantly, what will work, what won't. But that's not really coded into software anywhere, right. To make it easy for people to use. So generally they'll go through, try and quantize and there's no feedback loop, there's no methodology, it's just, hey, I was able to apply it in one shot, but it lost 5% accuracy talking about quantization. But if you do, you know, a quantization or training scheme, generally you'll recover all of that back and it generally works completely for that same thing. On pruning, pruning, you'll definitely see more of a drop and it's much more of a requirement to do training aware on the pruning side at least to get to really high sparsities. But you definitely will see this kind of the choices that are made in the hyper parameters that are chosen, those can significantly affect the recovery and the quality. So generally I'd say, you know, if they were seeing drops in performance, it's primarily because of those choices and those issues and just the wide breadth that's available right now and not knowing how to narrow it down and that's, you know, what we're actually working on. That makes sense.

Yeah, yeah, that's good. For people to myself, I want to develop a little bit more intuition around these things because like you say, sometimes you just like, oh, here's the command that I run on the command line and like I get this file out that is smaller. Right. But I don't have a great intuition about like similar to hyperparameter tuning. Right. Like that takes time to figure out, like, okay, how should I think about changing my learning rate if this happens or if that happens, that sort of thing you mentioned. One thing which I think would also be good to kind of clarify and help people understand is like training aware optimization versus just non train, I guess optimization or I don't know what the counter to that is. Could you talk about those? Like how they're differentiated? Some people might guess what that means, but like how are they differentiated and how does that work out in practice? In terms of how you would optimize.

A model technically we have three categories generally available. One, and the two you're going through is one training awareness and then we have post training or one shot, which are kind of interchangeable. And then we additionally have on sparse transfer, which is something that we've been pushing a lot because the research has worked out quite a bit for it. So I'll cover all three of those in a little bit more depth. So for training aware, what we're doing is taking the exact same model that you're wanting to deploy in the exact same data set it was trained on and continuing the training process further. So while we're continuing that training process, we could be continuing it. This is where the hyperparameters come in, but generally it'll be about half the time that originally took to train it. We'll train it for that much longer. And as we're doing that training or iteratively pruning away or we're applying quantization or both. And the reason we're continuing that training is because as we're iteratively applying these optimizations, we're slowly moving the model away from its local men. Right. And it has to adapt and adjust back. So by slowly doing that and training over some time, we allow small jumps that the optimizer can recover from and adjust the remaining weights for, rather than doing it all at once. And the all at once piece is where we get into post training in one shot where we're not going to try and retrain the model at all. We're going to take a small calibration data set and then we're going to use some heuristics or Some heuristics or algorithms to figure out using that calibration data set, how to optimize that model and remove weights or quantize. So the most common case would be static quantization where you're using a calibration data set to figure out the activation ranges for each layer. Right. And once you have the activation ranges for each layer, you can set up a simple quantization scheme to say, given that it's going from this layer is going from negative six to six. Now I need to fit that range into an intake scale of 0 to 255, right. And be able to map that in. So that would be a simple post training or one shot application. And then the final one is sparse transfer, which works exactly the same as transfer learning or fine tuning. It's just we're starting with a sparse model to run through and that's a lot of what we've pushed up in neural magic into our sparse zoo are these open source sparse models that we have, you know, sparse birds and reset 50s and YOLO v5, things like that. And you can just take those, plug in your data set and transfer over to it so the sparsity math stays in place and it just adjusts the remaining weights to fit your data set. We have a few papers out on that as well that shows that sparse transfer works just as well as regular transfer.

Yeah, that's really interesting. I guess this would somewhat depend to like, if I'm just thinking of like the average practitioner out there, right. Probably in a lot of the space that I work in, in like the larger language model area, then like I'm not going to be able to retrain one of these large models on the original data set, right? Or even half of that or for half of the epochs or whatever. But these other things are certainly things that I do all the time, right? Like fine tuning transfer learnings. So it's cool to understand that there's options out there. Am I correct in assuming that like you mentioned the zoo, the sparse zoo that people can find on your website and we'll link in here too, that like researchers, your team, practitioners, whoever are the people out there are also putting in work to actually release some of these sparse models publicly to the community so that I can take those and then maybe do fine tuning on that, or maybe it's just good enough from what's released in the community. Could you tell us a little bit about that community and like what's being released?

So on the open source side, pretty much everything that we have on the sparseu currently has either been from our lab, we have a few that are from Intel's lab up as well and some hugging face examples and things like that. It's primarily because a lot of the sparsification research is all built around a few models like Resnet 50 Bert and things like that. And they don't expand it out past those models to prove out their algorithms. So we have the best of those. And then yeah, that's exactly what our team is working on. And well, we get some community contributions in every once in a while for sparse models that people have generated or transferred. Right. So our goal is to be able to push these up so that as you said, anyone can come in from the community and be able to pull those down and get value out of those models. So you can think of them as sparse foundational models rather than the dense foundational models.

Mark, in addition to the sparse zoo, which is really cool, I know that Neural Magic is producing some pretty interesting and useful tooling otherwise as well in terms of actually doing some of this optimization themselves. And I see certain things deep sparse, sparseML. Could you describe a little bit about. If I'm a practitioner, I have a model and I want to do optimization. What does that look like for me right now with the tooling that's available?

So we have sparse ML, which is our open source model optimization framework. It's built primarily on top of Pytorch. And then we have integrations with torchvision, hugging face, YOLO, ultralytics, YOLOV 5. Pretty much all the common repos that most people are using we've already integrated with. So you can use our integrations and just plug in your model and go along with that. And then the other part is that we have recipes and I'll go through recipes in a second, but we have those kind of pre coded integrations. Or you can create your own integration. Usually it only takes a few lines of code. We've done all the hard work in terms of making sure that whenever we want to optimize a model that you just have to wrap the optimizer in Pytorch, essentially wrap the model and the optimizer, which is what our code handles, and then it's going to go through and handle the optimization past that. So there's no coding really from your side or from the practitioner side on implementation. And then the other part is then coming up with an optimization recipe that people want to use. And what that's going to lay out is saying that I want to Prune from this epoch to this epoch, for example, and then apply quantization and target these layers. And at this sparsity level, things like that. We have automated ways to generate those as well as examples on the sparse sew and in other places. That's generally what it would look like is, you know, get up and running. We definitely recommend checking out the SmartSU first, seeing if there's anything that you can do to just transfer the model onto your data set, because those are the quickest and fastest ways. And then otherwise, if you do have a specific model architecture that you're looking at, then you can start going down this integration pathway and generating your own recipes. And that's the current state that we're at. The other thing that I wanted to call out is that we're working on a SaaS platform right now to make all of this more intuitive. So you have a UI to be able to predict where that model's gonna end up before you start optimizing it and then additionally actively benchmark it across your different deployment scenarios. So this is called sparsify. We've had an old kind of alpha state for a while that was downloadable and we're actually going through alpha testing right now. So anyone that is interested in trying that out, definitely reach out. We're going through alpha currently looking to go to beta in probably next month to two months and then ga following up after that. So definitely check that out as well.

And yeah, yeah, that's great. Yeah. I love how you're thinking towards usability as well around these things because I do see this as something that kind of blocks people on optimization a lot because they get, they get stuck. Like you say, it's like, well, what, what recipe do I use here? It seems like there's a million options, like where do I start? So that's really awesome to hear. Just so people can understand, like, there's options available in the sparse zoo. I'm kind of scrolling through there now. There's a lot, even ones that I know that I use like Distilbert, which is Fine tune on Squad. Like I use that all the time for question answer and apparently I should use the sparse one because yeah, that would help out a lot, both in terms of compute and speed. But let's say that a model is not there and in particular, like people of course are interested in like all of these things being rapidly released all the time. So one of the things I know that I've seen in optimization platforms over time is like, it's hard to maybe support new architectures as they come out. So how are you all approaching that and like, what is the state of like, being able to be flexible with these optimization schemes for a variety of architectures?

Looking at the base framework that we have pushed out in SparsML for example, everything's implemented such that it's supposed to be able to run with any model architecture. So there's no real big assumptions on there other than you have convolutions and you have linear layers inside of your model somewhere that we can target for optimizations. So everything's set up very generically, which means that whenever there is a new architecture that comes out or new weights, things like that, it's going to take some time for us to tackle that. And that's if it gets onto the list of top used models, things like that. But that is the nice thing about having an open source community is that people are welcome to come in. The tooling and the framework there should work out of the box with everything. And they're more than welcome to be able to commit or push up, you know, whatever they're working on. And we have an active slack community and GitHub community for people to come in. Our engineers are actively on that, so they can come in, look at it, and easily get support on any issues they're running into.

Awesome. Yeah. And we'll make sure for the listeners who are wanting to get plugged into this, we'll make sure and include the links to the, the Slack group and the GitHub in our show notes. So make sure you visit there and get plugged in and start optimizing your models. As we kind of get a little bit closer to the end here, I'm wondering about a couple of things. One is, I know that like you're saying you're actively involved in research in this space. What trends are you seeing in research around optimization and in particular, like, what are the directions that your team is sort of excited to go into in the near future in terms of the research side of this?

So I would say there's two big trends right now. One is around the focus on post training. Our principal research scientist, Dan Alastair, he came him along with his lab, came up with this algorithm called OBC obq, which actually we have a webinar which will have already aired by the time that this comes out. It's airing tomorrow. But that algorithm specifically is where a lot of effort's going around, which is requiring as little data as possible and no retraining and trying to increase sparsity as much as possible. So that's one of the key things that people are trending down, is trying to do that. The second one I would say is a large push around quantization in terms of getting to lower bids. And that's something that's been around for a while, but it's something that's becoming more and more prevalent as we're looking at the larger models, mainly because their execution time is mainly dominated by trying to pull in these large matrices of weights. So there's been a big push there now around trying to get down to, you know, pass NA down to N4, N3 and 2 quantization on these active representations. I'd say that's the other trend, the final kind of bonus trend that I throw in there because I said to the third one would be more research around sparse training. So specifically trying to figure out how to start with an unoptimized and untrained model and be able to make it sparse from the start and then keep sparsity throughout training. Because generally what we do to guarantee accuracy, we'll start from a dense converge model and then iteratively print on top of that, which adds training time. Right. So now there's a lot of research going into trying to figure out how quickly the model can be pruned and then be able to carry that over as training. So say that's where the other big, big piece is. And all three of these are definitely active areas that we've been investing heavily in, especially looking at the generative AI space now going through that.

Yeah, yeah, I know it must be a crazy time for you all, just like it is a crazy time for everyone. But yeah, I think this is a really important piece of it. I know one of the trends we've even talked here on the show about, it seems like a lot of people are talking about like serverless deployments of machine learning, deep learning models. And I know a lot of the issues related to that and things that people are dealing with is cold start time and loading models into memory. I don't know if that's impacted you all at all, but it seems definitely relevant. Like if you're going to run your model serverless, you probably want it as small as possible, I would imagine.

Yeah, absolutely, absolutely.

So you mentioned where people can find out about neural magic on Slack, on GitHub. I would really encourage people to do this as we close out here. What are you kind of personally excited about during this? I mean, like I say, it's a crazy time for Everyone right now with generative AI and the way things are trending, what's exciting to you right now about the AI community and certain things you're seeing, what do you see as kind of positive trends?

I guess the part that I'm most excited about is that generative AI space, specifically in being able to augment humans. Obviously there are a lot of privacy concerns and data concerns and bias issues and things like that in this, which I don't want to see, you know, LLMs deployed everywhere becoming the default response for like Google Search or something like that. But it is really exciting to see, even in my day to day, starting to use these actively to augment what I'm doing around content generation and framing and things like that. So it's one piece that I'm really excited for. And with the work that we're doing in Neural Magic, we're especially looking at these because one, we want to see that continue to grow to open source. And I think that's been the other push that's been really big and really exciting to see is that whenever GPT4 came out, it was completely privatized. They put out, you know, a little white paper on it that had no details about it at all, a lot of data concerns and things like that within that. But the open source community has already released, I mean, I can name probably 10 models so far that have been released since then that are chat GPT like or GPT 4 like. So it's really exciting to see that. I think the next stage from those open source models is going to be making them runnable anywhere. Right? So you don't need this big GPU cluster farm to get something that is usable. And that's where we're really looking at going. We're actually working on the LM deployment issue right now and hope to have something out in the next few weeks, the next few months that people can start actively using, download it, they can run it anywhere they want on any CPU and it'll be just as fast as GPUs.

Cool. Yeah, well, keep us posted. I know, I'm personally interested in that one. So yeah, thank you so much for joining us, Mark. This is a really fun conversation. I love getting into the weeds of these practicalities because this is, this is a topic where people get stuck. A lot is on the deployment side and the optimization side. So yeah, thank you for all that you and your team are doing at Neural Magic in this area. And yeah, keep up the good work. We're excited to see it. So thanks for joining.

Thanks, Daniel. It's great talking with you.

Thank you for listening to Practical AI. Your next step is to subscribe now if you haven't already, and if you're a longtime listener of the show. Help us reach more people by sharing Practical AI with your friends and colleagues. Thanks once again to Fastly and Fly for partnering with us to bring you all Changelog podcasts. Check out what they're up to@fastly.com and fly IO. And to our Beat Freakin residents, Breakmaster Cylinder, for continuously cranking out the best beats in the biz. That's all for now. We'll talk to you again next time.