Rise of the AI PC & local LLMs

Foreign.

Welcome to Practical AI if you work in artificial intelligence, aspire to, or are curious how AI related tech is changing the world, this is the show for you. Thank you to our partners at Fly IO, the home of changelog.com Fly transforms containers into micro VMs that run on their hardware in 30 plus regions on six continents so you can launch your app near your users. Learn more at Fly IO.

Hello and welcome to another fully connected episode of the Practical AI Podcast. In these fully connected episodes, Chris and I keep you connected connected with everything that's happening in the AI world and hopefully share some resources with you to help you level up your machine learning game. My name is Daniel Whitenack, I am CEO and Founder at Prediction Guard where we're safeguarding private AI models and I'm joined as always by Chris Benson, who is a principal AI Research Engineer at Lockheed Martin. How you doing Chris?

Doing great today, Daniel. How are you?

Oh, I'm doing well, yeah. Yeah, lots going on, lots of fun stuff in the AI world. But Chris, I have to say so one of our listeners pointed out something which I realized afterwards as well. On our last Fully Connected episode, the one about GPT4O or Omni, we played some clips of the voice assistant and we talked a little bit about the model and the model of course was released and we were using it, but the voice back and forth wasn't yet plugged into GPT4O. So so they're just, I think releasing that shortly. So I think there was a bit of general confusion in the community about because when you click that button it's like you just take over into the voice interface and it's like I was on GPT4O and so probably could have been maybe handled a bit better on our end and maybe their end too. But OpenAI got us, I guess we were fooled. But yeah, still really cool stuff.

Still.

The things that we talked about are are still what GPT4O is. But just wanted to clarify that for our listeners if they had listened to that previous episode while we're, while we're doing that.

I realized I had made a mistake on that same issue after the show that I wanted to confess on. I had watched all the videos over and over again about the video portion doing that. And then when we were in the show I had, I claim as a senior moment where I was thinking just I watched so much of video, I was thinking, oh yeah, I, I did that. And so it wasn't afterwards I was like, no, that was the videos I was watching. What was I thinking. So anyway, I wanted to confess. So we had a couple of blunders on that one, but having fun.

There you go. In our eagerness and excitement to talk about GPT4O and record it very quickly afterwards, we got got.

So we both got got there. So all good, we'll move on. Well, for listeners who don't know, the show is fairly spontaneous if you haven't been following us long, and we dive into stuff pretty quick and occasionally we fumble a little bit with that.

Yeah, yeah. So thanks for journeying with us through this, this wild and crazy world of, of AI. And now I think, something that's been on my mind quite a bit, Chris, with some recent announcements, but also things that have been kind of developing over the past months is this area of local offline AI and AI PCs. There's sort of people are using this terminology, AI PCs and current marketing hype. Yeah, the current, the current marketing hype. So I thought maybe we could dig into a little bit of, A little bit of that today and talk through kind of what, what exactly does that mean? How are people using AI models locally? What is an AI PC? What are the relevant kind of types of models and optimizations that can run locally? All of that seems to be a big. Well, it's a little bit hard to parse out some of that. Maybe if you're new to the, new to the space and, you know, what is a GGUF versus Olama or, you know, all these things, you know, coming out. So, yeah, I thought that would be good to dig into. And I know that you have been interested in kind of AI at the edge for some period of time. How do you see whether it's a staff laptop or maybe a heavy compute node in a manufacturing plant or something where you'd want to run AI quote at the edge or locally, let's say generally locally or offline. What are the reasons that people would, would sort of want to go that direction?

Well, I think, you know, this is kind of a thread we've talked about off and on, on different episodes over time. And, you know, AI models are hosted in software and they're going to always be wrapped in that. And as we, you know, the software expands from the cloud all the way out into every device that we're already using. And so it's only natural that as AI becomes more accessible and cheaper to deploy, that it's going to, you're going to start having models, you know, that are kind of ramping up existing software out there and there's of course we're going through all the hype that's associated with that, but the way I see it is simply just a natural evolution of where software development would go. And to your point a moment ago, about the hardware side, we don't talk about that a lot. We're very software focused in general, but the hardware side is really going through revolution. I know that you, I know that in your business you have partnership with intel and are seeing that in that capacity and I certainly see that in my day job. And so there are so many more hardware capabilities coming out to support these functions, many of which will function or targeting low power disconnected environments. And so this is a timely topic. And if you look back for precedent before that, we've always seen over the years in software and cloud development kind of a shifting back and forth between local capability as suddenly you get a new generation of hardware and things will go a little bit more local and you'll have your own equipment and then things will move back into the cloud. And that natural give and take is part of the flow. And I think right now we've been so cloud focused the last few years because that was really the only available option that now we're seeing a lot of new capability rolling out on both hardware, software and models that are going to enable edge functionality to really explode over the next few years.

Yeah, I was asked, I was at a conference last week and I was asked which direction things would be going, either local AI models or hosted in the cloud. And I think the answer is definitely both. In the same way that there is a place for, if you just think about databases, for example, as a technology, there's a place for embedded local databases that operate where an app application operates. There's a place for databases that run kind of at the edge, but on a heavier compute node that serve maybe some environment. And there's a use case for databases in the cloud and sometimes those even coexisting for various reasons. And in this case we're talking about AI models. So I have a bunch of files on my laptop, I may not want those files to leave my laptop. So it might be privacy reasons that I want to search those files or ask questions of those files with an AI model. So privacy, security type of thing, or in a healthcare environment, they may have to be air gapped or offline sort of thing, or public utilities sort of scenario where you can't be connected to the public Internet, but then it might just be also because of latency or performance inconsistent networks or flaky networks where you have to operate sort of online offline. There's a whole variety of reasons to do this. But yeah, there's also a lot of ways that as you said, this is rapidly developing and people are finding all of these various ways of running models at the edge. And we can highlight if you're just into this now and getting into AI models, maybe you've used OpenAI's endpoint or you've used an LLM API. If you wanted to run a large language model or an AI model on your laptop, there's a variety of easy ways to do that. I know a lot of people that are using something like LM Studio that this is just an application that you can run and test out different models. There's a project called Olama which I think is really nice and really easy to use. You kind of just spin it up. You can either spin it up as a Python library or as a kind of server that's running on your local machine and interact with Ollama as you would kind of an LLM API. And then there's things like Llama, CPP and a bunch of other things. These, these I, I would kind of categori local model applications or systems where there's either a UI or a server or a Python client that's kind of geared specifically towards running these models locally. And then there's a sort of whole set of technologies that are kind of Python libraries or optimization or compilation libraries that might take a model that's maybe bigger or not suited to run in a local or lower power environment and run that locally. So if you're using the transformers library from Hugging Face, you might use something like Bits and Bytes as a library to quantize models, shrink them down. There's optimization libraries like Optimum and MLC, OpenVino, these all have. Some have exist for some period of time actually. I think in the past we've had the Apache TVM project on the show and we talked about OctoML. So this is not a new concept because we've been sort of optimizing models for various hardwares for some time. But these optimization or compilation libraries are also usually kind of hardware specific. So you optimize for a specific hardware, whereas other other of these local model systems are maybe more general purpose, less optimized for hardware specifically. I don't know if you've got a chance to try out any of these systems. Chris, running some models on your laptop?

I have a little bit. I've used Ollama. I think that's my go to.

And you have like an M1 or M2, M3, whatever M there is now.

MacBook I have an M2 that I may. That I have. I have a couple of different laptops, one that's old and one that's. Well, I guess an M2 is old by today's standards, so may have to upgrade that one pretty soon. But. But yeah, I've used. I've used Ollama primarily. I probably haven't used as many of the tools as you have given, given the business that you're in. It's. I think one of the things that I'm really interested as people are doing this now is understanding because we're really focusing on this kind of the infrastructure, the plumbing of making all this work locally and doing the integrations with the cloud. But I think there's another thing just to throw into the mix of the conversation and that is how as people start, whether in the cloud or particularly local, having multiple models out and you have the infrastructure now, you know, as we're talking about to run it, but starting to look at the, the API and the middleware to enable inferencing across APIs without direct human intervention and have it make sense where you have different responsibilities, much like we have had for a number of years in the software realm. So as we're talking about that, I wanted to throw that in as another topic that I think is going to be really, really important and hasn't had nearly as much attention as the fundamental infrastructure.

Yeah, I think that gets probably to a couple things. One is that one current major difference between sort of hosted cloud models and offerings versus local models is likely you're not going to run a mix of 10 different LLMs on your local laptop all at the same time, all loaded into memory. That would be a pretty, a pretty significant, at least right now, a pretty significant ask to kind of switch between models in that way. But there are certainly cases where, you know, I think the market is showing that people want to not be restricted to one model family and they're spreading out their usage against multiple models. So that definitely needs to happen in the cloud and from, you know, model providers. But that doesn't mean you couldn't throw in the mix a selection of local models as well for specific purposes. And I think that gets to the other thing that you're talking about, which is kind of data integration, automation, pipelining, all of that sort of thing. I saw a comment on LinkedIn, I think it was even this morning that those that are really winning in the AI space are those that have taken what they've learned kind of from automation, data pipelining, data integration in previous cycles of data science, and integrated those with generative models at various stages. Because a lot of, a lot of the value, and I've seen this too, a lot of the value that you get out of these models is not the models themselves, but the system that you build around them. And you know, that involves a lot of data integration and automation and maybe even routing between different models in the case that we're talking about here, maybe even routing between local models and cloud cloud models. And so yeah, I think that that's really stressed, especially as you talk about running these models kind of everywhere, quote unquote across cloud and local and on prem and data center environments. But yeah, it's interesting. I don't myself have an AI PC yet, maybe at some point I will. But yeah, I'm excited to see where all this goes.

Indeed. I think as we push forward, I think one of the things that I'd like to see especially for local, whereas you know, we have, you know, we mentioned a few minutes ago and some of the other tools for infrastructure, I was actually, as we were talking here, I was looking through Yann LeCun's various posts because he was proposing recently, in the last week or two prior to this conversation, kind of a way of structuring different model interactions and from a responsibility standpoint that they're doing at Meta and of course at my employer, we have our own version of that on how we structure different things. But really interested in seeing if the community kind of comes around with kind of an open framework, you know, best practice framework around how to do that. You know, be able to do it on a laptop, on an M2, M3, M4, be able to have those interactions locally and a framework that would span between that local and the cloud interaction so that you have something. And I don't think that right now everyone seems to be doing that on their own and I and there's a lot of similarities between them, but there doesn't seem to be kind of a standard approach to how that all comes together.

If you're anything like me, you have a certain tendency to put things off until the very last minute. It seeing the dentist, going to the doctor, home improvements, that never ending chore list of yours. And while most of the time it works out just fine, the one thing in life that you really cannot afford to wait on is setting up term coverage life insurance. You've probably seen life insurance commercials on TV and thought, yeah, I'll look into that later. No, later doesn't come. This really isn't something you can wait on. Choose life insurance through a Ladder today. Here's what we love about ladder and why we allow them as a sponsor. They are 100% digital. No doctors, no needles, no paperwork. When you apply for $3 million in coverage or less, just answer a few questions about your health in an application. Ladder's customers rate them 4.8 out of 5 stars on Trustpilot and they made Forbes Best Life Insurance 2021 list. You just need a few minutes and a phone or laptop to apply Ladders. Smart algorithm works in real time so you'll find out if you're instantly approved. No hidden fees. You can cancel anytime. Get a full refund if you change your mind in the first 30 days. Ladder policies are issued by insurers with long proven histories of paying claims. They're rated A and A plus by AM Best. Finally, since life insurance costs more as you age now. Yeah, right now, now's the time to cross it off your list. So go to ladderlife.compracticalai today to see if you're instantly approved. Again, that's ladder.compractical AI l a d d e r life.compracticalai.

Well Chris, there's an increasing number of options if you were to explore this space and kind of that interaction of local models with your systems. There's an increasing number of choices of quote AI PCs, which I think is a hyped term. Now one of them you mentioned intel, intel coming out with AI PCs. I think Lenovo is shipping some with the Intel's Core Ultra processor. I think the name is Meteor Lake, the code name at least and similar probably to. As a response to maybe what's more familiar, I already mentioned the M1, M2, M3, et cetera line from Apple. But Nvidia is also working hard on like the GeForce RTX AI PCs, which there have been kind of gaming PCs with like GPUs in them for, for some time. But I think most of these quote AI PCs are more of an integrated type of processor or system where, where it's not just like an add on to the laptop. But in the case of like the Core ultra or the M2, there's actual processing in the architecture that is optimized for executing models and so they're sort of AI ready, shipping AI ready. And that brings up kind of some interesting questions in my mind which are, well, how do all of these AI PCs compare if I'm about to get myself an AI PC, where should I go? And in thinking about that and looking at some of these benchmarks, I was really encouraged to see that ML Commons, which is the organization behind MLPerf, which is a set of benchmarks that benchmarks and working groups that have been working for some time to benchmark various systems for performance on running AI workloads, machine learning workloads, they've just announced this spring an MLPerf client working group which is really geared towards essentially an application or a workload that you could run across these various AI PCs or maybe AI enabled edge machines and that sort of thing to really do kind of LLM based workloads and do some benchmarking for both training and inferencing on these quote clients. So they're kind of referring to these generally as clients. So they say the new ML Commons effort will build ML benchmarks for desktop, laptop and workstations for Microsoft Windows and other operating systems. So quite interesting. I'm glad that this, as far as I can tell someone in our. Again, sometimes we get things wrong so someone in our audience can correct us. Maybe David from ML Commons, who's been on the show before, he can correct us if something's already been published. But I couldn't find this sort of set of benchmarks. I think it's a work in progress, but really excited to see this when it comes out.

Yeah, I think it'll be interesting as these laptops come out. It's hard to imagine that the entire industry doesn't have to go full in on this regardless and thus kind of making the distinction of an AI laptop a little bit of a redundant thing because there's a point, and maybe we're already, maybe we've already arrived now, where the idea of purchasing a new laptop that is not an AI laptop is a ridiculous thing. You know, it's a, it becomes a must have feature to have going forward and therefore all laptops kind of have to go that direction at some point here.

Yeah, well, I definitely think that that could be one downside of this whole thing is there's, I mean, I know the prices will go down, but these things are really expensive right now. So there is going to be a sort of disparity of those already for some period of time. If you're a new developer, maybe an indie developer, that purchase of that MacBook is a pretty significant expense for you already. And myself, I just use a refurbishment thinkpad from like four years ago, you know, not an AI PC it's a Core i5 and you know, not a terrible laptop, but definitely not, not anything that anyone would necessarily be jealous of. Now I can run some models on this laptop, you know, using Olama and other, other systems. And I think that that gets down to maybe another element of this. So there's going to be on one side these clients that get increasingly sophisticated and build in more AI enabled or accelerating functionality into their chipset and into their hardware. But there's also going to be increased sophistication on optimizing models that can't run locally, such that they can run locally. And this is where people might kind of. I think this is also a point of common confusion that I've heard in workshops that I've given at conferences and other places where people kind of look at, let's say it's llama 3 or something like that. I want to run llama 3. Well, you go to hugging face the top downloaded llama 3 is the base model and then you've got these fine tunes for instruction or chat. And then you've got all of these other flavors of Llama three, like you know, gguf, ggml, qat, aq, awq, you know, all sorts of like acronyms that are really difficult to understand. So maybe it would help to just break this down just slightly. There's usually when a model is released, they release a base model or the pre trained model or whatever it's called a base model. So that's that kind of shortest name, usually the meta, you know, metalama3. That usually is maybe a good model that you might fine tune off of, but not generally the best model to start with because it's, it's a base model, it's not fine tuned for any sort of set of general instruction following or chat. And they usually release along with that then a set of fine tune models for instruction or chat. So you've got Meta3 or Llama3 instruct, which is usually the better model. And then you've got this whole world of community members out there that build pipelines. So we had noose research on they have pipelines built so that when a model is released they can create all these different flavors of it which include flavors for running these optimized in certain ways. So these would be these other acronyms that we can dig into a little bit. But these are most of the time either additional fine tunes or quantized versions or somehow optimized versions of these models that are meant to be run kind of a diverse set of environments.

Foreign.

What's up, friends? Do you remember when Chat GPT launched? I do. It felt like the LLM was this magical tool out of the box. However, the more you use it, the more you realize that's just not the case. The technology is brilliant, don't get me wrong, but it's prone to issues like hallucination on its own. But there's hope. There is still hope. Feed the LLM reliable current data, ground it in the right data and context. Then, and only then, can it make the right connections and give the right answers. The team at Neo4J has been exploring how to get results by pairing LLMs with knowledge graphs and vector search. Check out their podcast episode about LLMs and knowledge graphs throughout 2023 at graphstuff.fm. they share tips on retrieval methods, prompt engineering, and so much more. Don't miss it. Find a link in our show notes. Yes, check it out. Graphstuff FM Episode 23.

Well, Chris, I kind of started getting into the Alphabet soup a little bit. I don't know if you're sometimes as confused as, as I am with all of these, all of these model names, but they're getting increasingly long.

Yes, you know, one of the. Finish your point there and then I have a question for you afterwards.

Yeah, yeah, sure. No, I was just going to highlight a few of these different quantization methods so that people could maybe have them in their mind. So there's the flavors that are GGML or gguf. You might see those letters. This is GPT Generated Unified Format. That's. That's what that stands for. And this is a, this is an optimization of a model that will take that model that maybe requires a GPU to run as larger and creates a C replica of the LLM and allows you to run it in quantized versions, meaning that the parameters of the model are taken from, you know, numbers that might have 32 or 16 digits behind the period and get that down to 2 or 4 or 8, that sort of thing, which makes the model smaller and more efficient. So these are mostly geared towards cpu, sort of CPU or laptop kind of environments. There's also gptq, which is really a focused kind of quantization method, but it's still meant for GPU only. So these usually ship in similar kind of formats to previous models, but they do kind of some calibration, informed quantization to get that model smaller. There's qat, which is quantization aware training, which as it might sound, involves some actual training retraining of the model to inform the quantization. There's others like AWQ and this is another quantization method. So all of these sort of letters that you see, if you're wondering what those are, those are all kind of referring to these different kind of flavors of the model that might be generated for either running the model in an optimized way locally on a CPU on a laptop, or optimized still on a GPU but in a smaller format that's more efficient.

What are your thoughts on the, on the CPU derivative, what are your thoughts on, you know, performance and capability relative to the its own base model and its GPU siblings?

Yeah, I mean, I think that the reality right now, and then maybe where it's headed, I think the reality right now is the CPU based models. Even you can run some models that are Even, you know, 7 billion parameters or something in some quantized version on a cpu. You're not going to get the same throughput as you would with hosting the model on a gpu. In other words, the kind of tokens per second, the speed at which you're generating will be lower, but it's possible to run those. What you're not going to do is, and this is what I also tell people like the same way that like you're not going to host your microservice for your company on your laptop, you're not going to host some AI functionality for your company on your laptop. Usually that's going to still live in the cloud. But if you have some sort of private use case where something can't leave the laptop, or maybe it's like you're deploying laptops in a disaster relief scenario and there's going to be not that much connectivity. Right. It's still enough throughput to get responses. So if you had like chat over your docs, disaster relief docs and that on your laptop, like it's still enough to get an answer in that scenario and you know, people can push it pretty far. I think the difference is just again, it's not one or the other, it's the, it's the use case, I guess.

Yeah, I think, and just to add to that use case slightly, I think there's also disconnected or partially connected mobile platforms where you can't necessarily rely on the cloud access and you have devices that are out there as well. Just to lump that in, kind of pulling around full circle for a moment, back to the laptops, these AI laptops coming out. Kind of thinking back to, you know, the natural kind of segregation of responsibilities that we have in software development aside from just the AI world, would you imagine that a reasonable level of support in those would be to be able to actually do training on like 7 billion, you know, size models the smaller range that are so, you know, that are. There's so much more of those where that maybe in the not so distant future I'm training a model, it's in that range. My own laptop can handle that. Not just for inference but for training and then for very large models that's still cloud based. Do you think that that is a reasonable level that that we might see AI laptops able to support?

I mean I think that you can see some people trying some more training or rather fine tuning sorts of things on diverse hardware. I think basically what I'm seeing right now is still primarily inference on local machines and utilization of things like in context learning and rag type workflows to integrate data rather than fine tuning locally. So I think that that's, that's kind of the reality of where it is. I think there's a possibility in the future maybe of you know, some type of training type of scenarios that will happen maybe not on client devices but spread across client devices. So you know, it's been a while since we've talked about federated learning but it will be interesting to see if that kind of rears its head in this world. I know that there's been efforts to kind of train LLM adapters in a federated way and there's some papers about this and share kind of parameter efficient updates to weights across different client devices. That seems really intriguing to me. But yeah, I don't know, we'll see where it goes. Maybe I'll be proved wrong but I think I'm increasingly more of a proponent of. Most people don't need to fine tune, you know, a lot of it can be done with rag and chaining and agents and you know, selecting the right models. So I think that, that especially as models get better that will be the case kind of moving forward. But yeah, I am looking forward to getting an AI PC and you know, putting it through its paces eventually.

Yeah, right now I have to get my M2 is through my employer so my next one will be. I'm hoping to be in either an M4. I'm thinking about possibly a non Apple one as well. So the M4s are supposed to, supposed to be able to do quite a bit more than the earlier generations. So looking forward to being able to pursue this?

Yeah, yeah, definitely. Well, I hope our audience will include a few links to some blog posts about these quantization methods and some of the systems like Ollama and LM Studio and others that we talked about. I encourage everyone to to get hands on and and try your own hand at it and you'll get a sense for the performance of these models locally. So definitely give it a try.

All right, well thanks a lot Daniel. That's great information today. It was good show and thanks for bringing that.

Yep, we'll talk to you soon.

All right, that is Practical AI for this week. Subscribe now if you haven't already. Head to PracticalAI FM for all the ways and join our Free Slack team where you can hang out with Daniel, Chris and the entire Changelog community. Sign up today at PracticalAI FM Community. Thanks again to our partners at Fly IO, to our beat Freaking Residents Breakmaster Cylinder and to you for listening. We appreciate you spending time with us. That's all for now. We'll talk to you again next time.