In this tutorial Daniel Bourke demonstrates NVIDIA's Chat with RTX demo application, which runs a local large language model (LLM) on an RTX 30/40 series GPU and enables Retrieval‑Augmented Generation (RAG) over user‑provided documents. He outlines system requirements (Windows, RTX 30/40 GPU with ≥8 GB VRAM, driver v535.11), the installation process (download 35 GB zip, unzip, update driver), and the automatic download of two int4‑quantized models (Llama‑2‑13B‑int4 and Mistral‑7B‑int4), explaining the trade‑off between parameter count, storage, and performance. The demo shows how the app scans a folder of PDFs (e.g., the Attention Is All You Need paper, a nutrition textbook, a personal novel) to split text, generate dense embeddings, and cache them for fast retrieval. Users can then query the LLM in a chat interface, receiving answers sourced from the relevant document with citations. Daniel highlights limitations such as lack of LaTeX rendering, and suggests extensions like building a custom RAG pipeline with TensorRT, Gradio, or replicating the workflow from his Colab/GitHub tutorial. He also points viewers to NVIDIA’s blog post on RAG, the upcoming GTC 2024 event, and invites questions for future videos.
Chapters
Hello everyone, Daniel here. In this video, I'm going to show you something really cool, and that's the Chat with RTX application by NVIDIA. So you can click download now if you want to get started. Just beware, it is a 35GB download. It took about an hour or so on my computer. There are system requirements if you have Windows platform NVIDIA GeForce RTX 30 or 40 series. So on my PC, I have a GTX RTX uh, sorry, 4090 GPU or an Ampere or Ada generation GPU with at least 8GB of VRAM. So on mine there's 24GB of VRAM available. You can check this out on your PC with the command NVIDIA SMI. So you'd be looking for 30 or 40 series and seeing how much memory you have available there. So unclick download. I've already downloaded this. So I'm not going to run through that. But Chat with RTX, it's a demo app, so it's not fully fleshed out, but it just gives you an idea of a pretty good proof of concept that lets you personalize a GPT-style large language model or LLM connected to your own content. So I'll show you what this looks like in a second, but it works by using a retrieval augmentation, augmented generation technique or RAG. And NVIDIA have a fantastic blog post on this. I also have quite a lengthy tutorial on Google Colab and GitHub. I'll link those below. And if you want to replicate the application yourself, it's based on this workflow here: trt, uh, Tensor Runtime, LLM, RAG, Windows. So that would actually be a really good project if you could replicate this kind of application yourself. Uh, if you're looking to be an AI engineer or ML engineer, This is really cool to try and work on replicating for yourself. But anyway, let's say you've downloaded this, right? You've gone through the requirements, you're gonna get a zip file in your downloads. So I'll just go to my downloads, zip file, extract all, then you go into here, Chat with RTX, go through the setup steps. Actually what you'll need, what I had to upgrade is the driver, and I could do that through the app GeForce Experience. Experience. So you need the latest version of the driver. So I took some more notes here on Notepad. So download, unzip, install, and yeah, be sure to upgrade your NVIDIA graphics driver. You need version 535.11, and I didn't have that, and so my installation failed. I upgraded it, started the installation again, and then it started to work. Now while you're going through the setup steps with the Chat with RTX app it's going to download two models, Llama-2 13B and Mistral-7B. The 13B and the 7B denotes how many parameters are in each of those models. And 13B is 13 billion. 7B is 7 billion. Generally, the more parameters an LLM has or any machine learning model, the better it performs. However, all of those parameters come at a cost of computation and storage. What int4 means is that those models have been quantized into the int4 data type. So they take up less storage and they are generally faster to compute on. However, this comes at the cost of slight performance degradations. However, in my testing, I've noticed that these models still perform quite well. Once you've gone through all the install steps, you can just type it in here, Chat with RTX, in search. That'll open it up. It will also open up by default after the install, and you'll see it loading Mistral 7B Int4. Llama2-13B-int4, then some example questions. Beautiful. Once that loads up, we've got a web server here with Chat with RTX. We can select our model over here. I'm going to leave this at llama2-13B-int4 because generally models with larger parameters work better than models with smaller parameters. But this is— take that with a grain of salt. It's not always the case. And here's the fun part. We can chat with the model and it's going to run on our local machine. Now we can ask it a, a general question because LLMs are trained on internet data. So give us a breakdown of the macronutrients, carbohydrates, proteins, fats. Now what if we wanted to ask a question of our own data? So I've got an example folder here of nothing, and that has nothing in it on purpose just to show that it's not giving us any sources. But then if I change the folder path, if I go into my Documents folder, I have one here called Textbooks. Now it's asking for a folder path, not necessarily a file. So I'm gonna select that. And then we're gonna see a few things happen in the terminal. It's gonna start to generate some new values. I'll just show you what my Textbooks folder looks like while that's loading on the left there. So textbooks here, I have, they're not all textbooks, but they're all PDFs. So I have the Attention Is All You Need paper. I have a novel by myself, which is called Charlie Walks. I have a paper on embeddings. I have a human nutrition textbook, which is about 1,200 pages in length. And I have the machine learning for, oh, sorry, Mathematics for Machine Learning book. So what it's going to do is it's going to go through all of these files, split them up into small chunks of text and generate embeddings. So this is part of, uh, RAG pipeline, Retrieval Augmented Generation. So that's going to generate some embeddings. I'll just show you. Here's the Attention Is All You Need paper. So this is a great tool for if you were studying something. You could just put all of your study materials in a folder in PDF, or I believe it also works with doc files, text files as well. There's the embeddings paper, which is about 80 or so pages long by Vicky Boykus. It's open source as well. And we have a human nutrition textbook. I've downloaded that as a PDF. Mathematics for Machine Learning. And then Charlie Walks, which is my novel. All right, so this is about 300 pages of PDF. So rather than searching through all of this, With keywords, we can now chat with our documentation. So if we come back here, we'll have a look at the terminal. All the embeddings have been created, and you'll notice a file in your documents, textbooks vector embedding. So that's going to cache all of these embeddings so that the next time you open up Chat with RTX and select this file, you're going to have those embeddings ready to go. So how about we ask some questions? So we've got, what are embeddings? Let's ask that. What are embeddings? I'd be happy to help. Based on the context provided, term embeddings. Embeddings are dense representations of deep learning model input and output data that are used to compress information. That is exactly what we're doing. We're compressing the information in our textbooks folder and improve efficiency of machine learning computations. The reference file is embeddings.pdf. How cool is that? So that's that embeddings file there. And now what about we choose another one? All right, we'll go, what is the attention mechanism? The attention mechanism is a way to hold the state of the entire input by continuously performing weighted matrix multiplications. Okay, that's from the Attention is All You Need paper. So if we go attention Do we have mechanism? Attention mechanism. Beautiful. So we can chat with this PDF. Now what if we wanted to find out what matrix multiplications are? What are matrix multiplications? Or what is matrix multiplication? Matrix multiplication is a binary operation that takes two matrices Okay. As input and produces there another matrix as output. It used the reference file of the machine learning mathematics for machine learning. Do we have— if we go matrix multiplication, is that in here? Yes, it is. So that's a little limitation of this application as it stands. Chat with RTX is that it doesn't necessarily display LaTeX or LaTeX, which is to denote mathematical notation. And then what if we were to keep going? So what if, what is Charlie's secret project? Oh, is related to using technology to detect skin issues and prevent skin cancer. Oh, that's, that's sort of close. Who does Charlie work with? That's not exactly perfect, but it's still close. Steve. Okay. Eric, Brett and Aaron, Amber. Wow, that's pretty good. That's pretty good. That's from my novel, Charlie Walks. So, that's really cool. And then, what if we do one more? The question we asked at the start. What are the macronutrients and what are their functions in the body? How's this gonna go? That's pretty quick. Now, this is all running locally on our own GPU. References human nutrition text.pdf. How cool is that? Now in the NVIDIA demo, they also showed that, that you could use a YouTube URL. However, I don't really have that available. We can chat with the model on its own, or I've just got the folder path as well. Now this is a really powerful tool and it's all running locally for free on our own GPU. So if you wanted to, you could just put in a whole bunch of study notes, or if you're working on a theater play or something like that, if you're working on your own novel, you could put this in here and you could just start to chat with it, and it will use an LLM along with RAG to produce a chat-like interface. That's so cool, right? So if you have an NVIDIA GPU, you have a local version of ChatGPT that you can use with your own data. So if you thought that was cool, be sure to check out Chat with RTX, download it, run it on your own computer, see how it goes with your own data. And if you want to learn more about these kinds of techniques, I'll link NVIDIA's blog post on Retrieval Augmented Generation, as well as my tutorial on setting up an own— coding your own RAG system from scratch on your own machine. And then a really cool project would be to replicate something like this. on your own. You could use Gradio, which is a fantastic application for creating machine learning demos, and they have a chatbot interface that you might want to try out as well. One last note, NVIDIA are hosting a GTC event from the 18th of March to the 21st. If you want to learn more about all of the amazing AI things going on in NVIDIA, be sure to check it out. There's a GTC 2024 keynote It's all free to attend virtually. There are a lot of cool sessions, um, such as best practices for building LLM RAG using NVIDIA AI. But if you have any questions or you'd like to see a video where we build an application like Chat with RTX in the future, let me know in the comments below. Otherwise, I'll see you in the next video.
