NVIDIA ChatRTX: Private Chatbot for Your Files, Image Search via Voice | How to get started
LARRUZ walks through setting up NVIDIA ChatRTX, a tool that lets you chat with your own documents and search images by voice using your RTX GPU. The app runs locally on Windows, keeping your files private, and supports different AI models.
Chapters
If you have an RTX series GPU and you're currently using this on a Windows PC or laptop and want to use it for chat with your documents, Nvidia Chat RTX is probably one of the best options. It's Nvidia's own implementation of retrieval, augmented generation or chat with your documents. And it leverages the latest Tensor RT LLM along with RTX acceleration. This application was released earlier in the year, but now they have made some interesting updates. It not only lets you chat with your documents, including text, PDF files, documents and images, but you can use voice chat to interact with ChatRTX. And now it even lets you search photos and images on your local machine. And all of this is happening with the help of local LLMs and vision models. In order to run this, you will need a Windows machine that has an RTX 30 or 40 series GPU with at least 8 gigabytes of VRAM. You will need a system RAM of at least 16 gigabytes and the driver needs to be 535.11 or later. The application itself takes about 12 gigabytes. So let's download this and set it up and let me show you different features of ChatRTX. So I downloaded the file. It takes about 12 gigabytes of disk space. Now when you start the installation process, it will download ChatRTX Mistral 7B model which is the LLM that is going to be using. And it will also download Whisper for speech to text conversion. Next, just follow the instructions on the screen to complete installation. Once the installation is complete, you will see this new icon on desktop called Nvidia ChatRTX. So just double click on this and this will open up a new window. Okay, so this is the interface that you are going to see. Now by default it's going to be using Mistral 74 bit quantization. You can also download other AI models. So there is an option available to download clip or you can Download this chat glm3 which is a 6 billion parameter model. Now there are two options. Either you can select AI model which is default right now or you can select folder path. You can provide Folder path where you store your documents and this model will be able to do rag on those documents. So let's first look at just starting with the with the model. For this we're going to select AI model default and let's ask this model some questions. So we're going to start with what is the distance between the earth and the sun? Now this is the actual speed of generation which is pretty amazing. It's extremely fast, at least on the Windows. I have seen that this is much faster than something like Ollama or LM Studio. And the reason is that it's using the Tensor RT LLM along with RTX acceleration. So the response is the average distance between the Earth and the sun is approximately 93 million miles, or about 149.6 million kilometers. All right, so Bing gives us exactly the same response. So it seems like Mistral is right. And here's the response from Google. Another good feature is that you can use voice input. So if you click on this microphone option, it will start recording for 32nd. What is the distance between the earth and the sun? Okay, so when you click on Stop, this basically converts your speech into text and then you can send it exactly the way we did it before. Now it would be great if they can implement both device input as well as voice output. That is going to be pretty cool. Okay. Okay, now next let me show you how you can use this tool to chat with your documents. So we're going to clear chat history and in this case we will select a folder path. Okay, now to test the rag capabilities of ChatRTX, I created two dummy files in this folder called docs. One is an employment agreement between a person named John and a company called awesome AI. The second one is a lease agreement between John, who is going to be the tenant, and a company called Royal Court Apartments. So let's see if we can retrieve some of the information which is present in these two files. Now in order to do that, you will need to provide your folder path. So you can click on this to change the folder path if you need to. But we're going to stick with the folder path that I have selected. After that, just click on this icon to refresh your data set. So this will basically create chunks out of those documents and compute embeddings. Now everything is ready, let's see if it can do retrieval. Alright, so I'm going to ask is what is the start date of employment? And let's see if it can actually retrieve that information. So it says the start date of employment is May 15, 2024, which is correct. And it is able to reference to the correct document. So this is pretty cool. You can actually click on that small link over there and this will open up the document for you. So this is pretty neat. Alright, next we're going to say what is John's annual compensation? And let's see if we can get the correct answer. Yeah, the answer is correct. So the retrieval is actually pretty good. Now let's ask is what is monthly rent? Now in this case it has to look at the second document and it says the monthly rent for one bedroom apartment located at the address we actually didn't provide an address in the lease agreement is 2500, which is correct. So this is pretty neat. It's able to retrieve that information. So I asked a little more complicated question. What is the monthly rent and what is John's annual pay? Now it got the answer right. So it says the monthly rent for the one bedroom apartment is 2,500 and John's annual pay is $350,000. But for some reason it's just picking up only one of the files, not both of them. Okay, now you can try these other models. So for example, I selected the ChatGLM model and then click on download. So it started downloading that model. So this downloaded the model file and now in order to install it, you need to click on install. This will basically install the model and reset your chat app. So the installation process can take some time. But once the model is downloaded and installed, you're going to see it here in the list. Next we're going to test the clip model that is available within ChatRTX. So I downloaded the clip model. We'll click on install and again it shows me this message. Proceeding with installation will change the model, reset the chat and activate the new model. So we'll have to wait for this to complete the installation process. So now we have three models I'm going to select. The clip model will reprocess everything. Now the directory is structured a little different. So I have this main directory called documents and within this I downloaded some images from Internet. So here's an image of some llamas, then a skydiving, some skiing images, and then there are a couple of images of cats. Okay, now let's see if I ask, it will show me images of cats. Let's see if it actually can pull those up. So okay, so it actually simply figured out where the cats are in the images and just pick those images. One capability that they highlighted was photo and image search. So let's see if we can use this clip model to search for specific images. Okay, so I'm going to say show me images of cat and let's see if it actually can pull those cat images. It says here are the top matching pictures from your data set. So it's just looking at that data set and it picks both of those images of cats, which is pretty good. Now next I'm going to ask it to show me images of a person skydiving and let's see if it actually can pull those. Okay. Yeah, this is this correctly identified both the skydiving images and all right, now let's ask it a tricky one. Are there any images that contain dogs? It probably will confuse some of those images. Yeah, so it actually incorrectly retrieved images of cats and shows us those are top matching images. This is actually a pretty neat feature. So you can have all your images dump in here and then if you want to recall a certain image or retrieve that image. So rather than searching for it, you can just type it in and the clip model will try to figure out the closest match that it thinks. So this is a pretty good feature that you can actually use. Now overall I think it's a really cool idea because this lets you chat with your documents. You can also do image search and it has this wise chat ability. Now as I said, currently it's only available for Windows but I think it would be actually very beneficial if they can implement the same thing for Linux as well. Because I think most of the people who are using these Nvidia GPUs for LLMs or AI related tasks, they are probably using Linux because it's hard to do AI related things on Windows though ChatRTX can help the make it easier and it will also be pretty cool if they can implement bidirectional voice chat. So chat input or voice input as well as voice output, that is going to be pretty cool. Now it's also pretty cool that the hardware manufacturers are actually building these software tools around their hardware. That way you can get the full potential of the hardware that you are running on. So if you have a Windows machine with an RTX gpu I think it's worth trying out. Let me know what you think about it, how your experience looks like. Thanks for watching and as always, see you in the next one.
