Local Multimodal RAG Pipeline End-to-End Tutorial | On DGX Spark

Ladies and gentlemen, welcome back to the Machine Learning Cooking Show. I am your host Daniel Bourke, and in today's episode we are going to build a multimodal RAG system locally on the NVIDIA DGX Spark. I just want to put a note that this is a quick version, so we're going to walk through quite fast of all the concepts. If you want to see a full walkthrough, see the live streams on my channel. So the link will be in the description. There's a full playlist on my YouTube channel now, which is multimodal RAG. This is where we built all of it. Live. So if you want to, if you're looking at this going, wow, this is a pretty refined notebook, well, it took a few hours to sort of figure everything out. And that is contained in the streams. So without further ado, let's get into it. Here's what we're going to focus on. We're going to discuss what multimodal RAG is. So if you've heard of RAG, it's Retrieval Augmented Generation. I have a video on my channel which we do text retrieval augmented generation. But But now we can go multimodal. We're going to download the NemoTron. So NVIDIA have released two models recently, NemoTron Embed VL. So VL stands for vision language and Rerank from Hugging Face. We're going to download a dataset. We made this live as well on stream, a custom dataset, which is a 10,000 sample dataset of recipe images. So text on those images as well as text markdown. So that's what these multimodal embedding models are for. So that's available on Hugging Face as well. We're going to embed the dataset and save it to file. We'll create a demo for people to interact with, with the multimodal RAG system, and we'll discuss extensions to make our project better. So let's cover what is RAG. Oh, and of course, all the links you need will be in the description below. So if you want to jump ahead, feel free to do so. RAG stands for Retrieval Augmented Generation. And there's a paper here if you want to read the technical specification where it was introduced. But essentially what RAG is, is it inputs custom data to your model's context. So if you've used an LLM like Gemini, ChatGPT, uh, NeMoTron, Claude, you name it, you'll know that when you input a prompt, it can be anything, right? And so a lot of the time to get the best results from an LLM is that you want to put it in the best context. So, you know, if you write in a simple prompt, you might get okay results, but if you write a detailed prompt, you might get better results. So the outputs that the model generates are going to be grounded in your context. So if you want to compare RAG versus fine-tuning a model. So RAG is where you would probably inject specific knowledge into an existing model. For example, you're a large company, say a bank in Australia. That's where I'm from. Such as ANZ, you might have a million documents of company documents. And I'm making these numbers up, by the way. And if you want to create a chatbot for your employees, or a search system, you could definitely search them via text. But you could also search them to be in a RAG system by doing a similar pipeline to what we're doing here. So if you enter in a question, and you want that question to be backed up by specific documentation in your own database, in your company's database, or if you're an app of any kind, you want that information to be backed up to a source. That's what RAG is for. Whereas fine-tuning, if you want an LLM or any type of model to do a very specific task, that's probably when you want to fine-tune. So rule of thumb, RAG for specific knowledge, fine-tune for a very specific task. Now, if you want to read a bit more about a realistic RAG workflow, I'd highly recommend the article here by Martin Fowler. I'll just open it up. It's going to be in Safari. There we go. So this is a realistic RAG pipeline that you might see. We've got a request that comes in. This could be almost anything, but in our case it can be a text or an image. And you've got input guardrails, a framework, a rewriter, an embedding model, vector store, a lot of different parts. We're going to do a simplified version of this is what we're going to see. But I'm linking this to you so that if you want to expand on the project that we're about to go through, we'll probably do mostly this system here. So this one here and this one and then this one out here. So there's a lot of different parts that you could use here. And so that's why I've linked that as a potential extension. Now we'll keep going. This is what we're going to build. We're going to build a multimodal RAG pipeline. And by multimodal, I mean multimodalities. A modality is, say, text, image, or, well, vision and sound, you name it, there's different types of modalities. In our case, we're focusing on text and images because that's what the models have been trained on that we're going to use. We have 10,000 recipe images and the text associated with those in a dataset. Now, I want to stress that this workflow is very expandable to whatever dataset that you have. As I said, if you have a million company documents in a large folder or a large database, you can apply this type of workflow to that, whether they're images or text. Previously, RAG workflows were mostly with text, but these models were very recently released that can do images and text in the same embedding space. So we're going to embed all of our documents with an embedding model, specifically Llama-NemoTron-Embed-VL-1B-v2. Try and say that fast many times. This is a really recent model from NVIDIA, open for commercial use. We're going to store those in a database. In our case, because we're only storing 10,000 embeddings, we can just store that in a single file. Our file is going to be 40 or so megabytes. But if you had, let's say, a million embeddings, you might want to look into something called a vector database. Now, this is more so for a smaller dataset, 10,000, still not tiny, but not giant. If you were in a production setting, you might have a million plus embeddings. And then we're going to retrieve number of samples and we can send those to the re-rank model. So what happens is we input our query— sorry, we embed our data, save it to file. And then when someone comes along and say, in our case for recipes, the query is recipes for tomatoes and Parmesan cheese, we pass that query to the same embedding model. Then we compare those, the query embedding to our database, which is quite fast. That's probably the fastest step. And then we retrieve the best samples and we can either pass those to the re-rank model, which its job is to take, say we had 10,000 recipe images and text, say we have the top 100 or even the top 20, 'cause re-ranking is a little bit more, well, far more expensive than just retrieving information. We would re-rank them, that top 20 to be, okay, here's the query, here's the top 20. Can that order be improved? So maybe the first one goes down to 5. And the 8th one comes up to 1. But we'll see that later on. So that's an optional step. And then what we do after we've retrieved our x top samples, because we can, we can say how many samples we want to retrieve, say we only want the best 3 recipes, we can go, okay, give me the best 3 recipes. Or if we want the best 50, we could also do that. That's a design choice. Then we augment those into our context, into our input to an LLM. So if I go here, to— let's go to Gemini. I just want to give you an example. So compare RTX 4090, uh, versus RTX 5090. So these are Nvidia GPUs on compute capacity for, um, a deep learning PC. And what this is going to do, it's going to start searching the internet for information, right, so that it grounds its results in. There we go. Maybe this will give me some links. Will it? Anyway, the ideal system would be this would go out, read the NVIDIA docs, write its report on that. So the retrieval is go get the NVIDIA docs, and then the augmentation is to put that into the LLM, and then the generation is, hey, here's some information. Yeah, there we go. There's our links. Right. So what it's done behind the scenes is our workflow exactly. Well, an augmented version of it. It's gone, hey, searched for information in a database, retrieve those, augment those into the output, and then generate an answer based on those results. And then we get it back. So that what we're building is a simplified version of this. But the benefit of what we're building is it's going to run locally. So let's go to the next step. Our ingredients, we're going to have an embedding and a re-rank model. These are from NVIDIA. They're from the NemoTron series. Uh, re-rank and embed, very recently released. Um, there's a blog post which released the models and discussed how to use them. So I've put that into an application, uh, which we'll see shortly. And then a dataset which we made live on stream, which is, uh, recipes, which is images of the recipe and text of the recipe. But again, you could use your own dataset. I've only just created this one from scratch to practice. You could use your own dataset of whatever you like, whether it's text or images. These models are quite broadly trained. We also have some tools. We're going to be using the Transformers library. I'm going to be using an NVIDIA DGX Spark. So all of this is done locally. But this notebook will be compatible with Google Colab. So if you need to use Google Colab or something like that, feel free to open it up there and give it a shot. So what are embeddings? Embeddings, in short, are a learned numerical representation of data. So models and computers like numbers. They don't like text like we do. So in an embedding scenario, when you create the embedding, you put the text in, it gets turned into tokenization. So numbers, but then those tokens get compressed even further to an embedding vector. And so multimodal embeddings are not just text, they're text and image in the same embedding space. So that means you can match an image to a text pair. not just text-to-text, not just image-to-image. So if you've ever used a model like CLIP, if we go to— or SigLIP is my favorite, personal favorite, SigLIP-2. So SigLIP-2 stands for, um, language image pre-training. So it compares, uh, images and text and builds a representation of those. So when you can load an image in here, you can go, okay, which text does the image best relate to? And that's basically our problem in a nutshell. If we input a query of text, we're going to get back the recipe which the text most relates to, and vice versa. If we put in an image, we're going to get a recipe of image or text of that the input image most relates to. And so that's the big part here, is we're now multimodal. And for embeddings, size, generally bigger is better. And what I mean by this is whenever you embed something, your vector size will be dictated by by the embedding model that you use. And in our case, our input text and images will be in the same embedding space, and they get turned into a vector of size 2048. So that just means here's how many numbers are going to represent this piece of data. And so the computer is going to look at that and go, hmm, this is quite a good representation. I'm going to compare it to the other 10,000 samples and see which one is the best. Now, it's not actually going to say that, but that's just how I like to think of it. And so these are an example of the first 10 numbers. But of course, there's way more than that, right? And this is what it looks like to us. We don't necessarily understand that, but the computer is going to understand that. But the downside of having an embedding of 2048 is that, one, it takes more storage because each of these values needs to be stored. And two, it can take more compute power. So you just imagine you're comparing 2048 numbers versus 256, which is another common embedding size. You're doing almost 10 times more comparison and almost 10 times more storage. But that size is an arbitrary number. You can decide on what size is best for you, but generally bigger is better. If you want a good resource on where to learn more, this is a great paper by Vicky Boykis on what are embeddings. A bit more of a deep dive to what I've just said, but just think of an embedding as a numerical represent— a learned numerical representation of a data sample. Where data could be image, text, audio, you name it. Um, this is my setup, so I'm running this locally on the NVIDIA DGX Spark. Um, you could also run this in Google Colab as well. But yeah, everything we see and touch here has been coded on this local machine. Um, it's a great little powerful machine for exploring the world of AI, but little doesn't mean not powerful. It's quite powerful. So step. We're going to go through probably 3 major steps, which is one, creating the embeddings. Now, this is generally a one-time process. As you see here, it takes— depending on which embedding space you're using, it takes a different amount of time. So the important thing to note about these embedding models is that they're VL. The VL stands for vision language. So previously embedding models would store text in embedding space, but VL expands that to text and image. And what you'll notice is that when you do— oh, by the way, they can also work on text and image alone and then image and text together. So when you use text only, you get a fairly decent result. When you use image only, you get a fairly decent result. When you combine image and text pairs into the same embedding space, you get a good boost here. All right. Is that going to be world-changing? Potentially not. But it is a good boost overall for a pretty So I've got over here how long it took on the DGX Spark for a different number of samples to embed the recipes. 10,000 samples with only text took about 42 minutes. Actually, this should be updated. Oh no, 42 minutes is correct with a batch size of 8. Image only took about 54 minutes of 10,000 samples. Image and text took about 170 minutes. So just over an hour and a half. Because you're encoding more information. You're essentially combining that with that. And so, but this is, as I said, this is like a one-time thing, as in you create your embeddings once and then you can store them to file and then you can move them around wherever you like. So although this is 107 minutes, we do that once and then we can use them however much we want. And as you'll see, retrieval is very fast. Because when we've embedded our 10,000 samples later on, we're going to use a query. We embed that query with the same model and then we perform a comparison for the 10,000 embeddings with our query embedding and we get back the top results. And so you'll notice this is just with the embedding model. You get an okay result or a great result, 73%. But then when you add the re-rank model, which as I said, takes, say, the top X samples and then reshuffles them to be which would be the best order. You get a boost again. So you go from 73 without a re-ranker to 77.6. So step 1, creating the embeddings. And then we'll go step— oh, sorry. The only time you'd want to recreate the embeddings is if you used a different embedding model. So say, for example, NVIDIA brought out, um, NemoTrom Embed VL-1B v3, which was an improvement on v2. Then you might want to recalculate your entire database embeddings. So just keep that in mind. Embeddings are a one-time process until you want to use a new model. Then we retrieve the results. So say we had this query, which can be natural language, recipes with tomato and Parmesan cheese. You want to cook some Italian type dish, a pasta maybe. These are the actual results we got back from our demo. So I just want to show you what that looks like. So if we go here, this is learnhuggingface.com, notebooks. Multimodal RAG. We have— there's the dataset. I'll get that ready. VL model blog post, re-rank. And here's our demo. So this is the blog post I was talking about, very recent at the time of recording. So published January 6th, 2026, and it introduced two new models which we're using behind, behind the scenes here. And then there's a dataset that we've used, which is if we go through there, We have a look at a random sample or the first one. Not so random. Baked corn on the cob. Oh, what's happened here? There we go. So this is our image because we've just created this as a generic PDF because I just needed a simple way to create a dataset. In practice, you might have different looking documents, but that's what these models have been trained on. They've been trained on document, a realistic visual document retrieval benchmark. So a corpus of 10,000 documents from Digital Corpora. These are what they've been evaluated on that have a good mixture of text, tables, and charts. So if you're thinking of a business document with— so you might have text, figures, charts, you name it. That's what these models specialize in. And so that's what I've created here with a PDF. But then we also have the recipe in markdown, which we're also going to embed. So in practice, in our end, we're going to combine this image here with this text in the same embedding space, and that's going to give us, um, for this sample, this is number 0 or the ID number 1, um, in our embedding space there'll be a 2048 vector of numbers that we saw before representing this particular sample. So in practice, if we go Baked corn on the cob in our demo. Let's try it. Baked corn on the cob. Now we've got some options here. This is just our demo. We could also upload an image. Do we want to generate a summary? Do we want to re-rank? No, we're just going to go straight text query to retrieve results. Let's see what we get back. So there we go. Oven roasted corn on the cob. Baked corn on the cob and simple oven roasted corn on the cob. Right now, of course, you might go, Daniel Bourke, why don't we do just search for that exact text? And that's something that you could do. So you could do vector search, which is what we're doing, but an extension would be keyword search or lexical search. And then you could put those both results in the ones that you return back. So the beauty of vector search is that you can get like specific and go, hey, vegan foods for breakfast or something like that. Let's try that out. Or you can search vaguely, vegan breakfast recipes. Vegan breakfast sausages, vegan gluten-free buckwheat pancakes. Yeah, why don't we do vegan gluten-free breakfast recipes, right? You can get quite specific with these search terms. Vegan gluten-free buckwheat pancakes, vegan breakfast sausages, chewy granola bars, gluten-free, dairy-free. So as we see there, the query embed match time is less than half a second. We've turned off re-ranking and turned off generation. But then if we go back, and we were having a look at the models, this is the model here, Embed VL, and this is the re-rank model. So we go back to the keynote. Oh, we didn't even try our query that I said we were going to try. Let's just search recipes with parmesan cheese and tomatoes. Again, I've focused on food here, uh, because I like food, but you could expand this sort of workflow to whatever kind of documents and images you have. So there we go, we get pork cutlets. Can I expand this a bit? And we get a score. So the higher the score, the better. And then we get the text versions. This is just a nice way to display them in markdown. And then if we wanted to, we could generate some stuff here. And if we wanted to as well, we could turn on re-ranking. So let's keep going back in the keynote. But yeah, running locally, the retrieval time is very quick. So we put in a query, we have L2 normalization. A tip here for normalized embeddings, you use L2 normalization and cosine similarity. For non-normalized embeddings, you use dot product. But when we go match query to embeddings, we input a query, we have some target embeddings to match to, and we return the top K. So if you have 10,000 samples or a million samples in your database, you might not necessarily want all of them back. So in our case, we return the top 100. We just limit it to the top 100 that we want. Oh, sorry, compute cosine similarity. So because our tensors are normalized, we can use the dot product here, which is the symbol there. And then we get the scores back and the sorted indices. But we'll see this in practice in a second. We can also re-rank the results. So as you see, we get back the top 20. But a note on re-ranking, as we'll see later on, is that re-ranking increases latency because re-ranking, you do a pairwise comparison. So you go give it sample 1 and the query and go, give me a score for that. Sample 2 and the query, give me a score for that. Sample 3 and query. And then you re-rank them according to the scores. And so in our sample, we re-rank them, the top 20. And you notice that this one moves up, so up 5 samples, and this one moves up 3, this one moves up 4, this one moves down, this one moves down. So that's the benefit of re-ranking. Now, that's a design choice that you need to make. If, if you find re-ranking really helps out your system, then you can implement that. But if it takes too much time, then that might be something that is not worth the trade-off of the slight improvement. Finally, we generate. So the retrieval augmented generation, we can generate an output. So if we put a prompt in here, say, you're a helpful culinary assistant, right? We're building a recipe chatbot. Please provide a brief markdown summary with details about the following 3 recipes. And then we put in the the top 3 recipes that we've retrieved based on our query. And then the model is going to output a nice summary here of what we want to— what some recipes or the top 3 recipes that match our query. It's going to just output this summary here. But again, another thing to note is that generating an output also increases latency. So if you want to build, that's where you just have to decide whether you want to build a search application or so, just retrieve the best things based on a search query, or do you want to actually add in re-ranking and generation? Just note that those two steps will often increase latency. So tidbits here, but we'll get to that later on. So let's go to the code which I've cooked up earlier, and I'm going to restart this. As I said, you can get this at learnhuggingface.com. I'll put the link down below. There's a book version. This is this, so it's quite easy to read. You can jump through the different sections and see how they're created. If you want to see them step by step created, there's a series of live streams. But if you want to open in Google Colab, you can push the button there. So let's go to this. Say I've got this locally. Resources, there's some links, and then we're going to just start off here. Make sure we have CUDA available. If you don't have CUDA available, everything will likely be much slower, so just keep that in mind. We're going to load the dataset from— I've loaded, uploaded the dataset to Hugging Face. As we saw before, that dataset is available here, 10,000 rows with some metadata about recipes, and we have an image column, the one we're going to pay attention to, which are PDF-like images. So again, if you're in a business use case, you can upload PDF-like images and then recipe markdown. You won't always have image and text pairs, but that's something you can create these days with modern vision language models. So we're gonna view an example sample. This is just markdown. So we have artichoke hearts wrapped in bacon. We get an ID, a time, number of ratings, description, appetizer. And then we have some steps and then we have some reviews. And so if we look at this image, very plain image, right? But it's got some charts in there. Um, so yeah, it looks, looks very businessy, right? And that's what these, these sort of models are focused on, is documents which have text, images, charts, tables, you name it. And then we can go here, we can get all of our samples in Markdown because we have a Hugging Face dataset. So let's just go dataset train. There's 10,000 samples there. And we can access all of the recipe markdown in columns by just doing that. And there we go, number 42. So that's all of our recipes in markdown. So we're going to load the models. Now I've got a lot of these settings from the documentation page, modality to tokens. So if we go to here and then we go modality. So every modality has a different token length. So if you notice image, so the p_max_length of number of tokens, so an image can use 2048 tokens. Text, the max length of text input can be 8092, and the max length of image plus text can be 10240. So I've just created a dictionary to hold those different values, and then we're going to load our embedding model and re-rank model. Now, just one note here. I've added these commit hashes here because I was trying to run it on the DGX Spark and I don't have Flash Attention 2 installed. So I've— there's a pull request here from Oliver Hallworthy, which has enabled people to use— if we go here— SDPA. So that's scaled dot product attention. If you have Flash Attention 2, don't worry about this too much. but I don't. So I'm going to use SDPA, Scaled Doc Product Attention, or Eager. So this pull request enables that. So I'm just pulling the model from this particular code file, uh, so that I can use SDPA there and SDPA there. And then the same thing goes for the re-rank model, because behind the scenes both of these models use the same model file. So let's load that. It's going to take about a minute or so, uh, to run, so I'll let you I'll pause the video there and I'll restart once it's loaded. 5, 4, 3, 2, 1. Okay, so there we go. We've loaded the embedding model and— oh, sorry, the preprocessor as well. So embedding model, processor, re-rank model, processor. And on my system, it took about 1 minute and 7 seconds. That is on the NVIDIA DGX Spark. So if we have a look at how much memory is being used at the moment, it's about 6 gigabytes, um, so of GPU memory. So if we have a look at the repo, just so we all are on the same page, the embed model is about 3.36 gigabytes. And then if we have a look at the re-rank model, it is 3.36 gigabytes as well. And something to note, it does say 1B, but I believe these are 1.7 or 2B parameters, 1 1.7 billion parameters each. So personally, I would have rounded up to be 2B, but I guess if you're still in the ones, you can kind of get away with calling it 1B. Anyway, let's keep going. We'll get out of that. So we've loaded the models. Now we need to create or load the embeddings. And so as I said, creating the embeddings is a one-time process. Now I've already done that because this is a machine learning cooking show. But if you haven't done that, I've written some code here. So this is the embedding text. So Image Text Embeddings 10K SafeTensors. Now these are by default image and text embeddings in the same modality. So you can make just image only and just text only. But we've merged image and text into the same file. So they give the best results according to the blog post. So it won't actually load this because I don't have it saved here. So let me just show you what it looks like if we were to create them and then I'll bring them back in. So some code here. We're going to create embeddings. So proceeding to create new embedding files and then we loop through the dataset train and then we select the image and text. So the image and recipe markdown and then we encode the documents, images to embed, text to embed, and then we stack them all together into a list and then we concatenate them all in one big image text embeddings all, and then we save them to file, which is up here called image text embedding safetensors. And we're using safetensors to save it because it's, it's a really fast approach to do so. So as you see here, if we were to let this run, this would take about an hour and a half. But once that's done, we can reuse that file over and over again. So I'm going to pause there. I'm going to stop that and let me just see how quickly we can do this multimodal RAG. So I've got, as you see, I've, I've made this file previously. I'm going to just copy this into notebooks so that we don't have to sit through that. So let's pretend we've already created them. There we go. So file found, loaded embeddings from Image Text Embeddings 10K. And then our embedding shape is 10,096 because that's how many samples we have. And then each of those embeddings is a 2,048 vector. So if we look at a single sample, there we go. We get the first sample from our embeddings file. And these are the first 10 values. So again, they don't mean much to us, but numerically, in numerical feature space, they mean a lot to the computer. So when we go, hey, give us sample 1, Right, if we typed in here 1, it's going to have some different values. And then if we typed in here 2, different values again. But when we compare these using cosine similarity, it will give us a score of how much they are related. So we can now retrieve samples based on an input query. Um, once we've stored our embeddings, a very important point to note is— I'm going to write this down— so note is if you have stored your embeddings, uh, with a particular model, you must embed the query using the same model to get similar results. So if you don't do that, say for example you embedded your dataset and then you loaded it and then you embedded your query with a different model and then you compared them, you're going to get probably poor results. So just rule of thumb, embed the documents, embed the query with the same model. So here we input a query, which could be text, we could even change this to recipes for sleepy time tea, right? If you wanted tea before bed, or for breakfast time tea, right? That could be anything you wanted. Right. And another point while we're here is if you— we go here, input guardrails, because our system is designed to only work with recipes because of the generality of this, your query could actually be yellow tractors and then the model would still return stuff. So maybe you could have a step at the start that checks whether or not the query is food related. So that's an extension that we are not going to cover in this video, but just something to keep aware of. So we do L2 normalization. I'll let you research what that means, but essentially it just means that our queries— sorry, our embeddings are going to be normalized into the same feature space. So then we can compare them quite easily with just the dot product operator here. And then we sort them with torch.argsort, and then we get the sorted top scores, and then we return those sorted scores and sorted indices. And so our query, because we're using the vision language space, can be an input string or an input image. So that's really cool about these vision language models. We also put in target embeddings to match in our case, which are our ImageText embeddings. We saw those before. So that's sample number 0 and then we get a top K back of 100. You could change this to be whatever you want. You could be, I only want the top 20. I'm going to leave it at 100 because it's a fun number. Now, if we do this, notice that this is quite quick. So if we do it, 70 milliseconds. So what this is doing behind the scenes is it's embedding our query, right? Query, query embeddings, and then it's computing the cosine similarity across all of the target embeddings. So we are comparing our query to 10,000 embeddings in 8 milliseconds. That's how fast that operation is. So that's very cool. The retrieval step is very fast. So our embeddings, document embeddings took about an hour to create, but the retrieval is now sub-10 milliseconds. That's awesome. So then what do we get back? So this is our result sorted scores, result sorted indices. These scores are going to be approaching 1. Between 0 and 1. So the closer to 1, the more similar our query is to the document that it's matched with. Now you probably won't get perfect 1, but I mean, if you do, that's awesome. It's probably— you're probably, uh, matching on the same input to the same sample in the document. Um, so for example, if you matched— if you searched for sample 0, ideally you'd get back sample 0 as the best related result. And then as you see, the score slowly starts to drift down towards 0.4. So these scores are still all quite high, right? Like they're— again, it's going to be depending on what your dataset you're using, you're going to— the best way to sort of figure out what a good score is, is to try a number of samples over and over again. And then tensor— sorry, this second tensor, results_sorted_indices, is going to give back a list of indexes which match this score. So if we look at index number, So dataset train. And what was our query? There we go. Pork cutlets with Parmesan tomato sauce. Yeah. So that's what we found. And then if we look at image, there we go. This is what we get back. Pork cutlets, Parmesan with tomato sauce. So that is our query. Recipes with tomato and Parmesan cheese. What if we did yellow tractors? Let's have a look at that. Do we get lower scores? Yes, we do. We get much lower scores, right? But we still get one back because the model is always going to retrieve something unless we put in guardrails. Fried yellow squash. Okay, not too far away. It's got yellow in the title, but not ideal. We'd probably put a check in there to make sure it's food related. And then one more time, recipes for breakfast time tea. What do we get back? Okay, we get some much higher scores and we get— this is number 1. What do we get back? We get traditional English tea time scones with jam and cream. That's pretty good, right? So let's get out of that and let's go back to our original query because that's what this notebook is set up for. But just so we could play around, that's what I encourage you to do is try different queries. You can also test with an image. So say we have this image. Sorry, I should have been a more prepared cooking show. I'll get this image, uh, moussaka.jpg. So there's our image. If we wanted to query this, because our model is VLSPACE, so we can open this image and we get that image. We're like, hey, I want to find a recipe to cook this. What can we do now? We can put in the, uh, test image query, and behind the scenes our match_query_to embeddings function is going to embed the query as an image. Now a little tidbit is that— note, now this might need to be a pull request to the embedding model page— note, for images, embed them as a document. Okay, under the hood it's going to use the same method as that, but for some reason this method, uh, in the model page only handles text. So we go down there, I just want to show you what this looks like. What do we get back? Mediterranean eggplant aubergine cakes. And if we have a look at that image, what does that look like? There we go. Mediterranean eggplant aubergine cakes. So if we look at what that looks like, eggplant, aubergine, that's the same thing, right? Cakes. I've typed that incredibly wrong. No, it's okay. The one we got, Mediterranean eggplant aubergine cakes. And what do we get back as number 2? We're looking for mashuka. So this would be a mile-high crockpot lasagna with zucchini and spinach. What does that look like? Again, this is not our database. This is just the recipes that are coming back. So that's pretty close, right, that we searched for that image. They're pretty close. I like that. So try your own images and see how you go. But the point about this is that we can now do text and text queries and image queries. So let's keep going down to the next stage. We've got our retrieval, we've got embedding creation, we've got retrieval. So the first letter of RAG, retrieval, and we can do one more text-based query and we can display the top results in some nice HTML cards. What do we get back? There's our query test image. So the first text matches are Mediterranean eggplant aubergine cakes, mile-high crockpot lasagna with zucchini, vegetable lasagna without lasagna. And then these are the image matches. Remember, our recipe image matches are not natural images, they are document images. We can also now re-rank the outputs. So if we go back to our Overview. What have we done? We have embedded our dataset. We've saved those to database. They are in a single file, embeddings.safetensors. Uh, we have now retrieved the top X samples, so we're up to re-ranking. Now the re-rank model is, as I said, a very similar backbone to the embed model. And if we go to community, I've loaded it from this pull request. So this commit, I've loaded it from here so that we can enable SDPA use. If you have Flash Attention 2, probably beneficial to turn that on on your system. So let's try this query here. Banana sour cream bread. Rerank the outputs. Let's see what happens when we rerank. So we get back our— oh, did that work actually? I think we might need— one second. Oh yeah, here we go. So this is where we got those results. Text result sorted indices, text result sorted indices. And then there's our text query, which is banana sour cream bread. Beautiful. That's working correctly. Now next is to do some re-ranking. So we can select how many samples because we're going to get returned back. We've set the top K to be 100. We have 10,000 samples. But when you re-rank again, it's going to compare query to sample, query to sample, query to sample, query to sample. So that's going to slow down. That takes compute time. Generally, you might want to limit this to the top 10 to 20 samples rather than the top 100. But this, of course, depends on your system setup. So we re-rank the top n samples. In our case, we're going to re-rank the top 20. Then we extract the text to re-rank and the images to re-rank, and then we format a dataset here. Now this is all taken from the re-rank page. I've just, I've just modified this for our, our system. So we have images, documents, and then we put them into a dictionary. So where's that there? Oh yeah, there it gets turned into a dictionary of examples and then we go batch dictionary and then we pass it to the models and then we resort them again. And so if we have a look here, this is just that exact same thing again. Samples to rerank, we put in the question which is our test query text, we put in the document text which is the text to rerank, and then we put in the document image which is our image to rerank as well. We make sure we move the samples to the target device. If they are tensors, we should definitely be using the GPU. And then we get the re-rank outputs from the re-rank model. And I'll just show you, we're going to run this live. So re-ranking top number of samples, 20. This is going to take some inference time. As we saw before, the retrieval was incredibly quick, but the re-ranking, as I've found— and if you have an idea of why this might be taking longer, please leave a comment. There might be some ways that we could speed this up over time, but that would be something that I'd need to investigate. Yeah, so that took 16.3 seconds, which is not ideal, right? Like, if you want recipes quickly, do you want to be sitting around waiting for 20 seconds? Now, where it would be ideal is if you had to prioritize the best possible results and latency didn't really matter. But in a search problem, latency generally always matters. So there's a trade-off there. Now we get the logits. This is not going to be the exact— so the logits are sort of the score that the re-ranker model has given, but they're not going to be as clear-cut as the score that our retrieval model has given them. So let's have a look more so at the re-ranked indices. So if we get the re-ranked indices, what do we get? This is our tensor. So this is kept 0 and 1. So if our re-ranked tensor was the exact same format, it would be 0, 1, 2, 3, 4, 5, 6. Because this is an argsort tensor, this means that index 0 and 1 have actually stayed the same. So the 2 original results that were the top 2, 0 and 1, index numbers of course, uh, are still at 0 and 1, whereas number 7 has moved up from like down here or down here to here. So number 7 has moved up 5 spots. But I've got a helper function to help us look at this. So this is a, a helper function for re-ranking samples. We put in our dataset, query text, sorted indices, number of samples to re-rank, model, re-rank processor device, text column, image column, dataset split. And then we get back data samples to re-rank and re-rank sorted indices function. And then finally we can display the re-rank changes with some nice color. So there we go. So this is the original sample. So sample 0 was originally number 1, so there's no change. Sample number 1 was number 2. This is 0 index, by the way, and this is +1 index. There's no change in the top 2, as we said before. But then have a look at this. The re-rank model has put up sample number 7 from number 8 to number 3. So it's got +5. And then here, sample number 6 was number 7, but now it's number 4. So it's moved up 3 and then so on for the rest of the leaderboard. So now depending on how dramatic some of these changes are is where you want to consider using a re-rank model or not. To me, from the vibes, we haven't really evaluated this because it's a bit of a toy dataset. The vibes for me, the retrieval step was enough. I personally think re-ranking in our specific problem, 10,000 recipes, doesn't add that much value. But that's going to be for you to decide in your own system. So let's have a look before and after re-ranking. So I've got Claude to help me generate some HTML viewing code here, but it's just the same information, just viewed differently. So query is banana sour cream bread. So that's what we typed in. So these are the original retrieved samples, banana sour cream bread, that's perfect, banana, banana sour cream bread, best banana ever. So the re-ranked samples, let's have a look. We've got the first two are still the same, but now the third one has moved from number 7 to number 3, sour cream chocolate chip banana bread. Now, is that a better result than this one? Best banana bread ever. If you're saying that our query had sour cream in it, I would say that this is a better result. So in that case, re-ranking has worked. Did it work well enough to warrant 16 seconds of re-ranking time? Probably not. If we could get that re-ranking time down to under a second, that then it would probably be worth it right now. So we've done our retrieval and we could do augmentation, right? Augmentation is probably one of the easiest steps because all we had to do now is put the top 3 recipes or the top X recipes into a prompt for a generation model. That's just augmentation, right? If you are a business and you want to retrieve some stuff and then you want to augment those results into the input, remember the whole goal of RAG is to give our LLMs specific context. We've got Augment ready to go. So now Augment and Generate kind of come together. So we can now generate answer based on results and depending on the model you want to use, for the generate step. I'm going to use, uh, in this demo, QuenVL 2B Instruct, because it's just a small model, right? But it's also got vision language capabilities. If you wanted a high-quality model, you'd probably look at something like a Nemotron V3. This can run locally on the DGX Spark, but at the moment it's just quite a bit too large. Well, I need to try out— an extension could be is to integrate Nemotron V3 to run in this pipeline. So I'm going to stick with a smaller model to begin with. But again, if you want it more for a production system, you might go for a NemoTron 3 Nano. I'll just put that there. But because we're not doing anything too sort of complex in the generation side, I've chosen a smaller model. We're just generating a small summary. If you wanted to do complex generation, then again, yeah, you probably want a bigger model like NemoTron 3 Nano or something like that. So let's load the generation model. Uh, something to note as well, NeMoTron 3 Nano will use about 40GB of memory. So versus Qwen, a smaller model will use 4.6GB or 4.26GB. So if we go here on the DGX Spark, we probably actually have room for NeMoTron V3. But yeah, see, we're kind of pushing the limits here of a general consumer GPU. That's 45 gigabytes of memory that we've used so far, which is okay for the Spark because we have 128 gigabytes of memory. But if you're in a GPU-constrained environment, just something to think about. So, um, quand3vl2binstructor is loaded. We're going to create a function here to generate a recipe summary. So basically what this prompt says is it gives me a recipe summary given our recipes as the input. So recipe text, we can pass them in as a list. So when we go back to Gemini, what Gemini would have done is kicked off a search, right, based on RTX versus 4090 or 5090, found some documentation links as you can see here, put in those information into the context window, and then returned a result here based on those links. As we can see there, links, links. Links, links, links. And that's what we could do as well in our system is we can generate a summary and then we can provide links to the actual recipes where that summary came from. So then it's grounded in context. Now let's run that. So if we want to just pass in 3 example recipes, we can just get 42, 43, and 44, the markdowns into our generate recipe summary, summary model. Function, sorry. And then the model is going to output this. So recipe summary: artichoke hearts wrapped in bacon, a quick and easy appetizer. Key ingredients: avocados, stock up on sale and freeze. Tips: a simple and cost-effective tip. Easy, easy, right? And we could customize this output. I've just done it nice and simple to be a summary. But again, the world is your oyster with LLMs. You could make this to be whatever you want. So let's now— this, this demo is well and good for us as developers, but how could someone else use this type of system? Well, of course we're going to make it into a Gradio demo. So let's first remind ourselves of the query. We're going to use this query here. We'll match our embeddings to the query with our match_query_to_embeddings function. Okay, these are just the retrieved results. Now we can create a helper function for showing recipe text. This is just going to be in HTML and then we're going to put everything together because Gradio, how does that work? Gradio works from input, which can be text or image, function, output. So in our case, it's going to be the input, text or image, retrieval, rerank, generate and display results. But in our case, we're going to set rerank to be optional and generate to be optional. So if we go to our Gradio demo, Right. This is our input, which can be a query of text or a query of an image. And we can put generate a summary to true and re-rank to true or vice versa. So let's just let this run in the background. So if we wanted to search with an image, so you took an image of your kitchen bench. I generated this with, with Gemini. Let's see what we get back from that. We'll let that run. But this is what our retrieve function does behind the scenes. When we go on Gradio, we input a query text or a query image. If you input both query text and query image, we focus on just the input text. But if— an extension to this would be to query on both text and image combined. So that's my challenge to you, is how would you update this to query on the image and text combined? And then we do our retrieval steps. So we just do match query to embedding and then we collate those two scores and samples and then we create an image gallery and an output text gallery here. So that's what's going to display in our demo. So tomatoes roasted with pesto. How does this look? For some reason it's not letting me go full screen. That's all right. Tomatoes roasted with pesto, roasted tomato salad, scalloped tomatoes, barefoot Contessa. So tomatoes, olive oil, dried oregano, kosher salt, fresh ground black pepper. That is basically all the ingredients that are present in this image. So that is one of the cool things about these new models is that they use SigLIP. And so SigLIP 2 has been trained on a lot of natural images. So although they've been fine-tuned for documents, what we found with this multimodal RAG pipeline is they actually also work for natural images as well. And then we get our recipe summary. Tomatoes roasted with pesto, roasted tomato salad, or scalloped tomatoes with Barefoot Contessa. And then we get— yeah, see, the generation time was a few seconds there. There are many optimizations we could probably put behind the scenes, but for a demo, this is pretty cool, right? We can go from text query or image query. So we go back here. So we've got our retrieval steps. Then optionally we have our re-ranking steps, which is just exactly what we've discussed before. The only difference is here, if we create— if we do re-rank, our outputs are going to be slightly different. But then if we don't re-rank, we're just going to set re-rank to turned off. And then finally, generation is also optional. And so that's where we do the generation steps. So if generation_summary_option is true, which is just selecting here and true or false there, then we run some generation steps. But of course we've made it optional because if we just want to do search query to recipe, then we can do that. And then we return a timing dictionary so that we know how long each step took. And so if we want to do a demo test query, a cup of tea, Cheers. I got a cup of tea here. Let's see what we get back. And so we've put in our demo test query, no image, uh, we're not going to re-rank, but we are going to generate a summary. And let's see what happens. Query text: cup of tea. Input query: we re-rank option set to false, skipping re-ranking. Generation summary set to true, generating summary. So as you see, the query embed match time was very quick, but the generation time was about 7 seconds or so. So a few optimization steps would be probably our first extension to this project. And let's have a look at our summary. What do we get back? Recipe summary: clove and cinnamon tea, spiced chai concentrate, Afghan tea. So if we— this is of course running locally. If we pass this into our demo, Let's go here. Cup of tea. And we do want to generate a summary. Let's see what we get back there. But we can also display these clove and cinnamon tea, spiced chai concentrate, Afghan tea. Delicious. And then we can display our images in a gallery. So these are our recipe cards that we get back. And then we can turn this into a Gradio demo, which of course, like all good cooking shows, we've already kind of done. So if you want to see the Gradio demo code, that's available here, but it's going to give us back this exact setup that we have up here. So cup of tea, clove and cinnamon tea, spiced chai concentrate, Afghan tea, uh, an authentic family recipe. If you want to see how this is run, just go to files. You've got all the app files there. That's the main one that's running on Hugging Face. If you want to run this locally, you probably want app_local.py, and there's our embedding file there as well. So let's go back. We're nearly finished here. So if we search cup of tea, this is of course running within our notebook and we're going to generate a summary just to make sure we've got the same output behind the scenes. The retrieve results are going to be very fast, as we've seen. And there we go. Clove and cinnamon tea. We get some recipe images back, we get the scores back. Lovely. We get the recipe text as nice markdown HTML, and then we get our summary here. We could of course customize the prompt for the summary, and that's something that I encourage you to try out doing, is maybe you could display this a bit nicer than what I've got there. And then we've got, yeah, timing. So let's interrupt that. If we wanted to upload this to Hugging Face, how would we do that? Well, we need to turn our Gradio demo into files. So, um, I've got a demos folder here already existing, so I might make a directory demos and then, um, let's call it multimodal RAG. Did that make that? There we go. Okay, I want to show you that this is live. I don't necessarily need to change all this, so we're going to create our app.py file. Which if we go— I'll just write this there. So writing that to there. If you want to have a look at this, this is going to be on the Hugging Face demo. But essentially what this does is it loads the models and the dataset. There's the loading of the dataset and the models. It creates the helper functions. So just what we've seen in the notebook, the re-ranker function, generate recipe summary, and then it gives us some visualization. So this turns a recipe markdown into an HTML card. And then we go spaces.gpu so that we can, um, use the GPU that Hugging Face provides on Hugging Face Spaces on zero GPU. Then we're going to do our retrieve, and here we'll do retrieval, we'll do re-ranking, which is of course optional, and we'll do generational— which was generation, sorry— which is optional. And then we create our Gradio interface, and that is what's going to give us this demo here. So we'll go back. We've written that to file, so there, app.py, and then we can change this to requirements.txt. So this is going to be the requirements that we want our app demo to install. Then we can also change the README. There we go. Um, I want to write this as well. So video, I'll write that there. Just so you know that this was authored here, I'm going to make a slight change because if we upload this to Hugging Face without changes, it won't rebuild the space because Hugging Face behind the scenes is smart. It knows when your files have changed, just like GitHub. So, um, if we go for a full series of how this system was built see the YouTube playlist. And so I'm going to go to Chrome, YouTube. I'm going to copy this playlist link there. So we'll write that. There's the README, and I'm going to copy this to— oh, excuse me, I'm going to put the embeddings file in there. I don't need to do this because it's already up on the space, but I just want to show you this full workflow so you know we've gone through this end to end. I'm going to copy that into there. So that'll be in there. So now we've got— this is our demo folder, app.py, image text embeddings, 10K, SafeTensors. So remember, we created that at the start when we created our embeddings. That's going to be stored in our— I don't want to play that. Up here in our space. And then we also want to store this in there as well. So this is a demo image. This is just from there. There we go. Kitchen bench. That's the demo image that we used before. And then we can upload this to Hugging Face. I'm going to pass in the path of the folder I'd like to upload. I've defined the space that I'd like to call it. It's not going to recreate this space entirely. It's just going to upload some new files here. Multimodal RAG with NemoTron. There we go. It's a repo type of a space. It is Gradio because we're using Gradio for the demo. We're going to create a repo. We're going to get the full repo name and then we're going to upload a folder here. Using the Hugging Face Hub helper functions. And then we're going to go uploading files to demo for the YouTube video, which is what you're watching right now. And then this should be quite quick. So there we go. It's going to start uploading some files. Demo successfully uploaded. Now you'll notice that only the files that were changed get uploaded. Yeah, less than a minute ago, README was the only one that really changed there. But the image text embeddings stay the same because those files haven't changed. So that's the beauty of Hugging Face repos is that they are smart behind the scenes with zett. They only know or they only change when your files have actually changed, which is really good. So some notes to finish it off. We've got the re-ranking trade-off. So of course you, you probably will get better retrieval results, but you do increase latency. The generation trade-off, it produces a nice summary and overview but increases latency. So, uh, there are some questions. What do you want your app to do? Do you want it to be fast? Do you want it to be the best results? Do you want it to be generative? That's going to be a design choice for what you want to build. Uh, image and text embeddings, they do give the best results on paper, but they aren't always available. In here, we've defaulted to image and text embeddings that recipe images and the markdown text in the one space. But if you have images only, you could still do that with these embed models. Or if you had text only, you could still do that with these embed models. But if you want to enhance your embeddings, perhaps if you have images only, you use a vision language model, um, to add text to that or metadata, and vice versa, right? Like if you have text only, you might want to generate an image to associate it with that text. Now Where would I go with some extensions? I would add some actual evals. So right now we're just basing it off vibe. We're basically looking at our pipeline and going, hey, it's returned these results, how do they look? And if they're pretty good, well, that's awesome. But if you wanted some metrics, you would probably have to create a dataset that would go, given this sample, it should retrieve these 5 other samples and then you could get some actual metrics on that. If we wanted to improve our RAG pipeline, we could definitely put it somewhere towards like a lexical search. So combine the vector search with text-based matching. So if we put in an input query of tomato and Parmesan cheese, then it will match on that text directly as well as that vector. And then we could do an input checker, query rewriting. But again, I'm going to leave this document here as something you might want to read too. To how we could expand our system. So there's some great guides here: direct prompting, embeddings, evals, fine-tuning, guardrails, hybrid retriever, query rewriting, re-ranker. This is where we'd go next if we wanted to build this out as a more production-ready system. And then finally, if we wanted to improve our dataset, because we, we created this dataset from scratch, I chose 10,000 as not a small but not an extremely large sample. We could upgrade this workflow to do a million plus samples. And in fact, many businesses will have that many samples that they need to search over. So right now we're using a single embeddings file. So that's SafeTensors. But at a million samples, we might want to look into a dedicated vector database. So that's something that I'd recommend probably looking at from there. Of course, you could try it out with just using local files, but depends where you need to run this. And then layout creation. Could we our PDFs right now of our recipes are pretty bland. Could we upgrade them to be a bit more diverse to actually reflect actual recipe cards in the wild? And then could we improve it? Say we want to specifically search for different charts. Could we make our search pipeline a bit better for searching for different ingredients, nutrition, serving time, all that sort of stuff? But that's more specific to the recipe search. Now, these things I just put here because you'll have to think about whatever your specific dataset is, how you can improve the search for that. But that has been a multimodal RAG pipeline from scratch. If you want to see it, of course, where's the— how it was built, you can go to the YouTube playlist that's on my channel. But if you want all the code, I'll link that below. There's a great notebook here where you can just read through If you have any questions, um, on anything at all, or if you find any errors, please leave a comment or an issue on the GitHub, and I'll see you in the next video. Hey!