Local Multimodal RAG on the NVIDIA DGX Spark | Part 2 - Creating the pipeline

Testing 1, 2, 3. Can you hear me? Can you hear me, ladies and gentlemen? Just going to make sure that the sound is working. Testing, 1, 2, 3. Just going to make sure that the sound is working. Sounds like it is. Beautiful. Now, important thing, we put the phone away. Okay, so today's livestream, we need to build a multimodal, uh, RAG pipeline. Isn't it too early? It's 9:53 AM here in Australia, so we're probably later than what I would usually want. But let's do our multimodal RAG stream. Hey, so I'm going to open Keynote and we're going to build a little document of what we're building. So let me make a title, What We're Building. And if you're in the chat, first of all, please say hello. Second of all, uh, welcome, welcome, welcome, welcome. Third of all, uh, please say where you're from and let me know if you can hear me slash everything is going okay. Hello, Ashik, what's going on? 5 AM in India. Wow, you're up nice and early. And we got New World Pies. Hello, Daniel, how are you? Daniel from Kurdistan, I'm your student, make a lot of AI with my courses. Thank you so much. Well, first and foremost, welcome to everyone. Second of all, let's start this stream off with building a little diagram. So in the last stream, we created a dataset, right? RAG dataset. We're building a multimodal RAG pipeline. Okay, we just need this to be a simple graphic. And we're doing it all live because why not? Errors and all. Hello, Green Code, what's up? Green Code, are you based in Australia? I've seen some of your videos and I feel like I remember you having an Australian accent. So we want Multimodal RAG overview. Okay, we need some fun shapes. Shape dataset. This can be our data and the text. We need it to be white. Hello from Pakistan. Hello from Australia. That's where I'm from. I'm from Brisbane. And then we want to— now we need to do an overview of what our data looks like. So I used to use Icon Finder. Icon Finder. I hate the, the sign-in pop-ups. So where's search? We have image. Boom. Surely this is free. Download. You're not— oh, you're not an— you're not a— you're not Australian. You are a night owl. You live in Europe. That's cool. And we have Alex. What's up, my big dog? Thank you everyone for joining in. So let's— we're creating a graphic of what we're building so people know. How do I just allow always? Oh my gosh. Okay, our data is going to be images. And do we have JSON? This will do. Oh, maybe this. No, JSON. And we need another text one. Maybe we do text. That's a nice TXT file. Oh no, I'll sign up to premium later on. So our data is comprised of images, text, and JSON, but really it could be anything, right? Because that's what we're doing. We're doing multimodal RAG. Where's my keynote? Westsider, what's up? Arthur Varn has— I gave an interview based on RAG. I learned from different platforms, but I still didn't able to answer the question, uh, which was from distance in VectorDB. Hey Faith, what's going on? I honestly love this kind of raw technical content. Oh, thank you so much. I love it too, because what we often see is the published version of everything. It's, it's very, um It's already done, right? So all the error and troubleshooting, which is where the real work is actually done, is just the errors and troubleshooting. Let's— so data, we've created this dataset, right? And then we're going to embed it with NemoTron Embed VL. We could use another one like Qwen. Qwen just released Qwen3VL Embed. So I'm just going to put this here. This is a multimodal embedding model. I'm going to put the chat over here so I can see it properly. I'm trying to— I'm still trying to work out my like fully stream setup, you know, but you guys let me know if there's any sort of better setup. But I think just simple camera with, um, screen is probably the easiest way, you know. We don't need multiple angles and stuff like that. So this is what we have to do today. We've got our dataset. So this is going to be embedding model, right? And I've got a RAG video on my channel, but this is multimodal RAG that was text only. I'm loving the inter font embedding model. And then we have dataset. So this is on Hugging Face. We'll embed, um, and then we need— I think we just install or store the embeddings in pickle. And I want NVIDIA. NVIDIA have been on an absolute tear with their releases, except they kind of have so many things now that it's hard to find their RAG. This is what we want, NemoTron RAG. Re-rank. There we go. So we'll save that image for later. But we're going to store them in— I think it's pickle. If not, we'll just go database. Do I need to— all right, let me create an account on here because I like using icons. Very good channel, bro. Much support. Your GitHub is amazing. Thank you so much. Hello from India. Hello everyone. I'm gonna see what monthly costs. Sorry, I usually sign up to things monthly first before I know yearly is Cheaper. So sorry everyone, I'm just signing up. I'm putting it on another screen so I don't leak all my login credentials to the world. Apparently I can train my own AI styles, but I don't need that. Okay, we need a database. Here we go. Boom. Database. Okay, and then we need a person. Someone's going to make a query. I need, I need someone like me, man. Mm, emoji. If in doubt, emoji it out. So, man, Frankenstein. Where's the one with glasses? There we go. Person. So maybe we need to make this a bit smaller, but we'll see what happens. So the person is going to make a query, and that's going to— the query is going to go into the embedding model. So person plus query, and we're going to go, um, recipes for tomatoes and parmesan. All right, because we're building a recipe querier. And we need a different font here. We're going to pass that into the same embedding model. And then we're going to Let's do the query arrows in a different color. And then we're going to text match query to, um, database embeddings. And then we get back a list of results. So we need our re-rank model. Which will be here. And I want this to be— how big is this? 398 or 400-ish. So the important thing is we're embedding our dataset with the same embedding model that we embed our query. And then we— let's get some results back from this database, and they can go to the re-rank model. This diagram's getting a bit packed, but I'll keep it simple. Um, and we'll go return top X queries to re-rank model, and then we get these. This is what we'll build first. No generation. So list. Results. So we want re-rank model, re-ranks top X. Results. Top X, um, samples. Okay, does this make sense? We can put this over here. Could even group this, make it a bit smaller. We got our cool person making a query, we embed it with the same model, and then we go, where's Nvidia. So next we want to finalize. Optional. So if we wanted to just return the results, we could. Optional generate answer based on results. So what we're going to build first is just a retrieval pipeline without generation, because that's simpler, right? We always break things down into the simplest possible thing first. and then we can increase complexity as needed. So let's move our things up here. Maybe we make each of these slightly smaller. So the dataset is what we built yesterday. And then we're going to embed that dataset and we're going to save those embeddings to a database. This is the database, but this is a— and we could also— I'm going to put this in Aqua. Once we've got this overview, then we can start coding. All right, because now we know what we're planning, we could just go, okay, let's do it. And that way anyone who joins the stream, we can just go, hey, Here's the diagram of what we're doing. I'm going to make this over here. Why not? Because we need a little bit more space. And then either the results or the generated answer can be shown to the person. Okay, so does this make sense? Multimodal RAG pipeline. A nice diagram. Thanks, Shadow Lilac. And Namaal says, I've been watching your old videos and your signature sentence, keep learning, keep creating, has been stuck in my mind. We need more of that. Yes, we do, ladies and gentlemen. Let's, let's build this. Hey, so we've got our dataset. We've done this. This was yesterday's stream. Go and watch that video if you want to see it. But today we're going to build this step basically. So I'm going to put a little— this is done. So always use color. When you can. I want no fill and border. I want a line green. We put green there because it's done. So we're going to do this today. Generation can come later. I just want to build the retrieval pipeline first. Gonna put yellow, and then I'll send this to the back. This is what we're building today. Okay, let's get on it. So we'll get out of these things. Icons, we can come back to that later. And our dataset is on Hugging Face, NemoTron RAG. This is what we need. This is our dataset. So as I said, we've got recipe.json and recipe_images. They're really plain images, but the point of this is not to have the highest quality images, it's to have— oh, is it not working? There we go. Right, it's to test multimodal RAG. So we need images, but we've also got text if we want. So let's copy the link to this dataset, and we're going to go to the DGX Spark. And yesterday is where we created that dataset. Let's now create Notebook 2. Create multimodal RAG pipeline. As you can see, we're running on the Spark. I upgraded my connection to Ethernet, so I was getting some dropouts and I'm pretty sure it's because of the Wi-Fi here. So we're going to import datasets. Um, from datasets import load_dataset, and then we're going to go dataset equals, um, load_dataset, and path is going to be this, and then dataset number of samples. In our dataset. Len training split. So the dataset's not that big, and that's on purpose. Okay, we'll let that download. And we're using 10,000 Our dataset is 10,000 samples, and lo and behold, that's the dataset of— that the size that NVIDIA benchmark their models at, 10,000 documents. So we've got a dataset that's equivalent to NVIDIA's, uh, dataset download— oh sorry, dataset they evaluated these models on. Now we want to download Llama-NemoTron Embed VL1B v2. Very convoluted name, but hey, at least it tells you exactly what it is. And so I believe this is a combination. There we go. So the model is Llama 3.2, 1.0, 3.21B and SigLIP2 image encoder. So that means it's going to fuse text and image representations into one modality. 1 embedding. So we get output embedding sizes of 200— 2048 dimensions, sorry. So a vector, and maximum content slate is that. So related to the input, let's read about this. Each image tile consumes 256 tokens. We have tested this model extensively with these settings on config.json: max_input tiles 6, use thumbnails equals true, so that every image is split into maximum 6 tiles plus 1 thumbnail. Image at lower resolution, consuming about 1,792 visual tokens. If you embed both image and text, PageOCR, the sum of the visual tokens explained above and the text tokens should not be higher than 10,240. Now, where's the article? So they had— so this model, the beautiful thing about it it can be used with text only, image only, which get about the same results, but then image and text improve results. So that's cool. Now it's only marginally. We might not be able to visually see that, but then if we re-rank, we get even better results. So what we do is we do an initial matching. So if we go to here, if we go to Keynote, we do an initial matching here. Now then we pass those, say we got the top 50 results from our database. The re-rank model, its job is to look at those top 50 and then look at the query again and then go, okay, which of these top 50 should actually be the top 10? So we narrow the search space down. Does that make sense? So the initial matching of the embedding is to get the most similar examples and then the re-ranker goes, okay, these are the top 100 most similar examples. How can I re-rank them to be actually the top 10? So maybe sample number 22 is better than sample number 5 and it should be pushed up the list, but we'll see that later on. Now let's figure out how we use this model. Do we have Transformers? Okay, it recommends Flash Attention. I haven't installed Flash Attention yet, but I will in the future. You know what I'm going to do? I might just write this out so we can practice. I always like to practice things, so maybe the boilerplate code— no, actually, stuff it, we got time. Let me know if you can see everything that's on my screen. I'm just trying to do a few things at once. Let's go load embedding model. Uh, actually, we'll have a look at one of our samples. So load dataset. So let's just get 42. Okay, so the things we mostly care about is the image and the full recipe. So let's go Example, full recipe, full recipe. Okay, so this is just a JSON of the recipe. Name: Artichoke Hearts Wrapped in Bacon. Sounds delicious. Steps here. Ingredients, ingredients: bacon and artichokes. So in our system, if we entered a query, what can I make with bacon and artichokes? This should come back. Okay, that's our end goal. Something like that. Very simple use case. But the point of this stream is to just exemplify how you can use this kind of system in your own workflows for your own dataset. So example image is going to be— is it under image? There we go. Right. So this is an image of a recipe card. It looks terrible, but that's, that's okay. We just want it to be a document. If we have a look, these have been trained on text. If we go text. So responses are grounded both retrieved page text and visual content, not just raw prompt text. Okay, so let's load the embedding model source. Yeah, they tile the images. So I just found that out too. So Shadow Lilac has a good point. So SigLIP2 image encoder, each image tile consumes 256 tokens, so max input tiles is 6. That's in config.json. Let's load this. We're going to write it out once, and then we can copy from there on. That's our rule. I'm trying to learn the Transformers API, you know, ImageUtils. I didn't even know there was an ImageUtils part of Transformers. Transformers is one of those libraries where there's just so much in there that— now, I don't know why they put modality as Image. What does this do? Load model. Set device. Torch dtype is actually being deprecated. It is now just dtype. PSA. Now I'm really hoping that this just works, because in the past I've downloaded NVIDIA models and they haven't just worked. Put it in eval mode. Let's download the model. And as we see, the config.json config.json. Do we have patch? Patch size 16. Do we have image force image size 512? Okay, so it's going to reshape images to 512. Image size 512. Okay, something to be aware of. Now, where did they say 6? Maybe that's a typo. Should it be 16 max input tiles? Okay, it says max input tiles 2. Maybe I have to change that and use thumbnails equals true. Let me put this here. Okay, of course we get an error. Uh, we want Markdown. What did we get wrong here? Import error. FlashAttention2. I haven't turned on FlashAttention2. Okay, how long's Flash attention to? Oh no way, did Flash Attention 2 just get installed that quickly? In the past I have spent hours waiting for that to build. We'll restart. UV. If you're not using UV, use UV. I thought that wouldn't work. Let's see if we can just load it in with SDPA. Hey, Asim, what's going on? Oh, installing from pip won't work, you need the custom NVIDIA wheels for the DGX Spark, which sits in some NVIDIA repo. Yeah, that's what I was hoping. I wouldn't have to do. What if we just tried the default? Why did eager not work before? Okay, yeah, this is Let's try all the AIs. Ah, great. I want to try and load this. I probably should have installed Flash Attention 2 before running this. That's okay. I need to toggle this off. I don't want flash attention. Oh no. Let's check the issues. Okay. This is a current issue. Here we go. So the LLM config is currently hardcoded. This hasn't been pulled in, so we may have to change one of our files. See how it doesn't have— what is your opinion on DGX until now? I like it. Igor might be slow, but that's okay. I can do slow for now. So— So let's update our model to accept This is good. So we need to find the right place to fix it. I'm just going to copy this into here. Now let's go back. That must not have worked. So I've updated the modeling Llama NemoTron. Hey Hussein, what's going on? This is not going to work, is it? This really wants me to load Flash Attention 2. I'm going to install Flash Attention 2 in the next Okay, so I'm not in the right Oh, we're getting further than before. Do you have any recommendation for how to create— oh, we got SDPA, beautiful. Okay, we're on. So if you want to create a Nutrify app, I would start with probably just the Gemini API and then some Swift code if you want to run it on an iPhone. I would probably also— probably if you don't want to run it on an iPhone, you could start with like React Native or Kotlin for Android, and then you could probably also just put it into a website as well. Okay, so we're an hour in and we had to troubleshoot a model, but this is good now. Let me just get a summary. So that's on my to-do list. is to install Flash Attention 2. Now let's run the embedding model. Ah, there we sit, the max input tiles. Okay, we've got some push-ups and squats to do. Who's joining in? 10 push-ups, 10 squats, and then we'll get into running some code now that our model has successfully loaded. And I think we can start to clean up some of these tabs. YouTube automod nuked and shadowbanned me. Are you joking? Why did it do that, Shadow Lilac? No, I do 20/20 to get that blood flow. Well, we might be here for a while, so we're doing 10, 10 and 10. Okay, time to embed some data. There's a GitHub repo called eduIDE. GB10 wheels. Let's have a look. So we're going to default modality to image. But we got to figure out how to So go under releases, hey. Oh nice, it's got the Flash Attention Wheel. I'm going to install that off stream because I think that That could be several hours of troubleshooting, but thank you, uh, Shadow Lilac, for that. So for the image modality, there's a 2048, and then for the image and text modality, there's a 10,240. And then modality for text only is p equals— wait, do I have to choose ahead of time how my things are embedded? Wait, does it not mix modalities? So maybe I have to embed individual of image, embed individual of image text, embed individual of text. Do you ever use vibe coding tools in your day-to-day work? If yes, how and what AI models do you use? Uh, yes, I use vibe coding tools every day basically, and I use all of them. I use GPT, I use Gemini, I use Claude, you name it, all the open source models. I'm model agnostic. So each tile consumes 256 tokens. To me, there's no one best model. It's just use, use all the tools you have at your disposal to get the thing done. Like, the best model changes every 2 weeks at the moment, so I just use them all. But as you can see, I do a lot of manual coding too. Because I want to learn the thing the first time, right? I always basically write things out manually the first time, and then I start to use AI tools to speed me up after I know what I'm doing. So we want to query or embed some image paths. Let's get 10 or 100 images. What do I think of companies benchmarking these models? Seems to be a problem everywhere in academia. Yeah, I think benchmarks are schmenschmarks right now. Obviously you want a model that's pretty good on benchmarks, but always just test it on your own custom data. So, uh, if a model performs well on benchmarks, you can probably go, okay, I have an 80% confidence that it might perform pretty well on my data. But then you don't make a final decision until you actually test it. So, um, yeah, but I mean, you got to chase something, right? Like benchmarks, uh, once you hit the wall of scaling, like, they're probably So humans love just number go up. So benchmarks are a great example of just, hey, if we can publish something that goes up on a benchmark, that's really easy for people to understand. We can't— it's hard to market a trillion parameter model. Or since companies stopped marketing like or saying how big their models are or what they do with those, you can only— benchmark is probably like one of the last quantitative metrics that you can publish your model updates on because you can't say, oh, we trained a 1 trillion parameter model. Who knows what Gemini and GPT-5 and Claude 4 are, if you know what I mean. So benchmarks are like the final frontier at the moment. But then of course they all get gamed. Query. Okay, so let's get— so all the image paths. Let's test the embedding model. So we have 10,000 images. Image paths, 100. Oh, okay, spicy hot crackers, lovely. So say we wanted this. Query. We could probably do a PR to transform this image utils. We should be able to handle, uh, path. That's a PR option there. Might do that. Spicy hot crackers, yum! Okay, let's keep going with this. So I'm gonna put that for later. If you were handed Gemini's raw training data and a full TPU cluster, do you think you'd eventually figure out how to make a SOTA model? Uh, what's my timeline? Eventually, yeah, of course, if I had infinite time. But, um, probably not solo. If I want— if you had a— if you gave me one year and solo, probably not. Just because of the sheer amount of steps. I'm not sure. Gemini 2.5 paper. Look how many authors were here. So if I had infinite time, yeah, of course, could try everything. But if I was just solo? Probably not in a year. Now I think I just want to embed the images. So I need images 100. Okay, so we're creating embeddings now. The Spark is running, 96% memory utilization. There we go. And we get document embeddings. 100. Lovely. Hey, Daniel Bourke, where do you learn ML, LLM exactly? Is that Hugging Face LLM course worth it to take? I'm not sure which one that is. Oh, there we go. Yep. I think this would be great if you actually go through that. You would probably be quite good. So let me know if you guys— what I'm thinking of doing in a future stream is, um, oh, build reasoning models. This is cool. Might read that later. What I was thinking is having a stream series where we basically replicate LLMs from scratch. We build them completely with PyTorch and we train them and whatnot. And then so we do an LLM series of basically where we go completely from a blank notebook to full LLM build. And it won't be like a huge model, but it'll be like reasonable GPT-2 size probably. And then I can turn that into a course video on YouTube where we just have one single video where we build an LLM from scratch, train it, and then train it to do something. So let me know if you'd like to see that in the future. And how does the Spark compare with your MLPC? So I would say the Spark is incredible for just quick use, like you plug it in and it works basically, and it's got incredible memory, 128GB. But on raw compute power, it's, uh, much slower than the RTX 4090, but it can hold bigger models. So it's kind of a trade-off there. So for example, this, uh, took longer than I expected— 1 minute 13— even though it's only a minute. Whereas on my RTX 4090, I think it would maybe take 10 seconds, if that. I don't, I don't know, I haven't tested that, but that's something we can do in a future video. So let's Now, where are we up to? We need to normalize our tensor. I wonder how the DJI Spark compares to the M1 2, 3 Max you compared against your Titan car, the old video. Yeah, that's another comparison video we could do. Uh, I know for a fact that MLX on Mac has been getting very good. We could probably do a stream on that in the future as well. MLX. I've been waiting for, um, the M5 Pro and Max chips to come out so we can inspect those there. So we're just copying it from here so we can basically get the demo running and then we can, uh, customize it to our own use case. Oh no, I need to I'm new to AI space. I wonder how I can integrate a 4B model to a React Native app. I'd look at MLX. Depends on where your app is running. Or Google AI Edge Gallery. So check this out, this is how models are running on Android and iOS. So if you want to run a model on device, check out Google AI Edge Gallery. If you had to categorize all LLM companies into different clusters, how would you do that? What do you mean by categorize LLM companies into different clusters? Having that overview will be helpful when applying to jobs. What kind of skills would get you successful in each category? A lot of them are much the same. Are you good at code and math? Have you built— what AI systems have you built? There we go. Okay, ranking highest to lowest. We got too many, so maybe we just go So we have a clear winner. Maybe we return There we go. So let's get the first We're going to get the top 3 images. Beautiful. Okay. Nice! How do I set the title? There we go. So I'd like a recipe for crackers. Now we can do a new query. So we get Spicy Hot Crackers, Cinnamon Crisp, Oatmeal Cranberry Cookies. Now this is only with 100 recipes, right? Let's update our query. What's another recipe? I'd like a recipe with tomato and cheese. We'll encode our query. Tomato and cheese. Parmesan cheese, tomatoes. Nice. No gilled soup. Falti and Nese macaroni and cheese. Now that, a crock of mac and cheese, beautiful. What about best mac and cheese recipe? Mac and cheese. Mac and cheese, macaroni goulash, effectively known as kitchen sink goulash. Okay, so we got some— we've got a pipeline working, but now we might as well update our document embeddings to be all of the recipes. Can I encode a query So the image embedding model is a, by encoder that supports context and textual format, or the query Or the image of a document page. Yeah, let's do it with an image of food. So let's get an image of a tomato. There's our tomato image. Crockpot cheese dip, absolute trouble. So these get really low scores. Okay, so It doesn't work with a non-textual image. That makes sense, because that's what it's been trained on. So how could we turn our test image into text-related? Because if we look at our test image, tomato, Right, say let's just do a very hacky test. This is an image of a tomato. Because it said it was trained on images with text. Okay, so it doesn't work on natural images, but how could we convert this tomato image into text? So if we go to our dataset Oh, we haven't loaded our dataset. What could we do to convert this image to text? Because if I go— let's just do the simplest possible match. Query tomato. Baked baby Roma tomatoes, white crock pot with tomatillo. So it works really well on the tomato match. So we could import image, match to match to ingredient text, and then search with the text. Does that make sense? What is the premise of this application? Who will use it and what will the value be? So we're building a multimodal RAG pipeline. Now this is very, I guess, general in terms of what we're building, but we are doing it specifically for food. For say you had the Nutrify app, and you wanted to find a recipe that was similar to the food that you had. And so you could take a photo of it and then match a big database of recipes and get back information from that. So this is an app that my brother and I build. If we wanted to inject recipes into the app, we might use a pipeline like this. What is the best way to extract— I always get this wrong— from datasets? So we need a way to turn natural images into text. Yes, that's exactly right, Faith Kahan. So user will take a picture of an ingredient, get recipe recommendation based on that ingredient. Yes, 100%. There we go, so we've got 5,200 ingredients. Wow, we can just ex— access it like this. Oh. That's cool. So what we could do is a potential, uh, potentially put these ingredients to embed these, and then we upload a natural image. And then it can be— the ingredient can be matched, and then the text can then be used as the query. So next, embed all Um, what do we want to do? Embed all, all recipes, all 10K, plus save to file. I have a pathway for recipe or natural food image upload, return back highest matched recipes, and then we want to similarity matching, and then we want to re-rank. Re-rank the top recipes based on the input query. Okay, so that's what we're going to do next. I'll be back. Or the fridge picture, and based on the ingredients, a bunch of recipe recommendations based on the ingredients in the hand. That is exactly right. That is another output for what our, um, pipeline can do. Oh, would you look at that, we're 2 hours in. Time to do 10 push-ups, 10 squats. Who's joining in? Doesn't that defeat the premise you said that you don't have access to the original structured text? Gus, what's up, my friend? So that's a good point. So optionally, constraints to consider. Do we have access to the original text, or do we just have a large list of food ingredients? Do we want to perform image text image matching, or text text matching, or image plus text text matching, or all of the above. So let's think about what we got there. And then we can— this is, this is very easy to go, right? That's why we created this dataset, because we could go any of the above. We could do text-to-image matching, we could do text-to-text matching, we could do image and text-to-text matching. Let's update the push-up count. I think we gotta add 20 for today so far. Okay, let's— we've done the dataset. Let's create our embeddings, because once they're done, we can just store them. So I want to con— construct 3 types of embeddings. So we got images. Let's do a text match. And then I'm going to encode the full recipe. Oh no, I tried to print out 10,000 items. There we go. Are there any modalities you haven't worked with yet? Uh, audio is probably my least worked with modality. Text and vision, I've got extensive experience. So now we're going to have a look at our text. So we've created some embeddings. Oh, what's happening here? So— I might need to convert this to a string. That's all right. So we're going to convert this to markdown, so we have string. Nice. Awesome cat! Hello! We're just creating a simple function to turn our recipe dictionaries into markdown. Now the reason we want to do this is because our embedding model doesn't handle dictionaries. So we just turn it. Let's see how this goes. What's our encoder size? Can handle— I think it can handle 8,000 tokens. Maximum context length is 10,000— 10,240. So then we see what we get back for document embeddings. We're gonna embed all of them, all combinations. So this is quick. Nice! Now we're checking the top-ranked tags. So our top-ranked items for tomato— plain tomato juice. Nice! V8 tomato juice. Beautiful. Okay, let's change our query. What should our query be? Um, mac and cheese. Bread cheese. Query mac and cheese. 30-second sandwich, bread, cheese. So we've only got 100 recipes here. We should now do— we should really be doing all of the encoding now. So I want to get the full recipes, but they should have the same index. That's important. So then if we go— okay, that's good. So this might take a while. Or we can do paths to images. That's another thing. No, I think we'll be fine with the embedding text because I've only embedded 100 recipes. We've got 10,000. I don't actually want this, do I? I want the image name. Okay, this is good. I think I have the ID. Okay, so what we're doing now is we are getting the text embeddings, the image embeddings, and the image plus text embeddings, and then we'll save them to file. Beautiful. So we're gonna go We should get crackers or artichoke hearts. Yes. Okay. So now we have our image, images and Texts in order of appearance. So if we now go— where's our text? Artichoke hearts wrapped in bacon. Beautiful. So now we not only have our text, we have our images, and we can do text and images. Do I have SafeTensors? Yes, I do. Okay, let's create our embeddings. Can I pass the encode documents an image file? Okay, maybe we need to So we may need to create embeddings one at a time, or 100 at a time, or something like that. So I'll write some code to do 100 images embed, next 100 append. Does that make sense? Ah, okay. Outside of nutrition, what is your favorite domain to apply machine learning to? Um, waste collection. So I love that you can use computer vision to sort out different waste. And of course self-driving. And then of course health, any kind of health really. So we may have to update this for each modality. So when we want to embed our text, we have to allow our model to use the text max length. And then if we want to use image and text, we update that. And then if we want to use Image only, we use that. Oh, actually, um, wildlife as well. I recently worked with a, uh, a client on a project to— wait, did we get that image twice? On a project for identifying wildlife in the wild. And so the computer vision model had to run on device without an internet connection. So that was really fun. We don't want to run that. And that crashed. It's probably a much quicker way to do this than a loop in a loop, but at least we know that this is working. I'm gonna get a cup of tea. Who wants a cup of tea? So my favorite kind of tea is at the moment Chinese black tea, which is pu'erh tea. P-U-E-R-H. I'll show you. This one. I really like this tea. It's a loose leaf tea. But cheers to wherever you are in the world. We got a cup of tea and we're ready for our next workflow. So this is what we're currently doing. We've got so many large images here, but we have to make sure everything's in order otherwise. So our workflow is we're doing this: embed images only, save to file, embed text only, save to file. Because this is going to be a one-time thing. So let's start with just the images. We should have 10,000 of them. Now we found out that our images actually error if we try to embed them all in one hit. So let's write some Let's batch the image embedding so we don't overload our computer. I think we can do it. I think it's just torchcat concatenate. This is just all sweet. That's what we want. I think that's correct. So for the images, I want to set my modality. So we're going to P max length. All right, let's see how this goes. This might take a while, actually. Okay, if this is going to take an hour, what I might do is write all the code for doing it but then dramatically reduce the size, and then off stream I'll create the full embeddings. So let's just let that run. See, that's where I would say I need to— one, after this I'm going to install Flash Attention. Let me write. That just happened to work out. Now let's do text. It is working. Yeah. So we're going to embed our text with— where's all recipes markdown? So maybe we do 100 at a time. I'm going to stop this. I want to do 500, and then I want to see what it's like to save them. And then same for the text. Beautiful. GLM 4.7 flash is out. That's cool. Nice! Beautiful. So we can save to SafeTensors. Now let's do the same thing for text. I have no idea how long this is going to take. So maybe I need to embed slightly less at a time for the text. Good morning from India. Good morning from Brisbane. Or good afternoon, we are in the afternoon now. Our Lama server is also running. Let's get rid of that. What model am I using to embed the text? So I am embedding with Nemotron, Nemotron Embed VL1B V2. But I should probably use— oh no, I shouldn't use a different model. I should be probably be using Flash Attention 2. Have I tried Ollama models on Claude code in NVIDIA DGX Spark? No, I haven't. I haven't actually used Claude Code ever. Is it good? What models we got out at the moment? Well, Zed AI is crushing it. So we've already embedded our images. There we go, we've got our image embeddings. Now we're going to do text. But we're going to do it in batch sizes of 8, because before we ran out of memory. Maybe images could be rather than 100 at a time. And we'll run a— we'll run this in 8 as well. Where— how are you storing the vectors? I'm storing the vectors in SafeTensor format. And it's really fast, apparently. Loading GP2-2 weights. So on CPU, it's faster by 76x. On GPU, it's faster by 2.1x. That's incredibly quick. Now we're going to do image and text. So now we're simultaneously embedding images and text. Are you querying the vectors as well? Yes, with cosine similarity. We've got it here. So re-rank we haven't built yet. I'm going to do the full 10,000 offline. How did I decide on the similarity measure? Uh, when you're doing embeddings, you usually use dot product, or if your embeddings are normalized, you use cosine similarity. Why are they 504? 504. 500. That's interesting. Okay, maybe we just get the first— let's just see if we can get the text-to-image match working. There we go. Recipes with almonds. Beautiful. Easy almond nut milk. How to blanch split silver almonds. Sliver almonds. Macadamia nut butter. Greetings from the US. Oh, I'm so glad this popped up on your home page. Hey, thanks for tuning into the stream. Feel free to come by at any time, and no worries for questions. That's what I'm here for, right? That's why I build these things live. Because we're kind of figuring it out as we go, you know. But recipes with almonds, we're getting image matching here. This is beautiful. Now let's do text matching. So I'm going to create a function. I'll be right back. Now to do some text matching. My car is getting serviced, so I just need to check my phone to see if the mechanic has called me. Oh, it's ready to go. Sweet. Okay, so we'll get the text and image matching as well as the Sorry, we'll get all the, the text-to-text matching, and then I may have to go and get my car. So this is a helper function query, and then we want embeddings So we're going to embed the query. There we go, beautiful. Recipes with almonds. Wonderful. So let's try another query. Parmesan chips, Parmesan cheese, Parmesan crisps. This is great. Parmesan basket, great steak. Steak and Shake special sauce, Brazilian style skirt steak. Ladies and gentlemen, how good is this? So let's do text-to-text now. Oh, nice! Okay, we've got a text rank list now. I'm gonna get Claude to write a display function. Let's display. Hello from Nigeria. Hello from Australia. Oh, actually, we got to do some push-ups and squats while this is loading. Let's add to the count. This is awesome. So that's TextRank. ImageRank. Now, how do we do a text and image rank? There we go. So that is very cool. Can I put an input? Chris Alex UK NVIDIA. Hello, awesome stuff. Yeah, so we're using— I'm not sure if the NVIDIA tag is on the end of your name is coincidental, but we are using NVIDIA VL models here. So we're doing recipe matching. We've got a data set of images of recipes that I created just synthetic, but they've been seeded with real images. And so this is, um, text-to-image. So we've got query is that, and then these are our PDF images. And then now we're just doing text-to-text, so text embedding. And now I would like to match, uh, text. How do I want to match this? Text to image, text embedding, text to image. And then even finally, and then We could do text to image text and then image text to image text, I believe. And then even image. Let's do image embedding. I see queries. Can queries be image? Oh, we got EmbedBatch as well. Yeah, they're actually— I think we can go image to image as well. Daniel Bourke, do you know how to use NeoVim? No, I don't. I've used Vim before, not NeoVim. Is it the same as Vim? Wow, that's so quick. Steak and Shake special sauce. Oh cool, so we get different Great steak. Brazilian style skate steak. Skirt steak. So do we get slightly different scores? Yes, we do. That's cool. Notice. Hey Alex, thanks for tuning in, really appreciate it. You sleep well, my friend. That's a cool thing to find out. So let's compare. This is a cool comparison. So Now they won't always— here's the thing though, they won't always be the same. Beautiful. So what did we get wrong here? Ah, they swapped them around. But we do get higher scores. Thanks, Chris. Yeah, you're right. Uh, Guan asks, what are we doing? We're doing multimodal RAG. So we're using NVIDIA's multimodal embedding models, vision language models, to— we have a dataset of image, uh, recipe images as well as their text, and we've embedded them. We've saved them to a database. We haven't yet done the re-ranking. We could do that. I'm going to do that later on when we have a larger database. But then we're returning the results right now, and we're getting good results. We're getting the kind of the results that we would expect. So let's update our query. So let's go best sandwich recipe. My most secret, my secret, most special, incredibly awesome toast. 30-second sandwich. Proper Melbourne toast. Now text-to-text. 30-second sandwich, fast and easy after a late night on the tiles. Momo's secret— my secret most special incredibly awesome test, proper Melbourne toast. And now we do finally 30-second sandwich ham and cheese roll-ups. Oh, that's a good one. Proper Melbourne toast. Now mind you, we're only doing 500 recipes. We could do 10,000. We've got 10,000 here. Um, do you recommend using the DGX Spark for training autoencoders, or is it better to stick with the Blackwell GPUs like RTX 4090 for training times? Personally, I've never trained an autoencoder. Um, if you're talking about the Vision Autoencoder, uh, oh well, depends what you want to train actually. Let me know the exact model you want to train, your inputs and outputs. But generally I'm going to recommend an RTX 5090 for training as long as your data can fit in there. The Spark is good for fine-tuning and more so exploratory projects like what I'm doing. The benefit of the Spark though is that you can store much larger models. Um, or load much larger models, that is. So the Spark has 128GB of VRAM, uh, versus the 5090 has 32GB of VRAM. So 4 times is less. So just think about that. If you want a model, you want a large model you want to load into memory, um, say a GPT-OSS 20B or 120B or NVIDIA NeMoTron V3 30B, you probably want a lot more memory. So then it would be the NVIDIA DGX Spark. But if you want to train faster, you probably want a 5090. Okay, so now let's see, can we do image matching? So when we embed our query, let's have a look at the source code. So they both use embed batch, but one process is a query, one process is a document. So how different are they? How different is the query to the document? Because if we just embed it as a document The embedding model is a bi-encoder that supports context in textual format, e.g., the query or OCR text of a page or the image of a document page. Typically, the embedding model is first used to embed, vectorize the whole corpus, document images, or— and embeddings are stored in a vector database associated associated to the raw content. Then at inference time, the embedding model is used to embed the query, the embeddings of the query, or an image of the document page. I'm quite new to this GenAI stuff. Just completed your PyTorch video. Do you suggest me go with these live videos next? You could go to This is my other website, learnhuggingface.com. And a lot of the new stuff that I'm putting up there. Otherwise, just keep watching my live streams if you want to learn more. So let's try to image to image to retrieval. Image to image matching. Let's generate an image. So we're going to try to match the pumpkin soup recipe to— now I think I'm going to have to embed Let's see if we can embed, embed this and match it to one of our existing recipe cards. So the query, this takes in a string. We could update this to be— to do— update this to take images plus string. So this will be our final batch, and then offline I'm going to embed all 10,000, and then tomorrow we can do— what can we do? Tomorrow we can try out the re-rank model because we'll have 10,000 results rather than— yeah, let's, let's try that tomorrow. So Here's our pumpkin soup recipe. So I want to I'm going to encode this as a document. And see if this comes out. Hey Chris, um, so the stream tomorrow will be probably similar time to now but 4 hours earlier. We'll start maybe even earlier. So is the NVIDIA in your name mean that you're from NVIDIA? Did you work on the RAG models? Because they're really good, I'm liking them. So if that's the case, thank you very much for your efforts and thank you for making them open source. So let's see if this works. I'm matching this test image. So if this works, we should get pumpkin soup back. Recipes or some sort of soup. Is that correct? Oh no, we need Brother Nematron Effect. Oh. Well, thank you very much. I'm loving the NemoTron stuff lately. I actually just ran NemoTron 3 Nano on, on, on the Spark. I've turned it off at the moment because I needed more GPU memory, but it works really well. So that's probably what's going to be part of next— yeah, tomorrow's stream is we're going to finish off this retrieval pipeline, and then if we can get nano3— NemoTron3 nano to return some results, uh, or generate some results based on our retrieved options. Now if we can get that running, that would be awesome, um, but I'd also like to turn this into a Hugging Face demo so we can upload our embeddings, um, and then return them in some sort of Gradio demo. If I can get this running, I'll be really happy. So thank you so much for all the efforts with the NemoTron stuff. Oh my gosh, we did it! We did natural image to Recipe image. That is the power of multimodality. That is very cool. So we can go from image now. Roasted pumpkin soup. So that was our input image and these are our recipes we got back. That is cool. Now what if we did a natural image? Okay, let's try a natural, just a pumpkin, because this SigLIP, SigLIP2 is the Siglip2 is the, uh, image encoder here, right? So Siglip2 should work. All right, this is our pumpkin image. We've done the text image. Now if it doesn't, that's okay. Because we can turn this into text and I'll show you how in a future stream. But Pumpkin Puree. Oh my gosh. Ingredient. We can go from natural image to recipe. So this is a little tidbit of we had to encode the document as an image. Because it's got SigLIP2 there. Of course it's going to work. Halloween party treat, candy corn. So Halloween. So not the best results because it's been trained on text, but we can do natural image to recipe now. Pumpkin puree. And this is only with 500 recipes, right? And no re-ranker. So we can say, uh, I need to do all of them. So embed all images plus text samples. And I'm going to do that off stream just because it's going to take about an hour or so, I think. And then we can do Pathway. So update. This works as long as you embed the image, the natural image, with the image encoder. So you need to pass it as a document. So that would be, uh, potentially an upgrade to the, um, Nemotron VL is if we go encode documents. Yeah, here we go. Yeah, so if we could upgrade the embed batch to the query being handled by— yeah, because it's using the same function. So all we would need to do is extend this encode documents to be— because it's, it's using the same embedding model, surely— to be, uh, image, handle image. Notice how the encode queries can only handle the string. So the upgrade here would be to enable it to handle images, but we can do a hack around by just using the encode documents. Note: to embed an image as a query, use model.encode documents images equals images rather than model.encode query. Okay, that is a beautiful thing. So tomorrow's stream, here's what we're going to do. Save to file, embed, rerank, and then generate. So bonus will be generate. Generate an output based on the retrieved inputs. So that is very cool. Oh, thanks for tuning in, Chris. Appreciate that. So I think that's enough for the stream. I'm happy with where we're at. I probably might even just get an HTML file. Actually, can I do this? I can do this all in Gradio, I'm pretty sure. For the demo. So I've got to go create some embedding files and we've got to finish up the stream. 4 hours 20 minutes. Can we do some push-ups beforehand to finish off? We've got 10 push-ups, 10 squats to finish off the stream, ladies and gentlemen. Who's joining in? All right, we've got our pipeline down pat. We need to use the re-rank model, but we can do that tomorrow, or that could be an extension. So bonus one and then bonus two. So we'll finish off, update the swap count. 72 weighted squats and 72 weighted split squats. That is an ex— exceptional leg day. Well done, my friend. Um, thank you everyone for tuning in. If you have any questions, please leave a comment on the video. Otherwise, I'll see you in the next stream. We are going to, uh, do this. We're going to finish off part 2. We're Well, we already kind of finished off part 2. We've made it work. And then in part 3, we're going to see if we can deploy this as a demo app to Hugging Face Spaces so people can try it out. But happy machine learning. Keep learning, keep creating. I'll see you in the next one.