Local Multimodal RAG on the NVIDIA DGX Spark | Part 3 - Putting it all together

Testing 1, 2, 3. I'm just gonna check, make sure we're working. This should be part 3. Of course, if you're joining the chat, please let me know how we sounding this morning. It's a beautiful Wednesday morning and I'm feeling very lucky to be able to make this type of videos. Of course, if you're joining the chat— there we go. Hello, Nakad, how are you? So right now we're working on multimodal RAG, of course. We have a dataset, we have an embedding model. So we turn our data into numerical representations and then we store that in a database. Right now it's just a SafeTensors file. And then I might just put a note here. So text can be— I might just copy this. This is our data storage, embeddings.save_tensors. And now we are up to— we could use the re-rank model, but I want to— maybe we do build in the re-rank model. Let's try that. So I want to turn this into— but overnight I was on the DGX Spark. So we see these embedding, image embeddings, image text embeddings, text embeddings. So let's go print. Image_only embeddings, and then we'll go image embeddings loaded.shape. And what this is, is, uh, we have num_samples and then embedding_vector_size. So If we go here, we have 10,096 samples. Each gets turned into an embedding of size 2048. This means there are 2048 numbers representing our— each of our samples. So if we have a look at one of these, it's size 2048. And then if we have a look at the raw numbers, let's just look at the first 10, right? So all of these numbers are— we don't have to understand them personally, like as humans. But the computer is going to love these numbers. Hey, Mohammed, how are you? If we're joining into the chat, let's please say where you're from. I'd love to hear where everyone's from. I'm from Brisbane, Australia, and my wife this morning said that I kind of look like Steve Irwin. So this is my shirt today, and apparently I look like an Irwin. So if anyone knows who Steve Irwin is, apparently I kind of look like this guy at the moment. Hey, hey, Zachariah. So I've been told I kind of look like Steve Irwin at the moment, which I like that because the Crocodile Hunter is awesome. So these are our embeddings, but we have 3 sets of embeddings. Hey from Paris. Hey from Budapest, Hungary. Wow. We have a beautiful eclectic group of people joining in, so thank you so much for tuning in. So we have, uh, text embeddings loaded.shape, and we got num_samples, embedding vector size. Now I've only got this morning, I'm going to do a quick stream because I have to get a haircut I've got a haircut booked in at 10 AM. So it's currently 8:11. So I'll go to about— sorry, 10:45. So I'll go to about 10:15. So we'll get 2 hours. Hey, I'm planning to publish Nourip's paper as a 16-year-old. Do you have any advice or would you try to help me? Advice? Well, first and foremost, that is amazing. Secondly, my advice to that is just believe in yourself. You got this. If you want to do it, do it. I personally haven't published a NeurIPS paper, but here we go. All of our embeddings are the same size. So, but yeah, if you want to publish a paper, You can do it. Nothing's stopping you. I mean, I don't know what the review process is like, but if even if you don't get into NeurIPS, you can just publish it on arXiv or on your own website or something like that. But the sheer fact that you're writing a paper at 16 is amazing. Alex, welcome. Oh, that's so awesome. So Daniel Serrano is starting a new job as a deep learning engineer, and part of it is from your videos and course of PyTorch. That is so kind. Well, first and foremost, congratulations, Daniel. And secondly, I'm so happy that my courses were able to help you. I'm going to screenshot that because I want to save that for later. I have a little box of memories, a folder where I like to store all the nice comments that are like that. Any chance to see a video about training an LLM from scratch? Yeah, maybe we do that in the future. So everyone put the number 5 in the chat if you want to see a series where we build LLMs from scratch. And I mean complete scratch. We start with a blank notebook and then we just build it up and then we turn it into a Python script and then we train it. And we want it to get like a GPT-2 level or size GPT-2. So put a 5 in the chat if you want to see that completely from scratch, and then I'll turn it into a YouTube video of basically 3 to 4 hours where we just go from scratch. 4 AM in your country. Nice. Okay, we've got some 5s. 5. We've got 3 5s. If we get to 10 5s— we've got 4 5s. We need 10 5s. 5 5s. You can't do 2 5s. We need 10 5s and we'll do an LLM from scratch series coming up. Okay, so now that we have all of our embeddings, uh, let's do some inference. So I want best sandwich recipe. MOE, we could do MOE, but I would just start with dense models to begin with. So how long does the embedding— so the embeddings are very quick to do cosine similarity. Look at that. Now we're getting the top 20. We got more 5s. Index out of range. Okay, so this is actually— we don't actually need this one here. So we're trying to find the best sandwich recipe. Okay. What do we get? The best cucumber sandwiches, egg tomato sandwich, deviled egg salad sandwiches. How good is that? So this is pure image-based. Your birthday is on January 27th. That's cool. Not very long away. So this is on text matching. Search query: recipes with steak. That's not the search query, is it? Query. That's cool. What are we training? We're building multimodal RAG. And then finally, we can do image matching or text and image scores. Oh, this is different. Best sandwich recipe matching on image and text embeddings. Simple tomato sandwich. And now we can do image to image matching. So we can put in a pumpkin. So this is— let's say we were doing You good, Nico? So now we're going to match on a natural image of a pumpkin. Maybe we make that a bit smaller. Oh, what do we got wrong here? Oh no, because now we have all our embeddings. There we go. Pumpkin stew cooked in the pumpkin. Pumpkin spice. Roast pumpkin. That is really cool. Now can we go multi-food image? Let's try that out. So Gemini. Let's create an image. Have you used Azure Agents SDK? You can do this with much less lines of code. No, I haven't used Azure Agents SDK. You gotta go, Muhammad. See you later, my friend. What do you reckon for 5090 for LLM project comparing to DGX Spark? So I would say for training, 5090 is better than DGX Spark. For running large models uh, DJI Spark is better of course because it has 128GB of memory, but they will be slower. But the 5090 with 32GB of VRAM, you can run quite, quite high models. So let's just try this image, right? Say you took a photo of your kitchen bench. What do we get back from Recipes? Why is this taking so long to download? Okay, now let's get our Gemini image. Rename, going to be called kitchen bench. So what foods can we see in here? Parmesan cheese, bread, garlic, olive oil, tomato, basil. So if we put this image in, we would expect to get recipes back with this, these foods in it. Let's see if that works. I don't know if it will, but we'll try it out. Hey. Grilled pizza. Oh my gosh, simple parmesan tomato, Greek stuffed tomatoes and peppers. What do we get back in this? We could make a pizza. That is cool. Slight simple parmesan, tomatoes. That's exactly what's in our— ladies and gentlemen, we can do recipe image to recipe match. So what about, um, let's create another image. Create an image of an open fridge. What do we want to cook with fruits? Yogurt tub. There is granola on the shelf next to the fridge with granola on the box. Let's see how we can push this. Who wants to look up a recipe? Someone put in a recipe for something or some ingredients. Put recipe or ingredients in the chat and we'll see if we can match it. There we go. Let's see if we can get a recipe back with this. That's a very nice looking fridge, right? I have so many boogers in my nose at the moment. Daniel, genuine question, brother. No doubt you are an excellent teacher. Taught by you on Udemy, ZTM. Second one is complete data science and machine learning, also by ZTM. Thank you very much, that's very kind. Like this dynamic format, it's cooler. I hope to see it much more. What, what do you mean by, uh, the dynamic format, Fred Fred Freddy? Like as in we doing a live Fridge with fruits. Make a burger, double cheese and extra beef slices. Okay, great idea, we'll do that one next. So let's go fridge with fruits. Chewy coconut granola. Honey granola yogurt bread. Wow. So this is our input image. And we get granola yogurt. That is cool. Chewy coconut granola. Is there coconut? Isn't that awesome? So let's get the first image. What's the top one? Live format streaming. Oh, thank you very much. I think I, uh, I'm enjoying it. I'm having fun. What's— what ingredients are in here? This is a larger version. What do we get? Old-fashioned oats, wheat bran, sesame seeds, pumpkin seeds, sunflower seeds, flax seeds, flaked coconut, raisins, extra virgin olive oil. That is cool. Honey granola yogurt bread. Now, what if we were to match these to text? Can we do that? It shouldn't actually matter. So this is image, just image to image matching. Let's do image to text. So all we need to do is update this. So image to text. So text image— oh, sorry, test image. I'm going to do test embeddings or text embedded— text embeddings loaded. If I could talk, that would be great. So image to text, image to text, image to text. Ah, and this needs to go up here. So we're now matching image to text. Boom! Look at that. That is outstanding. Now we need to display this nicely. Let's get our display code. That is incredible. Greek yogurt dessert with honey and strawberries. So that's the image that we put in. So the best match is that— do we have strawberries there? Yeah, we do. Chewy coconut granola. Easiest protein fruit bowl, breakfast bowl. So we can do image to text matching, image to image matching. And what score do we get? Oh, fairly low scores there. Now, what about image to image plus text? We'll do one final one. Hello from New York City. I've just joined. Um, what embedding model are we using? We're using, uh, NVIDIA NeMoTron Embed VL1B v2. Uh, we're not using a re-ranker yet, but we should use a re-ranker. So make a, make a burger, double cheese and extra beef slices. So we're going to do 2 more. We'll do tape a piece of paper that says hot dog onto a cucumber. Okay. So we need an image. Make an image of a double cheeseburger. And then we'll go make an image of a, of a cucumber with a piece of paper taped onto it which says hot dog on the paper. Background is New York City. This one's for you, Not Bruno Rocha. There we go, we got a double cheeseburger. That looks delicious. It's 4 AM in India, Samir, and you're— today is an interview for an AI engineer role and I'm watching you. Well, all the best with your AI engineer role, my friend. That is very exciting. You can't find a proper path to become an AI engineer. There we go, Hotdog. There is no proper path. The best path of all at the moment is just build. Like, there's a lot of new stuff out, right? Just build the thing that you want to build. Keep practicing, keep creating. So we'll load the next image and then we'll do image to image plus text matching. Because we have 3 sets of embeddings. What do you think of Quen 3 VL 2B? Is it worth— good for these type of tasks? I'm working on production RAG project and I Don't want a VLM or OCR. Can such embedding model do the job? Uh, I haven't personally tried nested tables, uh, and stuff like that. I'm just focusing on PDF recipes. This is what I'm doing. So, um, NVIDIA one, if you want to check it out, is in this blog post. I'd say QuenVL would also be good. But personally, I haven't tried it. So what are we going to try next? We want to try this double cheeseburger, and then we'll try hot dog with the cucumber. So there we go. So we're going to try cheeseburger first. This is from Gemini. So synthetic image, but— oh, there we go. Oh, bacon cheeseburger quiche. We got a quiche, not a, not a burger. So this is where we probably need re-ranking. This is a good one. Bacon cheeseburger potato pie. McDonald's Big Mac. Let's bring in re-ranking next. Query: McDonald's Big Mac. Ingredients: hamburger bun, ground beef, salt, Thousand Island dressing, onion, lettuce, American cheese, dill pickle slices. Good, this is quick. Now let's bring in image to— well, we'll do the cucumber with hot dog. Hello sir, how's it going? Matthias, what's going on? So we want to see if our model can pick up hot dog or cucumber. Probably going to focus on the text, to be honest. Hot dog tagalongs, hot dog and bacon roll-ups, easy cheesy hot dogs. Yeah, I figured it would do that. Hot dogs, because the models like to focus on text. Okay, now let's do image to image and text matching. Maybe that will improve. I doubt it, but it might. Image to image text, because our— we have image and text embeddings in the same— all of our embeddings are the same shape. This is incredibly quick. Query is going to be, uh, cucumber with a hot dog, um, label on it. Yeah, we get the same results back. But we can also match to images because our image paths are in the same order. But that's going to give us back the same results, isn't it? So at the moment, our— it overfocuses on the hot dog. That's kind of expected. 700+ applications. Uh, not lane. Yeah, you're right, Matthias is right. Thanks, Matthias, for helping out. We're using these NVIDIA models here, so these are VL models, so they're focused on documents, but we're doing it for recipes. So now let's bring in re-ranking. Re-rank our results. So right now we're getting fairly good results with the embeddings out of the box. Let's see how the re-ranker does in terms of cleaning them up. So let's go back to our text. I want to see if we can do a high-level concept of— if we just put in a query of Query could be Indian foods. Indian chicken with vegetables. Kadigiri. Spicy Indian-style chicken biryani. Who knows what kadigiri is? Anyone from India who knows what that food is? Put it in the chat, please. I'm going to look it up. Oh yum! So it can search based on just Indian foods. Now what about if we do Australian foods? What is Australian foods? Australian tuna bake, Australian cheese and garlic chive damper, the Great Australian pie clots. Let's go even more specific. Australian dessert foods. Chocolate ripple cake, the Great Australian pie clots, Australian cheese and garlic chive damper. Okay, what about Australian breakfast foods. Great Australian pielets. Australian garlic cheese and chive damper. Australian tuna bake. So it might be over-indexing on Australia there, but also what if we do just breakfast foods? Okay, we've got try fegjelder. Tonight I'm going to give it a shot on these NVIDIA and ImageGen models. Tried the GINA API for embedding and re-ranking, but the rate limit is ridiculous. Uh, I'm running them just directly with Transformers, these models. I should have flash attention too, but I don't. So we've got this feijão. Let's see what it comes back with. Brazilian black bean soup, guisado de consulo, fez carrots. Is that correct? Let's have a look. Hey, I'll be pretty happy with that. So let's try text-to-text matching. Brazilian black bean soup. Portuguese kale soup with veggie version. Jagaceta, jag beans, and ripe from Cape Verde. That's cool. Brazilian black bean soup. I'm using HF datasets as well, and I'm just using SafeTensors. CLIP would work here too, probably. So there we go. We got to also remember that our database is not too large. It's only 10,000 recipes. So we could expand that to be more. But for now, we're just going to— let's go high protein. One more search and then we'll do re-ranking. Mile High Crock-Pot Lasagna. So That's got a significant amount of protein. Protein. Fat is the highest there. High. Yeah, so it's probably over-indexing here on high. Low fat peanut butter. Chocolate smoothie, high protein. Look here, low in fat and high in fiber, has lots of protein, and it's quite filling for 2 to 3 people. Low calorie asparagus egg whites. This is great. So you see how this could work in high protein. There we go. Vegetarian minestrone soup. I don't know if that's high protein. High protein, high fiber foods. Okay, let's try re-ranking. I need breakfast recipes. I'm gonna search in— I want some breakfast recipes. I want breakfast recipes with eggs. Green, white scrambled eggs, or scrambled eggs with cream. Bacon and eggs with tomatoes and mushrooms. Algerian poached eggs. 18 minutes. Let's see if we can do a time one. They take 10 minutes. Omelette in a mug. 10 minutes, 10 minutes, 5 minutes. Okay, so text matching is good. Omelette in a mug, Mexican eggs and rice, 10 minutes. Scrambled egg sandwich, 10 minutes. Okay, breakfast recipes with eggs that take 10 minutes. Let's go another obscure one. Dinner recipes with chicken that take less than 45 minutes. Easy, fast, easy chicken. That's 365 minutes. So on the image matching, it didn't work as well as we'd like. What did we get? Text. Chicken piccata, a delicious Italian chicken dish. 45 minutes. That's basically exactly Fast, easy chicken under 20 minutes. Chicken with tomato basil cream. Yum. Now let's try image and text matching. 65 minutes, fail. 20 minutes. 27 minutes. Okay. Croque Madame. Oh, thanks, Lane, appreciate it. Maybe you want to try the new Sayed. Maybe you want to try Light On. OCR. They just released LightOn OCR 2. I think this is the best OCR model at the moment, and while the most efficient— that's how they market it. This just came out. So if we look here, so 3 days ago, do they have a demo? Gonna like this demo. Say, Ed, link me or send me what you want as an example extractor. Let's try this image. Stop building RAG like a feature, treat it like infrastructure. RAG inherits every constraint of your organization. Scale heterogeneous data. When you treat it as simple add-on, relevance drops as document versions changes. Security blocks you because access control wasn't granted at— enforced at query time. The G in RAG only amplifies when the R provides. Let's try this. It's a pretty simple one actually. That was pretty quick. How do I— where's the text? Well, it got all the text and there it output the image as well. So it's fast. Korean PDFs that contain multiple tables. Let's try this. So there's our table. Pretty simple table, if I'm honest. That's pretty cool. Look at that. Obviously, that's, that's a pretty straightforward table, but that's, that's pretty good. So we'll do one more. Croque madame. What's a croque madame? Is that a sandwich? Classic French bistro sandwich. Oh, yum! I've heard of a croque monsieur. I've heard of croque madame. So croque madame must have the egg on top. Let's see. Yeah, LightOn is MIT. This one's Apache 2.0. So we're gonna do croque madame and then we're gonna do re-rank. Easy chicken croquettes. I don't think a croque madame would be in our— awesome Swiss and mozzarella ham hoagies. So a croque madame might not be in our database. So see how these scores aren't that high, but we still get back sandwiches. Simply Baked Monte Cristos, a French toast style sandwich that is baked and you can cook several at one time. So what's a Monte Cristo look like? It's pretty close, right? We don't have exactly a croque madame, but At least in the text world, we're getting back French type items. Hey, you're right, Sayad, try out LiteOCR too. They just released that and it's supposed to be really efficient. Okay, re-rank. And then we'll clean up the pipeline. So after we've done the re-rank, we'll clean it up. So let's write next. We've done all this, we've embedded all of our recipes, 10,000 of them. And how long we into the stream? We're gonna do 10 push-ups in 10 minutes. Next. Oh, sorry, 10 10 push-ups and 10 squats every hour. I've done that. So next we want re-rank the results. And then we want to create a demo of search plus display results. And then the bonus is— where can I reach out to you for more? Depends what you want to reach out for. My email is available on my website. You just go to MrDeBurke.com or here, YouTube. Feel free to leave comments and whatnot on whatever my videos are. Um, as long as you embed the natural image with the image embedding portion of, um, of the NemoTron. What's it called? I keep remember— forgetting. I'm just going to call it NemoTron Embed VL. There we go. Which makes sense, right? You want to embed the images with the, with the image embedding part. So next is up to re-ranking. ReRank, ReRank works on the following premise: take in a series of already retrieved results, e.g., top 100, re-rank the order of those top 100 results to be better suited to the query. Does that make sense? So what we're up to now we've built this. So we've got 10K, 10K recipes in our database, which is quite a small database, but that's what we're building. We're building this as a demo, right? And then we want to use our re-rank model to re-rank the top results. Would be cool to see them side by side. So let's see how we can do our re-rank model, hey? Uh, and we've got a bit too much going on here. Our embeddings are quite small, by the way. They're only about 40 megabytes for 10K embeddings at 2048 size. ReRank. Let's read about the ReRank model. So the Llama-Nematron ReRank VL-1B v2 was developed by NVIDIA for multimodal question answering retrieval. It is optimized for providing a logit score that represents how relevant a document page is to a given query. The model can process documents in the form of image, text, or image and text combined. The expected images are screenshots of document pages or slides, but we've found that it works with natural images. Documents are ranked given a user query in text form. The model supports images containing text, tables, charts, and infographics. We report the model's performance by evaluating our popular video retrieval benchmarks. The re-ranking model serves as a key component of a multimodal retrieval system such as a vision RAG pipeline— that's what we're building— where it helps retrieve over— improve overall accuracy. Sorry. A multimodal retrieval system often uses a multimodal embedding model, Dense, to return relevant documents given the input. A re-ranking model can be used to re-rank the potential candidates into a final order. The re-ranking model takes the query and document pairs as input, and its self-attention can perform deeper interaction between their tokens. It's not scalable to apply a re-ranking model to all documents in the knowledge base for a given query. Therefore, re-ranking models are often deployed to re-rank top candidate documents retrieved by embedding models. The model is ready for commercial use. So SigLIP2 Vision Encoder. Let's have a look. Okay. So we need to load the model. We might have to update the model code for this one to allow us to not use Flash Attention just like before. Because I haven't installed it yet. I will install it soon. Do we have a fix here? Use default attention implementation with option to override. Oh, there's no code here. Okay, I'll be back in a minute. And then we'll see if we can get the re-rank working. Maybe we— if we can't load the model in here, we might start a new notebook and go from there. Okay, who's ready to re-rank? And we're basically an hour in, so let's do 10 push-ups and 10 squats. I'm very sore from jiu-jitsu last night. 10 squads, who's joining in? So let's see if we can get re-ranking working before I have to leave at 10:15, and then I'll come back and continue in part 3.5. You skipped leg day today. Oh, come on, let's get this working. Hey, so Before we had to— let's do— what the hell, I think we're gonna have to fix this file, because by the issues of the re-rank model, there is use default attention implementation with option to override. I don't know if this one uses flash attention only. Oh, okay, we might be able to use eager. Should we make a PR to Transformers to update Update load image. Let's just see if we can do a PR. How much VRAM? I want to update this to handle path. I think that's all we need, cuz then we import 128 gigabytes, yeah. There we go, we made a PR to Hugging Face, to Transformers. Live on stream. Okay, we're doing re-ranking now. I got a new rule: if I have a PR that I want to do, I'm just going to do it. Modality, let's start with image. Do I still have modality to tokens? Oh yeah, we do. Load the model. Is it the best hardware right now? DGX Spark is— look, it's a very nuanced take. I think I need to make a slide or, uh, a video, probably DJX Spark versus another machine, because for me, DJX Spark— oh, hi Crab, sorry I didn't see you there. So DJX Spark is good for one plug AI, that's my review, and it's got a lot of memory. But if you want to do some serious training, you probably want an RTX 5080 or 5090 or an RTX 4090 or something like that. The problem is with that, you will be limited by the amount of memory that they have. So, rerank. Device is obviously going to be cooler. implementation equals eager. I don't have Flash Attention 2, otherwise I would use that. That's on my to-do list. Don't know how much of this is needed, but we're just going to copy the demo and see if the demo works. Now we're going to go— So they have the same token limit. I want to experiment on models running for my company. Yeah, RTX Pro 6000. Yeah, Matthias, that's my goal. I want to get that GPU, RTX Pro 6000. Free delivery, $15,000 in Australia. I wonder if NVIDIA will send that to me. I think they will because I'm feeling pretty lucky, you know. We'll keep streaming until we get one of those GPUs. So we're just going to append or prefix all of these with rerank. Okay, let's, let's see if we can load the model. Okay, we're going to download it. Beautiful. We get some errors. Flash attention to has been toggled on. Yeah, so we got to edit this file. Okay, let's edit this file, but we've seen this before, right? Oh no, we closed our notebook. That's all right. Oh, it's still live. Good. So that's what we need to update. I need to get better at terminal navigation. Can I just go— no, that's not it, is it? Here we go. So we're updating the re-rank model to allow the use of Non-flash attention to Might have to. Boom, we got the re-rank model loaded. Lovely. Oh, SDPA, Scale Dock Product Attention. Beautiful. So the re-rank model works now. We might submit a PR later on to the re-rank models.

But— Beautiful.

So let's now try out the re-rank model. How should we do this? Should we, should we create a quick demo of the re-rank model? Or should we put it back in the other notebook? I got an idea. Next, before we go, let's create a quick demo of the re-rank model, then we can put it all together in a single notebook. So we'll load the re-rank model. What else do we need to load? We'll load the embedding model. Might call this one embed_model_id. Embed model. Dunno if I need that. My spark might be overrun here. Let's clean up some space, hey? I don't know where this is coming from. So we might be able to get— bypass the flash attention in a space with Okay, we're using SDPA, but I think Interesting. We might be able to use that, but for now let's just try it out with some So next we want to get our image paths and of course our dataset. all image paths in order. And then we can load our embeddings, I believe, or recipe to markdown. We'll do that. Maybe we update our dataset. Empire Builders, what's up? We're building a multimodal RAG pipeline. So right now we've built a dataset, we've embedded our, uh, data, and we are now using the re-ranker model. So the yellow is what we're working on. But yeah, the green part is done, and the next thing we're going to do is re-rank our results. So essentially, we have a dataset of 10,000 recipes. I'm going to write this up here, actually. 10,000 recipe images plus text. And then the goal here is for someone to be able to basically type in natural language what they'd want and to get results of different recipes back. But we've got about 20 minutes before I have to go leave for a haircut, and then we'll come back and see Image embedding is loaded. There we go, nice and fast. So we've already done the embeddings. Query equals, what's your favorite food, Empire Builders? We'll try, we'll try, uh Load that. One of my favorite foods is steak. So let's just see what comes back when we do steak. That's nice and quick because our embeddings are loaded. Query. Beautiful. And then we can display them. Lovely. Steak bake. Steak Supper, South Beach Steak Diane. Now what if we wanted some gravy? Mean Chef's Turkey Gravy, Lazy Man's Gravy, Gluten-Free, oh, Savory Mints. Yeah, what if I wanted gluten-free gravy? There we go. Gluten-free gravy. Gluten-free Swedish meatballs with gravy. Stuffed Cornish game hens with cranberry glaze. Tajine. What is tajine? Lamb tajine, North African dish. Is that correct? Is it North African? Tajine. I've personally never heard of that, but there we go. Moroccan tajine. Tajine of chicken and preserved lemon and olives. Chicken tajine. How does that go? That's pretty cool. And what about Beef Wellington? Remember, we're only doing 10,000 recipes here, right? We could do many more. Beef Wellington. Individual Beef Wellington, London broil, South Beach Steak Diane. There we go. Now this is, uh, text-to-image matching. We could do text-to-text as well, but let's see what happens when we re-rank these. So I think we've got the— what we need to re-rank our samples. So we've got document_text. What do we get back in our list? We get the top 20. Maybe we take the top 50, because we've got 10,000 samples. Maybe 30. They have a suggested amount. So we've got our query, Beef Wellington, and then we need our images. Let's get the top 30. There's our top 30 image paths. And now let's go top 30 images equals— Siobhan, what's going on, my friend? Business is good. Examples to re-rank. We want question is our query, doc text is nothing, doc image is image. Excuse me. Okay, there we go. So we've got 30 examples to re-rank. Now let's see. Rerank processor. Process query documents. Examples to re-rank. Move to target device. Okay. V2 device if is instance. Beautiful. Run inference. Then what do our re-rank outputs look like? Oh, re-rank takes a while for me. Maybe we should only do top 20. I think 20 is enough. How many do you think? How many do they re-rank in? So we're going to do some research into re-ranking models. But let's just see what our model output, hey. Oh wow, okay, we definitely get some good re-ranking there. Okay, so this can all be experimental, but we got some good different order because if we look at this This is good.

Thank you.

We could normalize them. Yeah, let's see the different re-ranks now. Matthias, welcome back. We are getting the re-rank model working, and we've just got some re-ranks. I want to compare them to our top— our changes. So essentially, it's pulled up images from that were back down much higher. So it pushed rank 2 and even rank 1. Where's 1? One's right down. So what we need now is Beef Wellington. So I think it still scored Beef Wellington as the highest. Let's create an image re-rank list. So we got an image re-rank list. Beef Wellington, beef tips and gravy, roast beef tenderloin. Okay, let's compare these to our original. Samir, welcome back. Are these better? So Beef Wellington, these are re-ranked, so we've got to go re-ranked top 20 samples, and these are original. So we've got individual Beef Wellington, London broil, South Beach Steak Diane, but this is individual Beef Wellington, beef tips and gravy. Roast beef tenderloin with red wine shallot. Who wants to search another food? What should we search? Let's go recipes with tomato and parmesan. So we get pork cutlets with parmesan and tomato sauce, simple parmesan tomatoes, zucchini and tomatoes with parmesan dumplings. So let's re-rank the top 20. What do we get back? This is original. Pork cutlets with parmesan tomato sauce. Simple parmesan tomatoes. Zucchini and tomatoes with parmesan dumplings. So Where are we at now? So we're up to here, we're building the re-rank model. Well, we're not building the model, sorry, we're using the re-rank model. So right now we've got top X samples to re-rank model. Re-rank model re-ranks the top X results. And now we're looking at the re-ranked results. So these are re-ranked: tomato, pasta sauce, and parmesan meatballs. Rosemary tomato Parmesan bread. That's sensational. So I wonder if Can you create me a helper function to say how many to display how a re-ranked sample index has changed originally? For example, like a leaderboard with up and down based on its positions. Input, uh, list or torch tensor of arg_sort_items. Output Output is a, um, text-based printout of where various samples moved up and down the list. Example input: The embedding re-rank models can catch a query like breakfast to impress my girlfriend, something like this. Let's try it. Okay, this is good. We're going to try the query here, breakfast to impress my girlfriend, and then we're going to try our new re-rank display function because we have Wow, that's cool. So it stayed still. Number 18 came up 15 slots for that query, but 4 and 6 moved up— sorry, 5 and 8 moved up 4 and 6 respectively. This is cool. This is a re-rank model. So we've got a query now, breakfast to impress my girlfriend. What do we get? Breakfast for the gods. Easy peanut butter lover's breakfast. Cheese bacon breakfast muffins. So if you're tired of eggs for breakfast, give this one a shot. It'll stick with you longer than you think. It's a great surprise to serve your special one in bed on Valentine's Day. That's a pretty good option. Now I think it says quizzed the girlfriend. Let's see. So the re-ranking takes a bit of time here. I'm only doing the top samples— top, sorry, top 20. I should actually say the number, shouldn't I? So we re-rank. Ah. So these— okay, so the originals were pretty good. Here's our leaderboard for the re-rank samples. So if we look at the original top samples, this one moved up. This one was given to me by a girlfriend and it was yummy. We had the regular size muffins, still have to make it myself. I've included cooking the bacon and the cooking time. Update quiz the girlfriend. That's cool. Who wants another one? We'll do one more query, then I gotta go get a haircut for re-rank. But re-rank looks to be working. I like this output too. We can see what actually changed. Maybe we go I think we can do original rank. Gluten-free something. Okay, so, um, what is a dish that usually contains gluten? Can we do gluten-free chicken schnitzel? So my wife is actually gluten-free, so let's go gluten-free chicken schnitzel. Is that how you spell schnitzel? Yes. These are the original ones. Quick and easy gluten-free flatbread wraps, gluten and dairy-free chicken enchiladas. Maybe it's not in here, but That's all right. Let's see what the re-rank model comes up. So the re-rank model takes a few seconds, but oh, we've got a significant re-rank here. So sample number 9 has moved up 9 slots and sample 13 has moved up 11 slots. So our, our top 3 here is totally different to what the original model put out. What do we get? Gluten-free Danish, potato chip chicken gluten-free, wiener or Jägerschnitzel. Okay, quick and easy gluten-free flatbread wraps. Yeah, okay. Gluten-free Danish, potato chip gluten-free, or Potato chip chicken, gluten-free. What about we do one more gluten-free breakfast? Our next stream, we're gonna make this a lot more tidier. Gluten-free cornbread, soft chewy oatmeal raisin cookies, gluten-free banana nut waffles, gluten-free and vegan. Now let's re-rank. I'm loving the re-rank model. Wow, gluten-free, this is a big change. Look at that, up 10 indexes. So this is our original samples. Gluten-free cornbread, soft chewy oatmeal raisin. And then re-rank. Chewy granola bars, vegan gluten-free egg. Yum. Maple, banana maple bread muffins, gluten-free. Delicious gluten-free egg-free pancakes. Wow, the re-rank model definitely helps. Who thinks the re-rank model helps? I do. So far it does. So I'm gonna take a break from the stream and I'm gonna come back after this stream and, um, do— oh, we're gonna update the push-up count, don't we? +10, +10. Next, we're going to do the demo. So turn our, um, search plus embedding match plus re-rank into a demo so we can view it live. Okay, so I'll do 10 push-ups and 10 squats, and then I go get a haircut, and then I'll be back probably in an hour and a half or so. So it's 10:30 AM Brisbane time, about 90 minutes or so I'll be back online. So thanks everyone for tuning in for this stream. I'm going to finish off with 10 push-ups, 10 squats, and then be back. Yeah, Alex, two streams in one day. Christmas has come early. I appreciate your support. All right, I'll be back in an hour and a half or so. See y'all soon.