Local Multimodal RAG on the NVIDIA DGX Spark | Part 3.5 - Making a demo

We are live. Testing, 1, 2, 3. Okay, if you're joining back into the stream, please, this is part 2 of today, which is actually part 3.5 of this multimodal RAG series. So here's what we want to do. We want to finish off this portion and then optionally generate answers based on results. So the problem is if we had— this is not running anymore. Yeah, we're going to stick with just image text embeddings. So let's re-rank model test, we know that it works. So let's go a new notebook. Oh, for, um, make multimodal RAG demo. So goals: image input to retrieved recipes, text input to retrieved recipes. So use the image text plus text embeddings because they, um, are the best performing. Image-only, text-only embeddings perform well. However, according to the blog post, the image plus text embeddings perform the best. So that's this number here, because we have— so the image text embeddings on their own, but perform good. But then we get a boost here with v2. And then so significant boost over just the image and text. But then if we use the re-ranker, we get an even more boost. So we want— hey, Anoush, welcome. You're new to ML? That's all right, I'm new to a lot of things too. So we want optional re-rank the embeddings. This increases latency but improves results. And then we also want optional generate an answer based on the embeddings— uh, sorry, based on the, uh, output text. And this also increases latency but improves results. Does that make sense? So ingredients, we need embedding model, we need re-rank model. Uh, what else do we need? We need embeddings. Going to stick with just, um, image plus text embeddings. And Then we need generation model. Can we use NemoTron 3 or is it too big? And then we need Gradio for a demo. What is my take on QNLP? I don't actually know what QNLP is. Sweet. So these are our image text embeddings, these ones here. And we need a ranking function, don't we? Sorry, let's go to NemoTron. I don't know if I'll be able to load this. Quantum natural language processing. Yeah, I don't know enough about quantum. I've been hearing for years that quantum is going to do awesome stuff. But I don't know enough about it to comment on it. Yeah, see, the problem with this model is that it might be too big to run because we have the embedding model, which is 3.3 gigabytes, and then we also have— oh no, sorry, yeah, that's the embedding model, and then the re-rank model is also 3.3 gigabytes. We might be able to run them simultaneously. Let's, let's just build out the, the demo with retrieval first and then re-rank after. Sorry, retrieval and re-rank first, then generation after. Build the demo application from embeddings to retrieval. Rerank, optional. Retrieval. So let's start off. JL Marshall: Daniel, you're one of my favorite ML YouTubers, keep up the good work. Thank you so much, my friend. We got plenty more to come. Thank you for tuning in. So let's do— we'll load the dataset. We'll claim back some GPU memory, load the dataset. Can I get the images? Yeah, you see, if I got the all the images, that would break. I don't want to store all them in memory, actually. That's why I didn't do that. What if I update the dataset to include— that's a good idea. So I want to turn our recipe Full recipe. So when we upload our recipe now— oh, someone said something in Chinese. That's very nice. Someone has just said this, and how do we say it? Jibojo de Buktko. I probably said that really wrong, but Ji Bo, yo, the book. The livestream is done well. Thank you very much. Nicholas, to a young man, ma, can you speak a little on your code writing process, especially for functions and ML pipelines? Um, I just write code, to be honest. I don't have necessarily a process. I just work on the problem until it's fixed, until it's solved. This is what I'm building, and I'm just ticking off it piece by piece. Right now I'm uploading the dataset. So if we go to Mr. D. Burke on Hugging Face. We should have an updated dataset shortly. What happened here? Did it get it wrong? Now we may have a new column. Let's have a look. Excuse me. We're going to load the dataset. We do get markdown now. So I don't actually have to turn these into markdown. I've already got that. There we go. So we just extract the column there. Now let's load the models. So we're loading the embedding model. There's no magic bullet to me, it's just you gotta Keep practicing. Load embeddings. So now our dataset got updated with recipe markdown. Wonderful. So we can go— we want this blog post, load embeddings. So maybe I'll create some definitions. Okay, we have our embeddings. Because our embedding store Will you ever make a video about making custom datasets when you don't have access to real data? I didn't have access to real data, um, for this project, but I made a custom dataset. What do you mean, like you don't have— what kind of data are you looking for? Because these days it's, it's almost— there is so much data out there, and if you don't have it, you can, you can start by generating it. So for this project, I didn't have this dataset, but I went to Kaggle I just searched for food recipes and found this dataset on Kaggle, and then I've customized it for my own use case. But I get it, there could be many problems where you don't have a unique dataset, a custom dataset. But if you let me know, I'll, uh, oh, what do we got here? Members only. Wow, I can switch this to Members only. I can do Q&A as well. Let me know what, what type of dataset you're looking for, and I'll— I guess the more specific you are on not having access to real data, the more I can help you. How many live streams have I done so far? Um, I don't know, maybe 3 per week for the last 3 weeks. You can just go to my channel and see them. I'm looking to make, uh, a GitHub of streams where I just put all the links to the different streams. Maybe I could send Claude Code out to do that. Should I do more live streams? I'm liking them. So for example, I've made models for my company on whether a check-in was good or not. And I know there aren't available datasets like on that, on specifically on Kaggle, you know, if that makes sense. Okay, that, that does make sense. But you could also synthetically generate that now. You could create a a seed dataset and go from there. Of course, the gold standard will always be real data, but you could definitely synthetically generate some of those datasets. So query equals So let's put in the There we go. Let's update this to handle, uh, images as input as well. What's I FW the live stream? I'm in the US, it's almost 11 PM. So thank you for tuning in. That's very late. Yeah. I usually use JSON-L when generating synthetic data. Is there a specific format I should use to better train my models, or is it more of a personal preference? Synthetic data just depends. Can be any format really depends what you want, what you want your inputs and outputs to be. If JSON-L works, keep using JSON-L. I know, uh, Allen AI released OLM OCR 2, and they used YAML. So if we go to the PDF— Allen AI, I like their blog. Molmo 2, Molmo 3. Molmo 2 is a good model. YAML. Yeah, they use YAML here. Check out this paper. Yeah, the first— here we go. The first on OCR was trained to output JSON objects. We switched to YAML, which reduced the retry rate dramatically. We speculate this is because the model does not need to remember how many open quotes there are currently in the JSON and can simply output end of sentence token as soon as it's done. With JSON, we also found more incidences of repetition loops. We found no benchmark score difference, but with the need for fewer— but with fewer need for retries, this improved our overall inference efficiency. So that's probably what I would try. What's the difference between RAG and GraphRAG? I'm not sure. I don't know. Tuning in from Zimbabwe. Hello from Zimbabwe. That's very cool. In the future, can you teach us how to evaluate the models? Yes. So we need a way to query on image. Here we go. Let's combine this from typing. Let's import— I've got caps lock on. Now, union is used Query. We want query to be— I believe I can use it as union string or bill.image. And this is going to be a torch tensor. If query I want to default to using this, actually. Oh, I like the pipe operation. Look at us getting all fancy with our Python 3.10. Claude is very helpful for things like this. It'd be a tuple. That's beautiful. Uh, YAML, I think, yeah, is like a 20% decrease on, um, JSON, uh, tokenization. Depends, obviously depends on your fields, but I'm gonna try that in a future experiment. But to me, it's like, if they've done it in OlmocR with YAML, like these are pretty big models, right? And they've got some great results with YAML. It might be worth a try. And I mean, the conversion from JSON to YAML isn't that hard, and vice versa. So YAML, yeah, I was surprised by this as well. To me, it kind of makes sense depending on the spacing you're generating with YAML. But JSON, if the model's directly generating JSON a lot— if you're fine-tuning, that is— a lot of the decoding tokens are spent on the structure of the JSON. So YAML to me might be a bit more just straightforward. You're not spending so much time trying to create brackets, I guess, and, and, uh, commas. So let's handle the query. Where's Cloud? Beautiful. Oh, take care, JLN Marshall. Prep for class tomorrow. Hey, you have a good day tomorrow. Have a good sleep. Take care, my friend. I'll see you in the next stream or in a future stream. Matches query of input string or image to target input embeddings. Returns tuple of scores, sorted indices. I could improve, improve that later on. But for now, note, for non-normalized embeddings, use dot product for similarity. For L2 normalization, use cosine similarity. Is this correct in terms of similarity? Why would I use L2 normalization? We got a fair few many tabs open. We're researching here. These are going to be massive, so maybe we only get the top, top K back. Yeah, I think I only want the top K. Oh wait, is this going to return scores for all of them? Mm, I see. Whereas, okay, so we want I want the sorted scores back. Beautiful. Okay, this is why we write things out. Nice. Oh no, that's not what we wanted. These are already going to be sorted. Okay. We were doing a double sort there. Beautiful. That's what we want. So we get back the top 100. That's enough. Instead of getting back the top 10,000. Nice. Now we should also be able to put an image in there. Let's put an image in there. Banana bread. Nice, beautiful. Okay, now if we have a look at our sample, this is an easy way to check. Boom! Banana sour cream bread. All right, that was an example of how our banana works. So we can sort— search for a natural image, and we get back a recipe. Simultaneously, we can search for text. Our function is now multimodal. Right, we search for the exact title. Let's call it test image query. This is from the image, and then There we go, banana sour cream bread. Now they're actually slightly different recipe ID, recipe ID, but they're still the same recipe, you know. That's an interesting thing to note. Let's put in here. We still get 70 minutes, 75 minutes. Still get back this one is number 1. Okay, interesting to note, we might have to re-rank this. But that just goes to show we can use our query function with an image or text. So I think we show the top results for our image. So we want to display the results from matches. And then— and how they differ from— now let's write some extensions here. So we're up to here. I'll be back in a minute. Shoulders a bit sore, so I'm just going to do 20 squats. Okay, let's display. Because we're going to get back image and text matches. Let's display. Good idea. This is what we get back. Oh, this should be text. So we have the dataset. I think we can go— let's say we had dataset_train_select and then we had 34, 56, 75. Okay, we can select from our dataset. So we want to get top 3 samples by selecting from the dataset. Now, of course, we could do top 10, top whatever. I'm going to just stick with top 3. So now we want to— we've got our scores and indices. Beautiful. Okay, so we can do it like that because our dataset is ready to go and we've got all the information there. So I want to get scores and samples together. For score sample in zip This is how we could do it. Um, zip, we want score. There we go. Okay, so we can do it like that. Now I would like to next turn this into a function. def get_top_n_scores_and_samples. Scores and then top indices. Beautiful. Now the query— So I need an input. Let's define this. Def display, uh, query. Results. Input query results equals top, and then check if query is text or image. Display query as text or image. And then display text match results. Image text is If we go— where's our— what do they call it here? Let's get Claude to write us some HTML. This is one of my favorite use cases of AI. So we're gonna get just a a helper function. Why not use cursor or anti-gravity? Any reason for not using clawed code? Do I need to use them? These are just simple use cases, right? I don't need another app. I like less but better. Beautiful. Now let's see what happens when we get back. That'll do for now. Now let's re-rank. What's the model I'm running? Great question. I'm using, uh, NVIDIA NeMoTron Embed VL1B v2 as the embedding model. I'm about to run a re-rank model, so you can see the model here. This is the model I'm running. MyRAG used— MyRAG used in conversational chatbot hallucinates and takes each user message for similarity check, preventing deeper conversation on current topic. Any suggestions? MyRAG used in conversational chatbot hallucinates and takes each user message for similarity check, preventing Hmm, can you be a bit more specific? I'm not sure what's going on. I want that image to be a bit smaller, actually. That's a bit better. I don't like the text color here. Title— where's the title? These models can now run on edge devices, any local mobile workflows you could think of, apart from Apple intelligence gimmicks and voice agents. Many. We could do this entirely locally. Could run— one of the goals for the year is to run Nutrify locally. This is my brother and I's app, runs on a mobile device. You take a photo of food, this goes to Gemini. There we go. Want to get that running on device so we don't have to use Gemini. That's better. So now let's re-rank. My RAG chatbot is answering outside retrieve context from retrieval is weak or irrelevant. Every message, e.g., tell me more, triggers retrieval, giving random responses and switching to random topics. So you'll probably have to put in a check there that if someone enters tell me more, that you don't retrieve. So you could put in a function or a check of like, um, def check_message Requires Retrieval, input_message. So you go if input_message equals tell me more, return response. Else, uh, or elif equals, um, find me results on the latest tech updates, return, or do retrieval, then return response with retrieval, something like that. Now, obviously, this is hardcoded. But you could, you could build a model to determine whether or not you need retrieval or not. We get back Now let's rerank. So we're going to use the rerank model now. And so our re-rank model wants— now we can just go with our top samples. So we can re-rank just the image, or just— or the image and text pairs with our query. So I'm not sure if query re-ranking will work if the query is an image. This is all happening locally, by the way, Reckless. This is happening right on my, uh, you can't see it, but there's a DJI Spark right down here. But yeah, this could be done on an iPhone. Do I have the re-rank model? No, I don't. Let's load the re-rank model. Any reason as to why you didn't choose a bigger model? What do you mean a bigger model? The model we're using is a billion parameters. Rerank VL 1B V2. What could we use as a bigger model? Uh, online GPU providers, e.g., RunPod. I always prefer local. Just because I like, I like being able to just roll on into my office or whatever and start working on whatever computer I have. So I need to Rerank. So where's our top indices? And what's our current query? Our current query is this: test query text. And our current— let's just get these. So which items do we want? We want how many to re-rank. Let's test how long 100 takes to rerank. So I want the images top sample to re-rank, and we're gonna get If I go dataset train and then I want to select There we go, that's what I want. So what if I did item— this is going to be the text. Recipe markdown for item, and then item image for item. Oh no. And then we're going to go text to re-rank. Gotta remember these are Apache Arrow format. Boom, there we go. Okay, images to rerank. We've got our images. Now we want format our top samples for re-ranking. Fuck. So the question is going to be this, our query, and then it's going to be equals text, and then image for image, or for text image in zip. Samples to rerank, that should work. Okay. Now move. Wait, what did we set our modality to be? Image text. Yeah, there we go. Well, where did all that come from? So let's see how long it takes to rerank 100 samples. So We got Qwen 2.5 VL, which goes to 32B and beyond. I saw GPT-OSS 120 running on MacBook. Yeah, so you can always run larger models, but there comes a time where you need to trade off, uh, inference latency and results. So if your inference takes 10 times less or 10 times more time, so inference versus results trade-off If inference is 10x time but results are only 10 to 20% better, is this worth it? Same with, uh, re-ranking. Re-ranking trade-off. If re-ranking adds +30 to 40% on latency but only improves results 5%, is this worth it? So something to think about. So re-ranking on our sample, in our case, seems to be that If we try to do 100 samples, yeah, that's too long. Out of memory. Yeah, there we go. So if we try to rerank 100 samples, not only does it take too long, plus results in CUDA out of memory. Perhaps limit reranking to top 20 samples. So now we're going to restart our whole notebook. Let's make sure all of these are cleared. I don't think we need all the image paths in order. Hello from India. Oh, we've already run that. So we're going to re-rank only 20 examples. What's happened here? I've lost outputs here. Oh. What am I getting wrong here? So my re-rank processor or preprocessor wasn't loaded. So is this worth it? Wow, that still takes 16 seconds. Let's go. Note, depending on your pipeline, re-ranking may or may not be worth it. If you are re-ranking image plus text, keep in mind this will likely take— ah, that's what I should add in there. How long did Just writing some things for the video I'm going to make tomorrow. And we want re-ranking equals better retrieval results. But increased latency. So this is just before I forget anything. If you are re-ranking, it's keep the This will likely take a significant portion of your overall retrieval time. Now, if we have a look at the re-rank outputs— There we go. So we have a different— we have a different re-rank input to the output. And because these are arg sorted Zero and one stay the same, but number seven has moved up. No change. What text was that using? Banana sour cream bread. Ah, no, excuse me, we are getting this wrong. We need to do it from this sample. Sour cream chocolate chip banana bread. Edge Control, how are you? Welcome to the stream. How they can be influenced. So maybe I make a way to simply So this one was best banana bread ever. So that's the original, and that's the new. So— Let's do another helper function. Oops. And then 2x datasets, one before re-ranking, one after re-ranking, with the same fields. I'd like to display in 2 rows. Top row equals original, bottom row equals re-ranked. Top 3. I want the top 3 images and the top 3 Recipe Markdown Fields. Display the Recipe Markdown fields as a simple HTML card with only one X review showing. Use simple design and colors, e.g., black text on white background. Only. Keep designs to a minimal utility style for displaying, e.g., mimic matplotlib style plots. I'd like to run this in a notebook. Let's see what Claude comes up with. I'm using Claude mostly for the display. Miguela: Hi Dan, just shooting my shot, what's the chance of giving away an old GPU card to a hobbyist in the Philippines? It's not zero, right? It is not zero, but I don't have a spare GPU card to give away. I'm using all of mine at the moment. So re-rank time took 16 seconds. Is that worth it? Let's see what this displays. Where are my indices? I've got too many variables at the moment. Rerank sorted indices. There we go. So we do actually get better results with re-ranking. But is this worth the latency? So let's make a Gradio demo. So right now How could we get some— how much memory do we have on the Spark left? We're taking up quite a bit of memory here. So Gradio, we want— ideally our demo will enable Someone to input text. Input text and image, uh, retrieve results and display, optional enable re-ranking. And then display time, uh, slash latency throughout. So we want inputs Now for this, I think we're going to need a Gradio app.py. Here's one I prepared earlier. I need to sneeze. Excuse me. Ah, yes, we want So— And we'll go query image. So let's next go And then the next will be the output column, so I want this in line with that. Now we're going to output the top 3 images here. So I need a simp— I want a simple function of recipe to markdown card. So I want to return here. I need a function. So my retrieve function will need to So Retrieve is going to take optionally So based on the query text, let's write down some steps. Okay, that's about what we want. So we need our embedding function, which is here. I don't think that's correct, is it? Let's just put this last. Hello there, I am Ashim. I'm working for a project to make threat intelligence using dark web resources. I was assigned to the task to know how LM works and to train the AI. No knowledge at all. Hey, Ashim. That sounds like a cool project. What have you been learning so far? So output results equals retrieve query equals pizza recipes. There we go, beautiful. So we've got a retrieve function. Now we need to display what we want. Would you mind guiding me to start as I am finally a student? Who studies cybersecurity, not AI. Yeah, what have you found so far? Have you tried searching up what's your input and outputs? What's your ideal input and outputs of your problem? And then, uh, puts are going to be Okay, so I need the outputs to be— maybe we just do that for now. Where's our app.py? We don't need outputs, do we? What does retrieve output? Return format: image URL, caption tuples for GR gallery. We're just going to do an image gallery. Let's see what happens there. And then Hey Daniel Bourke. Love your vids. Unrelated question, but do you have suggestions for improving sleep? So my favorite one is probably no screens and no massive meals before bed. So as in, it's the simple ones, right? Eat 3 hours before sleep if you can. I often don't even follow that, but I generally find that I sleep better if I've eaten several hours before I sleep. And then of course, if I'm laying in bed, you can't beat a physical book. If I read a physical book in bed, I fall asleep within 10 minutes. So I try to keep my devices out of the room. But of course, I'm addicted to screens. So I always end up on my iPad at points. But yeah, they're my two main ones and be nice and dark. Or three main ones. So don't eat really close to bed. Try to limit screens an hour at least. Physical books are your friend. And then finally, really dark room. They're probably my 3 big ones. Oh, and then during the day, if you can do a lot of cardio, that also helps. So let's start off with just the images. I tried to search, but you know, the internet is quite full of a lot of parts. My inputs would be some available datasets like Cambridge University has crawled web data, text images. Do you have some videos on your channel so as we as beginners could watch and understand how things work? Yeah. Go to my PyTorch video or just go to the recent live streams I've been doing. The best way to find out though is to try, so So I'm gonna do Our scores and samples are going to be— so I actually want Nice. Okay, we might be on here. Pizza Hut, okay. There we go. Okay, we've got a Gradio demo. So what should we put in here? So usually if I wake up in the middle of the night, or if I whatnot, have trouble, magnesium is a great supplement to have as well. It's because I'm not— whatchamacallit— I haven't done enough exercise. So if I do enough exercise, I can usually sleep through the night very easily. So let's do a text query. What should we do? I want— magnesium really helps. So I just take regular magnesium at night. So magnesium, cardio, no screens, and read physical books. Really dark room. 5 big tips. So what do we want for dinner tonight? I want chicken thigh recipes. We got something wrong here. Oh, retrieve results. I don't want that. Retrieved images. Is that correct? That's not correct, is it? There we go. I need this. What error did we get? Oh no, we can't see our error. I wanna see this in my notebook. Dean Dre Taylor says Jesus loves y'all. Thank you, Jesus. I appreciate the love. Ah, we need a string. This is good. Beautiful. Let's see if this works. Oh my goodness, whiskey thighs, favorite chicken thighs. Braised chicken thighs. There we go, that is cool. Okay, we've got a demo. What else should we look up? I'm going to look up one more and then I'm going to end the stream. Who's watching and what's your favorite food? We need a recipe. For your favorite food. And if we don't get one in a minute, we will— we will party, because I've got to go. We're gonna try sardines. Recipes with sardines. There we go. Easy fish cakes. What do we get? Canned tuna, sautéed rockfish with capers, my mom's tuna fish sandwich, Werner's easy fish cakes. So none of them have sardines. What about just sardines? Fish cakes, tuna, tuna. I like sardines. What about best lunchtime foods? Hey, there we go, traditional English-style ploughman lunch. Special lunchbox chicken salad, egg tomato salad. We got mushaka. Let's look at what that is. Oh yum, aubergine potato-based dish. Moussaka style stuffed eggplant, aubergine. There we go. Greek stuffed vegetables with rice. We can zoom in a bit here. That works. Now I wonder if we go image to recipe now. So let me download this. I might need to convert this to JPEG. What do we get back? Cabbage roll casserole, pineapple and bacon, jam and oat bars. Okay, so we need to work on the image query. Maybe if I get rid of this. Let's go back to matching an image. Shout out to Recipe Teenage for this image, by the way. Mediterranean eggplant, aubergine. So that is working. The image input works here. So this was our input image. Not Lane, hello, good afternoon. Mediterranean eggplant aubergine cakes, mile-high crockpot lasagna with zucchini or spinach, vegetable lasagna without the lasagna. Image matches. There we go. So something's going on when we input an image. To our input function. Oh. Oh no, that's correct. Beautiful. Okay, so for some reason Why wouldn't that be propagating there? So the input query is coming back as none. So we're not getting back. Okay, thank you for the videos. Oh, you're welcome. I don't know why the image is not propagating here. What's the issue? Sorry, joined now. So we're building this pipeline and we're trying to build a demo so that people can interact with it. But right now we're getting a simple error of, uh, a demo not displaying an image. Ah, I see. This is better. No, don't restart, Daniel. Come on. There we go. Okay, we're on. The demo works. Yeah, I know, we want it to work with image or text, so we can go For some reason it won't let me copy and paste into here now, but there we go. So now we can do image search or text search. So the next step will be to add in text retrieved results. Then we will also— what else can we do? Hey Daniel, thanks for sharing such valuable content, helping us grow. I've been following you from Costa Rica for the past 3 years. Oh, thank you so much. That is very kind to say. So add in text retrieve results, optional. Add in generated results. And then finally publish the demo to Hugging Face so people can try it out. So that's what I want to do next. But I'm going to do that in the next stream because I've got it, or maybe actually I don't know if I'll be live tomorrow. I could go live tomorrow. Actually, no, I'm not going to promise that because I don't know if I can. And so we've essentially built what we wanted to build without the generation part. We found that re-ranking works, but it takes time. Pura vida. What does pura vida mean? Pure life, simple life. Hey, I agree, I like the simple life. So my next goal will be to publish this on Hugging Face as a multimodal RAG pipeline. And then the problem is though, because we load the embedding model and the re-rank model, by the time we load a generation model, we may run out of memory. You just subscribed because there aren't much people in this field who does this live coding. Yeah, you're so right. There's not many people who do AI and machine learning live coding. So I appreciate that, Afraz, but let me know in the future of what kind of streams you'd like to see of, uh, live building machine, machine learning and AI stuff. So that's cool, right? We can do retrieved results. Now we could even bolster this. I want to put down some extensions, other embedding types, query rewriting, um, normal text search. For example, just add in BM25 or another similar algorithm. So if we do, uh, Hugging Face BM25 You can just use that. BM25S. Fine-tuning, nice. I did some streams of those. Anchor Dev says, is live coding the future of your channel? Well, that depends. I've been doing live coding and, um, and videos recently, so I might do both. It depends what people want to see, you know. I'm having fun live coding. Live coding is obviously very experimental, and like, we're 3 and a half hours in. We might not— like, the content is not super pumped up, like, fast, watch it, do it, but it's like, it's as real as you're gonna get. Whereas, um, actual videos, like, you go, hey, here's what I've built, and it takes 10 minutes, and then I just talk about it, if you know what I mean. Whereas live coding, we might spend 10 hours building a pipeline like this, and then I summarize it in a 10-minute video. So it depends, depends what people want to see. I'm enjoying doing both at the moment. So BM25 library, Python. So that could be a good extension. I would love to watch them whilst before sleeping and while eating until you come live again. So you mean for Afraz, do you mean you, you'd watch the, uh, like the replays? Because I like YouTube. I mean, I don't know, you guys tell me, where do you watch live streaming? Do you watch it on YouTube or do you watch it on Twitch? Because the benefit of YouTube is that one, I have more subscribers on YouTube already, and two, um it just automatically saves your videos to your YouTube channel. Whereas on Twitch, I don't know if the saving to replay, if you know what I mean— what do they call them, VODs? Uh, is as good as YouTube's back, back catalog. But potentially I could just stream to Twitch and YouTube, but all the replays are on YouTube. I don't know, you guys tell me what's the best way to do it. So I think this is going to be next. Add in text, retrieve results, and display them. Optional, add in generated results from a generation model. YouTube. Okay, that's good. But multimodal RAG, we got it working from dataset to demo. Oh, we also need to implement re-ranking. Implement re-ranking. Implement, um, timing for the, uh, retrieval results, e.g., show the times on the demo. Accuracy will be hard because we don't actually know what the accuracy is. Demo Builder Nathan, how you going? Hey Danny, maybe you could use embedding models like BGM3 which provide both semantic and sparse vectors. Oh, do they? That's cool. Well, we're using multimodal here, so we're using, uh, NemoTrom and BadVL. Hi Daniel, just discovered your channel recently. I find this pretty interesting because I've been building open source multimodal security tools. Its sparse element uses Splay++ model by default, which has been proven to be better than BM-25 or its variants. Thank you so much, I'm learning so much, this is awesome, man. Oh, this is so nice. Log probs in retrieval. Yeah, we've got them here, we've got the score. This is the matching score. It's not accuracy, but that's how good of a match they are. Splayed++, I haven't heard of that, but I do need to go because I need to go to jiu-jitsu training. Is this Splayed++? Oh wow, that's cool. Like a HEPA filter. I like that. A HEPA filter of sorts against insecure inputs. You've trained a lot of Models on your own. That's, that's awesome. Well done. Okay, maybe that's something we could look in as in a future. But to wrap up this, I need to— I'm going to turn this into a YouTube video. So I need to finish our demo with the re-ranking, but I'm going to do this tomorrow, probably. And then show the results, and then potentially generate answer based on results, and then publish that demo to Hugging Face. Afraz, maybe I'm misunderstanding you. Do you want to send me a link about LogProbs? Or let's just go Claude. The problem is we don't— what's— how do we know what the accuracy is? We don't have the ground truth of what should be retrieved. This is, this is more so a demo. But I gotta go. Thank you all so much for tuning in. We are working towards building or finishing off our multimodal RAG demo. And it is working quite well, right? We can search from image to match recipe and we can search from text to match recipe and we get back image and text results because why? Because our embeddings are image and text tied together. So wherever you are in the world, thank you so much for tuning in. Keep dancing, keep creating, keep learning. And I've got to go to jiu-jitsu and do some Machine learning and muscle time. I gotta go wrestle some peeps, but I'll see you in the next one. Hey, leave a comment on any of these videos, by the way, if there's anything you want to see in the future, and I'll, uh, I'll reply to that.