Local Multimodal RAG on the NVIDIA DGX Spark | Part 1 - Creating a dataset

We are going live. Let me just make sure that this works. We're going to do some multimodal RAG in this stream. R-A-G. Retrieval Augmented Generation. Let me just make sure that this works. We're going to do some multimodal RAG in this stream. RAG. There we go, basically chatting with myself in the stream. If you do join in, please say hello. In the first stream, we need to make sure— or actually, we're going to do this by the end of the week. We want a project. All right, that's our goal. First, Victor, what's up, my friend? Hello, hello. So there's an article here that we're going to read through first. If you have any questions, please hit me up. But here's what we want to do. We got this. Let's go, um, Readme. Please consider that some of us don't have NVIDIA DGX Spark. That's okay, you can— a lot of this stuff you can run in Google Colab or another non— we don't need a DGX Spark, you can run it on NVIDIA GPU. Hey, you started the Data Science in 2022? That's cool. How's it going? You enjoying it? Oh wait, I need to change to my computer. Okay, there we go. So let's go multimodal RAG with NeMoTron. So goal: create a multimodal pipeline with, uh, NVIDIA NeMoTron models. So reasoning is— or rules, let's do a stream rule. We want 10 push-ups slash squats every hour. Now the goal is to— well, the reasoning is NVIDIA has been dropping a bunch of highly performant open source models. Let's try them out. And see how they work. So, resources, NemoTron Multimodal Blog. That's gonna start— we're gonna start there. I'll just get out of some things that we don't need. There we go. I'll copy this in here. So, how to build low-latency, accurate visual document retrieval with small Llama/NemoTron models that work out of the box with standard vector databases. Alex, how you going, my man? Good to see you. NemoTron. No, NemoTron— let me just— NVIDIA also need a little bit of help with their branding. So NemoTron equals a brand slash series of NVIDIA models. So just like, uh, Google have— has Gemma slash Gemini. Nvidia have NemoTron. So that's like their, their— that's how I understand it. All right, so if we go to Nvidia's page, we've got NemoTron, NemoTron, NemoTron, NemoTron, NemoTron, NemoTron, NemoTron, Nemo Gym, NemoTron, NemoTron Rag. This is where we're going to go towards. So multimodal RAG. What, what is multimodal RAG? So you've got a large dataset. If you're a company, you probably have millions of documents, and most of them are probably going to be PDFs, and they've got charts, scanned contracts, tables, screenshots, slide decks. So if you try to do text-only retrieval, you're going to miss important information. So multimodal RAG pipelines change this by enabling retrieval and reasoning over text, images, and layouts together, leading to more and actionable answers. So we've got two new models here, uh, Nemotron Embed VL-1B v2. So a little bit of a mouthful, but this is for dense single vector multimodal image and text embedding model for page-level retrieval and similarity search. So it's going to embed the page. And then Nemotron VL1B Rerank, which is a cross-encoder re-ranking model for query-page relevance scoring. So they're small enough to run across with most NVIDIA GPU resources, compatible with standard vector databases, single dense vector per page. That's important. We're turning pages into vectors. Now it's not only just text, but because we'll be able to turn the whole page into a vector, we're going to have the figures, the tables, you name it. Designed to reduce hallucinations by grounding generation on better evidence, not longer prompts. This is good. Multimodal RAG pipelines combine a retriever with a vision language model, VLM, so responses are grounded in both retrieved page text and visual content, not just raw text prompts. Yeah. So we want to— there's going to be a few parts to this stream, to this week. So this week, part 1, create the dataset. We want this to be multimodal. Now, of course, we're going to create our own dataset because if we download off-the-shelf datasets, that's a bit boring to me. I want to create a custom dataset so you can see what it's actually like if you're working on one of these projects yourself in a business environment. And then part 2, we're going to create a bare-bones demo with multiple models. We'll see which ones we need as we go. And then part 3, we want to deploy the system to Hugging Face Spaces as a demo so other people can try it out. So this is basically data model Demo. So that's what we're going to work on: creating a dataset, um, interacting with different models, and then sharing it as a demo where people can use. And of course it'll all be publicly available. Can we still see what's going on here? Yeah, we're reading through this article. Embeddings control which pages are retrieved and shown for the VLM. Re-ranking models decide which of those pages are most relevant and should influence the answer. I'm going to move this over here. This is where I made the thumbnail, by the way. What do you think of the thumbnail? Good thumbnail or not? Let me know. There we go. So what we do is we retrieve with embeddings and then we re-rank. We're going to create a diagram shortly. Using multimodal embeddings together with a multimodal re-ranker helps keep generation grounded in the correct page, uh, images and text. So, are designed for developers building multimodal question answering and search over large corpora of PDFs and images. Okay. So that's a good thing. PDFs and images. So want a multimodal dataset, PDFs and images, e.g., the PDF can be embedded as an image. Single vector dense embedding model that efficiently condenses visual and textual information into a single representation. Yes, it's kind of like a CLIP model if you've ever used CLIP, but these have been specifically focused on documents, I believe. So they do Digital Corpora 10K. I think that's a great, um, example of what you should do is 10,000 documents. Earnings_v2. I think we want, want a data set of at least 1,000 to 10,000 samples to practice with. Someone says please fix resolution. Is there something wrong with the resolution? Let me know if in the chat if the resolution on the stream is bad. On my end, it looks okay. Wow, the re-ranker improves retrieval accuracy by 7.2, 6.9, and 6% per modality. So pipeline, we're going to go dataset embed to vectors. Store vectors. Retrieve based on query. Optional query rewriting. Re-rank the retrieved items. Show retrieved items plus potentially generate an answer based on the items. Resolution is good. Note: image and text modality means that both the page and its text extracted using ingestion libraries like NV-INGEST are fed as input into the embedding model for more accurate representation. NV ingest. I'm going to star this. Okay, we probably want a bit more similar than this. Uh, sorry, simpler. To begin, this is like a production setup. And VNGest, we always start simple and then we can overcomplicate later on. And yeah, see, we, here we got— so I'm just going to start writing definitions. And we have multimodal. Equals, um, retrieve, uh, sorry, embed text images in a similar embedding space. Two modalities: vision and text. This way we can not only search via text We can search via images as well. So it compares to other models here. It does well. Training technology: SigLIP2 vision encoder. Okay, nice. Jean says, first, thanks for your 25 hours tutorial for PyTorch, but it's been a while. You seem okay with the fact it's still good for 2026, 1.9 version later. What I wanted to ask you now, if you could please— what do you mean, Gene? Is it still good? Yeah, definitely. So the, the course, the PyTorch course teaches the fundamentals, so all of that is still valid in 2026. Yes, there will be newer versions of PyTorch, but my course basically teaches the bare bones fundamentals and and a lot of it is still the same. For fine-tuning, I ran into a lot into the authentication error. Any solutions? Can you be more specific with your authentication error, please? How organizations are using these models. I want to extend the interrupt controller to support a low power state. Show me which spec sections need— wow. Domain-heavy storage and infra docs. So IBM Storage treats each page of long PDFs, product guides, configuration manuals, and architecture diagrams as a multimodal document and embeds it and uses the re-ranker to prioritize pages where domain-specific terms, acronyms, and product names appear. So this is a great example, uh, each page of their documents as PDFs. ServiceNow chat over large sets of PDFs. ServiceNow uses multimodal embeddings to index pages from organizational PDFs and then applies the re-ranker to select the most relevant pages for each user query in its chat with PDF experiences. So what we do is we embed multimodal with Llama-NemoTron Embed VL1B V2. That's a mouthful. And then we re-rank. Let's try it out. Okay, so we've done enough reading now. There's also NemoTron OCR and then embed. Okay. Models, we have, um, Multimodal Embedder, and then we have Multimodal Reranker. All right, where's the demo code? Let's just see if we can get the demo code to work. Okay, we've got some code running. Beautiful. Now let's switch these up. Can everyone see that? Are you going to train a multimodal dual encoder? No, I'm going to use a pre-trained one. So for us, if you're having trouble, uh, in instantiating Gemma, how should I put it in the access token in VS Code? So I would go to hf auth login, this command. There we go. So read the docs here and see how you can log in locally. Okay, didn't let me send that through, but yeah, just search this. For some reason I can't send that. YouTube just Okay, YouTube won't let me send that link in chat. But anyway, give that a shot. So we need to create a dataset. Now I like food. Because Nutrify is based on food. So what if we wanted Nutrify to be able to, say, take a photo of recipes and extract information from that, right? This is my brother and I's app. Let's say we wanted functionality for this to take a photo of a recipe and extract data from that, or maybe someone wants to search a recipe. So our dataset should be inherently multimodal. Some ideas: 1. Text plus image versions of recipes can natively test the text-only, image-only, and text plus image Retrieval settings. I think that would be fun and also easily creatable recipes. Um, two, food menus. Text plus image versions of food menus. Can natively test the text-only, image-only, and text-plus-image retrieval settings. So let's just go to Kaggle. Over 2 mil. Okay, this is pretty good. Non-commercial. Okay, generating personalized recipes. This is good. And we get interactions as well, which recipes have been viewed the most. I think we download this dataset and see what we can get. Let me just sign in to Kaggle. So we've got some data. I just downloaded it from here. So what we could do is could get recipe info as text, turn into PDF, use PDFs/text as multimodal retrieval. Okay, I have a good idea of what we can do. So let's inspect some recipes. So we've got about 700K recipes, but I wanted to rename this to Recipes. And let's get pandas and start exploring some stuff. Okay, that's too big. Okay, now if we get Nice. So we want to get a good diverse number of recipes. How many steps there are? So 231,000. Okay, now what was the Is there a Jupyter Notebook Colab link? What for? For what we're working on now? Not yet, because we are working on it currently. So we also have interactions. Excuse me, my coding is a bit off at the moment. Okay, rating. Review. Nice. Okay, so let's find the recipes with the highest rating. We have 1 million interactions. And we have nSteps. Let's check out nSteps. And Ingredients is another one. Okay, so maybe we cap it at 20 ingredients. So we want to create— it looks like— how big is this? Maybe we cut it off at 20 steps. 20 steps. Idea: cut off recipes with— or maybe less, we'll find out. 20 steps and 20 ingredients. So numsToSample equals— so for num ain't numbs to sample. Less than or equal to num. We're doing data inference here, or data, data analysis. Okay, so it looks like the 95th percentile is going to be somewhere around 15 steps. So let's just put a percentage here. So 17 steps or less, I think that's still too many. Maybe 15 steps, that's too— still 200,000 samples. Now we do the same for ingredients. Okay, I think 15. We cap them at 15 ingredients and 15 steps, or even 20. Our hunch of 20 was right. Data analysis course? Not specifically, but yeah, you're right, the data is just as important as the algorithms, probably more important these days. So let's get a sample. I got some ideas. Data filtering. Want— let's get 10K samples. Um, step 1, sort to highest rated samples. Step 2, we're going to assume that those are good. Step 2 is, um, sort to end steps less than or equal to 20. Less than or equal to 20. And then Step 3 is evenly sample recipes with 1 through to 20 n steps plus n ingredients. Does that make sense? And we can append some reviews as well. We can also find the ones with the most reviews. So recipe ID 2886 has the most reviews. Let's find that recipe. This is Banana Bread, 1999. Why won't this give me the value? 2, 2, 1, 6, 8. There we go. 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13. Sorry, has the most ratings. So let's do Nice. I'm gonna call this one banana bread as an example. Best banana bread and Butter, granulated sugar, eggs, bananas, all-purpose flour. Now let's see if we can pass this into— is NemoTron running? Let's find a model that's running. There we go. Now we go to Gemini. Oh, I'll sign in here too. Rara, Rasputin! So it wants me to verify sign in. So just had to verify Gemini. But we also have best banana bread. Boom. Okay. So this is a pretty good option. How big is this model? Can we run GLN image? There's also Flux. Let's try get Flux running locally. So we want to go Just to see if we can— How does the DGX Spark go with generating an image? Because we could do Yeah, Shadow Lilac, you're exactly right. So what is going on here? So yeah, that's what I'm thinking of. So let's go back to the pipeline. Is So recipes equal searchable via image plus text. So can either do reverse image search, e.g., input image, get back recipes with relevant ingredients. Images. Photo of— yo, Green Code, what's up? Photo of tomato. Recipes with tomatoes. And then text: what can I make with cream and spring onions recipes back with. The reason why I'm trying to generate images is to increase the multimodal dataset. So creating multimodal dataset equals, um, have JSON with recipes. Can easily turn these into PDFs based on text and then e.g. just spin up Claude to create a random PDF recipe generator with seed recipes, right? But this will still be text only because Claude will just create a PDF with text, like it'll just place the, the text around different locations. And then next we want to text-based PDFs. And then finally, optional, um, use Image Generator to add in, um, a recipe hero image in a placeholder. So that there is some form of multi-modality rather than pure text-based PDFs. So probably what we'll start with is the text-based PDFs and see how far we can get with that. I wonder— Flux2 didn't work. Let's try Image Turbo, Z-Image Turbo. Where is ImageGen? Just want to create this as simple as possible. We'll just let that download. So let's filter our dataset. We want 10K, 10K samples, and then we're going to create PDF dataset from that and then embed the PDFs as images. And then we can do reverse image search. So we want to create the two, two-headed pipeline— text search and then image search. So what we want to do is collapse this DataFrame onto ID and then get the average rating and then reviews. So maybe we also append reviews to different How do I collapse this DataFrame to group recipes to average rating? Eg, I want a dictionary with recipe ID. So we're going to collapse this. Group by— yeah, that's what I want— average ratings. Do you think agent orchestrations and its knowledge is learning worthwhile? Just curious, you know. What do you mean by agent orchestrations? If you can give me a specific example, I'll, I'll let you know, but agent orchestrations is very, very broad. Okay, we've got recipes with average ratings. So we'll append this to the dictionary— oh, sorry, the other df. Okay, now how do I get a dictionary of, um, recipe ID to list of reviews. Okay. Nice. All right, now we're going to get— we're going to upgrade our recipes with their list of reviews. And actually, some of these reviews might be extremely long. I want the first 5 reviews. Okay, now— oh, we're generating an image! I can hear the spark going off. How long did this take? Yeah, Shadow Lilac, the reason Uh, MapReduce, Apache Beam to group the DataFrame? No, we're only working with 200,000 rows. We're keeping this in the small, small project, something that we can just run locally, you know. If we had like, yeah, a billion samples, we'd have to use something a bit more. But I haven't actually tried out CuDF or CuPandas or whatever it's called. So let's say like making unified system from multiple fine-tuned specific role AI models as a single system, just like how robots need a vision model with text-to-speech and other stuff's all put together. For sure. So my rule of thumb with like, is something worth learning, is like, would— do you personally want to build that thing, right? If you want to build a robot that can see and do text-to-speech, like that's— robotics is exploding right now. To me, that's, that's worth it. Like, does it raise your curiosity enough to to build something, then that's definitely worth it. Like, if the fact that you're asking that question to me says that it's worth it, if you know what I mean. Yes, it's going to take some effort, but it's like, what else are you going to learn? Okay, nice. So we know that we can generate some images if we want. So the reason I'm thinking of images, so let's just go banana bread recipe card, right? Something like this. We don't actually need exactly like that, but just so we could do a placeholder image, like we could get Claude to, to generate a PDF with text and then we can just throw in a placeholder image. But let's see how good these models are actually at doing cross-modal retrieval. So ImageGen works, that's great. Let's enhance our recipes dataset with— so maybe we only want 10 reviews. We'll cap the max. Reviews at 10. Yeah, so some of them are going to have— these turned out nothing like the picture, very dry and stayed in. So if we put in a recipe ID, we want to Oh, random 10 reviews. Num reviews to sample. So wouldn't you need to preprocess everything unless you want to wait a staggering amount of time on Hugging Face's CPU generating images? So Um, what I'm going to do, Shadow, is, uh, this stream is basically we're going to make a static dataset. So we'll do 10,000 samples. We may, we may add generated images, or we may just do PDFs. Um, but I'm just going to do 10,000 samples as a static dataset on Hugging Face, upload that to there. Uh, and then yeah, we can preprocess it before we— well, when we upload it will have preprocessed embeddings. And so when we're doing Hugging Face, um, a space, it'll just be loaded into memory. And because it's only 10,000 samples, it should just be able to hold it there. So yeah, by the end of this stream, we want everything uploaded to Hugging Face and, uh, um, preprocessed and ready to go. So my, my goal now is to just get 10K recipes, um, evenly distributed from 10 steps— or sorry, 20 steps and below, and 20 ingredients and below. And then we can optionally enhance those with some more information. But I think if we start with the JSON and then we either get Claude to just generate us a PDF generator, so then we go recipe JSON in and then PDF generator because they're all going to be the same spec in the JSON, the same— what's it called— data structure. We can get a PDF out, and the PDF out will have, I don't know, 10,000 random combinations of layouts. So I'll show you what I'm thinking of. So number of views to sample equals 10 if len target reviews I think that should work. I don't need try there. I think we might only get 5. 10 might be too many. So let's just create one example of what I'm talking about. So average ratings. Let's see if this function works, actually. Df_recipes.random_reviews equals df_recipes.apply extract_reviews. How do I do this? Lambda— oh no, sorry, is it recipe_id? I'm bit rusty at pandas. ID. Sample larger than population, or it is negative. Okay, if they've got no reviews, that's a good point. Beautiful. Random reviews. Oh no, I shouldn't have done that. I got an idea. How do I find Is it not na? That's it, there we go. Random reviews, let's have a look at these. Nice. Let's get clawed. Is this all local? Are you using cloud providers/services? Uh, this is all local. We're running on the DGX Spark with a little bit of Claude. Beautiful. So you can see I'm running on the Spark, and we can go NVIDIA SMI. GB10. What has Claude put out here? So Claw is going to help us with the visualization. Let's now add our def average ratings. This is a ID to rating. Or maybe we get rating count. That's a good idea. So do we— because then we're going to have— if we do average rating, we're going to have one— if there's a recipe with one 5-star, it's going to have a 5 rating. So I actually want rating counts. So recipe ID, can I get this? Nice. Okay, beautiful. That's what we want. Where's my VAR? Rating count stick. We're using count recipe count as a proxy. There we go. Now we can sort by ratings. What's Claude come up with? Beautiful. Okay, we're not getting— we need to format the list. So— Now we've got a new column for number of ratings. So what do we get for— So, one rating. How many recipes do we have? We want to make sure they have, I think, 20 total recipes. Oh, sorry, with over. I want to make sure this is over. So maybe we subset it to— it needs to have 5 ratings, so then that's 50,000. And then it needs to have up to 20 steps and up to 20 ingredients. So 5 ratings I think is a good cutoff. Nice. What is stuffed idlis? Cool. Wondering how you learned. If I learned from someone, my lag will be there. If I learned the way you did I think I can solve real world problems. Everyone learns differently. Everyone creates differently. If you want to know how I learned, you go to my website, mrdburke.com, and look at all the blog posts here. Okay, we're making good tracks here. And are we an hour in yet? We probably are, aren't we? Oh, we're an hour 20 in. We're supposed to do some push-ups and squats. Who's joining in? We've got 10 push-ups and 10 squats, so Next, before we get distracted, next, make a subset of recipes. So criteria, we want 20 or less ingredients because otherwise it's too many. 20 or less steps, and then we want, um, 5 ratings or more. Use this as a proxy for quality. And then we do, um, quantity or qual— no, quantity of reviews rather than just 5-star only, because some recipes may only have 1x 5-star, and then that is their mean. Quantity of reviews also adds into the diversity. Now if we really wanted to— extension— if we really wanted to get a diverse subset of recipes We could embed them and then cluster to top 1K and then sample from clusters, but we'll think about that later. Okay, who's joining in? 10 push-ups, 10 squats. Okay, feeling good. Be right back, just gonna go to the bathroom, and then we will create our subset Recipe 10K. San Pellegrino. Tick. Now we got to create a sample. So let's now— 20 ingredients or less, 20 or less steps, 5 ratings or more. These are our qualifications for being a sample. So we've got DF Recipes. We have 230,000 recipes, but filter recipes to 10K samples, uh, to make a small interactive dataset. We'll filter to 10K samples. We want 20 ingredients or less, um, 20 steps or less, 5 stars or 5 total ratings or more. Ideally our sample is evenly distributed across number of— we'll go here— n_steps and n.ingredients. So if we just did a random sample of df_recipes random, We go n_steps.value_counts. Yeah, we get all of these, so I think 20 is an okay number to stop, but we see most recipes have about 8. So say we want— what is it, 40 samples each? Oh no, 500 samples each. Up to 500. Yeah, okay, we should be able to hit that. And then N Ingredients. I think we should be able to hit that as well. Ayuki says, hi mate, do I use AI for your work? Yes, I have Claude open right now, but, uh, just to write helper functions, you know. At the moment I don't use Claude code. I haven't. I'm probably like the only engineer in the world that doesn't use Claude code right now. I guess you could say this is basically Claude code. But a lot of my stuff still is driven by me. And then I'll use AI for like random helper functions that I kind of already know how to do. But put it this way, if I'm building a system, I want to like build it 80% there with my own brainpower or like web development. So I'm not a web developer, so I use Claude for web development, like to create just random little websites. I'll use it for that. But for a lot of this sort of the structural stuff, it'll still be my own brainpower. Once I've done something with my brain at least once or multiple times, then I'm okay using AI because then it'll speed me up. But I still like to— and I'm not gonna lie, AI is a better coder than me for sure, right? But anyone who grew up playing video games knows and doesn't really care that a computer is better than you at something, because that's been the case since you were growing up. The computer has always been better than you at certain tasks. So where I add my own value is like, okay, here's this project I want to do. I get it along the way, do it myself, practice, and then AI comes in and helps. So once I know sort of the ins and outs of it, it'll definitely help me. But for things I don't know much about, like web development, designing websites. Like, I can do HTML and actually not that much CSS, to be honest, but, um, I get AI to help for that because I personally just want to focus on building the workflow and then AI can do the other stuff. But I'm definitely not adverse to using it. I just think that you as a developer, your main role is to deliver code and projects that you have personally verified to work. So whether AI wrote the code or you wrote the code, it doesn't matter. It's your job to personally verify that it works. Um, and so do I worry about AI being better than me? Absolutely not. I already know that it's better than me. That's fine. Um, where my value add is, is having the idea to even work on something like this to begin with. So let's keep going, but that's my sort of take on AI. Love it, but also still really, really think that it's way more valuable to have skills plus AI than to just use AI from the, from the start. Sure, it'll give you a little head start, but it's the same as AI is like Ozempic for weight loss. So AI for programming is like Ozempic for weight loss. If you like develop the habits to just be a healthy person and a fit person, that's going to solve— like, take you so much further than just like injecting GLP-1 drugs and then losing a dramatic amount of weight in like 12 weeks. How many people bounce back from that? I'd love to see the numbers on that. I haven't— I haven't got any data behind this. Sure, is it helpful? Yes. But ultimately, what's the end goal? The end goal is for you to develop habits of being better, whether that's for health, whether that's for programming. And so do you still need skills with AI programming? 100%. I think they become even more valuable because now you've got like this jet fuel or Fast and Furious. Think about if the cars have like NOS, the nitrous accelerator, but the car is crap, right? You're going to add fuel to a crap car and it just might explode, right? But if the car is built well and you add the extra motor, then it's gonna haul ass, right? So same thing with AI and programming is that if you already have good skills and you add in this AI pair programmer to help you, you're both going to go good. And in the future, is AI going to be probably better than a lot of human programmers? Sure. But where did all the data come from? It came from human programmers. So we still are in that paradigm of like, for me, I like to publish tutorials online as well, because even when the AI can help you with new stuff, uh, with programming, sorry, where is all the material going to come from for the new tutorials? Like, when something new comes out, where's AI going to get training data from? So that's a long-winded way of saying, big fan of AI, use it in specific use cases, and don't neglect your skills just because a computer can type out thousands of lines of code in a couple of minutes. That's an awesome superpower. But if you want to build skills, just remember, the computer writing code for you isn't necessarily helping you with your skills unless your skill is like project management. And in that case, run 20 Claudes at once. So let's go from here. Mohammed asks, what is he trying to achieve? I'm building a multimodal RAG pipeline. And right now I'm creating a dataset. So I want to use NVIDIA's NeMoTron RAG models for a multimodal, uh, setup. And which one have more demand in the market, computer vision or NLP? Both. There's a demand for all kinds of computer usage at the moment. So I've been on my radar is NVFP4 Forward Pass with BF16 Accumulate to achieve the 500 TFLOPS FP4, but one must use Transformer Engine, which is a garbage API and needs a lot of setup. Yeah, so that's my ridicule of a lot of NVIDIA APIs over the past couple of years, is that the setup to, to do them is just insane. The reason why Hugging Face is sort of just— Transformers has exploded because you do pip install transformers. And yeah, sure. Are you going to get the most optimal performance? Absolutely not. But at least you can get it up and running in, in one line of code. So let's create our dataset. We need to do a sample of 500. I want to— how would I evenly sample them? Across two axes. Maybe we start with one, like Linkin Park. Start with sampling over n steps or n ingredients. I actually think n ingredients is probably more helpful. So we want to go, um, ingredient_count_dict equals Let's go num ingredients 0 for num ingredients in range 1 through to 20. What does this look like? So 1 is the minimum. 1, 2, 3, 4, 5, 6. We want 21. Okay, so we want 500 samples or thereabouts of 20 ingredients in each. So what we're going to do is— or actually, can we do this with pandas? Let's see what Cloud says with pandas. What's the way to do this with pandas? I'm sure we'd create like a Lambda function, maybe. This is my df_recipes. How would I create a 10K, um, sample df with evenly distributed, uh, n steps samples up to, um, 20 samples. Sorry, not n steps, n ingredients, up to 20 ingredients. For example, Um, number of samples with 1 ingredient, 500. Number of samples with 2 ingredients, 500. Number of samples with 20 ingredients, 500. Does this make sense? So this is what we're going to do. We're going to use Claude to teach us how to do it, and then we're going to do it ourselves because I know this is possible. Create stratified sample. Ah, okay, we just create multiple data frames. Okay, let's do it. Could I share that article? This one? So the column to target is n ingredients, samples per group 500, max value is 20. So let's go— that's the word I was looking for— create a stratified sample with equal samples per group. And we want to go sampled DApps. So for i in range 1, max value plus 1, group equals df, df column. Ah, this is good. I like this. Uh, it looks like that model won't be supported for general purpose images since NVIDIA explicitly states its trained use case is images that contain text. Okay, that's a good thing to know. Let's look at the model card. The model supports page images containing text tables. Okay. I think it's got Siglip. Okay, well, we'll just create some synthetic PDFs and we'll treat it— those as our images, like PDF recipes. So now we go df_recipes_10k. Recipes. Boom. Beautiful. Okay, we get 500. Oh, but only one. Okay, that's all right, we don't need one or recipes with one ingredient. Yeah, 1 is not going to be very high. That makes sense. Okay. So what if we go— where are my ratings? Now I want to make sure— let's see how many of these Lilac, you saying that the, the NVIDIA VL models are sketch? We're going to try them out. We've got to create a dataset anyway. Okay, we've got one total rating or higher. Let's see if we can go DF recipes ratings over Num ratings over 5. Okay, let's filter this. Okay, that's not too bad. Let's up this to 600. There we go, 10,000. Beautiful. The problem is the way SignLIP is connected to the decoder transformer is likely trained on OCR, so you aren't comparing Align transformer embeddings to ViT embeddings. Well, we'll just see. We'll see if it even works to begin with. I think creating the dataset will be helpful, and then If it doesn't work with the NVIDIA NeMoTron VL embeds, we'll, um, we'll try a different model. Now let's see. I want to make sure we get 20 steps or less because otherwise it's too many steps for me. Yeah, so we only lose about 1,000. Beautiful. 10,000 samples, that's what we wanted. And on average they have 9.2 steps. Okay, so we've got our 10K recipes. Now we want it in data. Wonderful. Okay, now let's have a look at some example samples from Recipe10k. Nice, we get some reviews. Gingerbread pound cake. 1, 2, 3, 4, 5, 6. 6, 7, 8, 9, 10, 11, 12. There are a lot of reviews here. That's all right. I wonder if Siglip recipes 10K. Nice creamy cauliflower salad. Now, how long— where's a good open source model? Nice, this is good. So— Yum. Okay, who, who's hungry now that we've been looking at banana bread? So I want to see if we can get— how do we get NVIDIA NemoTron on DGX Spark. That makes total sense. I'm going to create a JSON, actually. Nice chocolate mint smoothie. Personally, I don't think I would like that, actually. Oh my gosh, look at this rating. This is a review, random review. I'd give this recipe 69 stars and I could— if I could. My boyfriend and I had it last night and I don't think I've ever had poop better than that. Okay, don't blame me. Blame the data, but this is from the internet, so we don't know what we're ever going to find, right? So let's now save to JSON with open Beautiful. Cool. So this is what I mean. This is good. Now I wonder how quickly We could. Thank you, Adarsh. I really appreciate that. What time is it in India? I need to work out a better live streaming schedule, you know, so I can figure out, uh, when is best for everyone. The problem with Australia is we're kind of in the middle of everyone's time zone. Oh, and look at that, we're 2 hours in. Let's, uh, let's do 10 push-ups and 10 squats. So we've got to add to the count. I kind of like this. We could make it a bit more complex. So right now what we're doing is we're creating a diverse implementation of, uh, basically JSON input to recipe output. And then we're going to create this. So let's go next is turn this into a script workflow I can run like so. Input JSON with recipes, um, one per field. Inputs, we want JSON plus, uh, number of recipes to sample, e.g., for experimentation. So I could just try 100 recipes at once. Outputs: folder of PDFs plus images, uh, folder of Recipe to PDF and image outputs. Images. All right, this is good. Claude is phenomenal, but an underrated use case is these days is being able to create synthetic data. So while Claude is churning away at making some code, let's get NemoTron 3 running on the DGX Spark. I wonder, can I just Let's just follow the docks, Daniel. Come on. Nvidia actually also just unleashed some So we're going to try get NemoTron Nano V3 running, and if we can't get that running, we might have to try GPT-OSS 20B. So let's— Nice. This is cool. So we're going to install NemoTron v3 and see if we can get it to run. We're just going to follow this exact workflow. So I want to build llama-ccv with CUDA support. Buying NVIDIA DGX Spark is really worth it if you use it. Yeah, 100%, you don't need anything, but if you try it out If you see a use case for yourself, go for it. Didn't you want the embeddings? I don't think you get much out of Llama CCP, so probably you'll need the raw PyTorch code. So we'll get the embedding shortly. But what I'm going to do is get Nemotron 3 Nano, and we want to see if we can run it locally to either 1, do query rewriting, or 2, um, do whatchamacallit, turn it into a— how do I say this— turn it into markdown, or turn the recipe into a readable format. Because we've got Claude which is creating our— whoa, this is good. Okay, let's see. I'm just gonna go to the bathroom and we'll see if one, we can run NemoTron v3, and two, we can create some synthetic recipes— or not synthetic recipes, real recipes, but synthetic PDF versions of those recipes. See, Alex, thank you for tuning in. I'll see you in the next stream. Hey, I appreciate you hanging by. Now let's download NemoTron. Saq Lane 3470, how are you, my friend? 40 gigabytes, okay. So while this is downloading, let's try this. Why not create a pretty f, f-string template with a markdown with the LLM? It's so— it's deterministic markdown. Great idea. Yeah, we don't necessarily have to reformat it. I'm gonna write some extensions here. Uh, I'm using Claude at the moment, but I'm using it via, uh, Claude.ai. I haven't used Claude Code yet. So let's try out this script. We're gonna need some dependencies, aren't we? Yo, Hussein, what's going on? Shiva, how are you? Love my PyTorch course. Oh, thank you very much. Really appreciate that. Okay, let's see what we can get. I want to do Let's try 200 DPI. So we're creating a synthetic dataset of recipe PDFs, and we're doing this so we can test out our custom pipeline for modal— multimodal RAG. Well, we'll just do 10 to start with. And recipes, 10K. Error. What did we get? Wow, that was quick. Nice, cool. Does this make sense of what we're doing? Multimodal rag. Beautiful. We might make some tweaks. Hello sir, I'm doing the same thing here right now. Matthias, you're doing a multimodal rag pipeline, are you? So let's see how our download is going. 40 gigabytes downloaded, beautiful. Okay, now can we run No way, this is cool! 50 tokens per second. We're running Nemo— NemoTron 3 Nano locally. Oh, ladies and gentlemen, here we go! This is running locally. Tags. That is cool. So we've got about 30 seconds of generation time at about 50 tokens per second with Neemotron 3 Nano. Very nice. Now remove the tags section. Wow, this is a good model. You're using Quentri VL embedding and re-ranker for this? Yeah, so I'm gonna— they are also other models. I'm gonna try them, uh, with NVIDIA's recent VL models, so these ones here. But, um, definitely gonna try Quen 3 VL in the future. This is awesome. Now, did it get it correct? I think that's not correct. Protein, 87. Where did that come from? Nutrition. I wonder what order the nutrition's come in. Am I going to be streaming every day? Most days this week, I think Monday, Tuesday, Wednesday, Thursday, I'm doing something. I got to create this in a video. And then Friday I'm away. Okay, I like this. So let's go back to Claude. It's still going. Great. So we're going to find out the order of the nutrition. So if we go to There we go. What is PD? Percent of daily value. I think it's going to be P daily value. Okay. Okay, this is a large serving though. We need— I think the recipes are actually missing amounts. I wonder if we can get I see we're actually missing amounts. That's okay, we can put that in extension. I've run out of Claude usage. I'm gonna buy extra usage of cloud. Okay, I'm back in the game. Nice. Uh, personally, what subscription do I find to be the most bang for buck? Uh, I don't know. I'm kind of doing it non-optimizationally. I have Gemini, I have Claude, I have ChatGPT, all on different subscriptions. If I was smart, I would probably put them all into one, but I haven't done that. So let's, let's try out this new pipeline. Beautiful. So now we have images, PDFs, and an output JSON. Nice. Okay, so let's upload this dataset to Hugging Face. Oh, thank you very much. Oh, hey ET, thanks for tuning in. And thank you to Smarth Cadet, appreciate your work and courses, Daniel Bourke, making such great content. That's very nice of you to say. Love from Haiti. Oh, love from Brisbane, Brisbane, Australia. So I think we're pretty close to having at least a good Okay, so we've got themes: Cookbook, Modern Minimal. Now what do we want to do next? We want to create a Hugging Face dataset of images and ground truth information. And then we want to save the dataset to Hugging Face so that we have recipes, because what are they running off? Running off this. I think we got a good pipeline here. Boom! So now we can do query rewriting as well. Can you give me a brief explanation of what this is meant for? I'm kind of lost. TBH, we're doing multimodal RAG, so Multi-modal search with visual document retrieval with LLaMA and RAG models. So we want to— traditional RAG, retrieval augmented generation, you would enter text, that text would get embedded. So that's called the query, that embedding would get searched upon a database of other embedded text, and then they would get matched. So the database of text could be like your, your business's whole database of documentation and workflows. Say it matches to 10, then you would take those 10 documents and answer the query with a generation model. So basically you would feed the LLM, hey, here are the 10 most relevant documents based on this query, please, uh, refeed, uh, please answer the question, the query, the original user input based on these documents. So that was a text-only workflow. Now we're going to do multimodal search with visual document retrieval. And so what we're doing right now is creating a dataset, our own custom dataset to explore this. We are creating a— we've got recipes, 10,000 recipes. We're going to turn them into images or PDFs, and we could also use the text as well. But then we're going to upload that dataset to Hugging Face. And then our system is going to be, for example, you search, what can I make with tomatoes and Parmesan cheese? Right. And then it's going to find recipes with tomatoes and Parmesan cheese. So using these queries, ideally, right, we don't know 100%, but that's the theory. We're going to try and build this out and see where we get to. So our next step will be to upload our database to our dataset to Hugging Face so that we could easily infer on those. And to make the embeddings, we're going to use, uh, Llama-Nemotron Embed VL1B-v2, which will embed images and text in a single embedding. So that's going to be cool. Let me know if any of that makes sense. Now, where could this be used? Well, if you're a big business and you have millions of PDFs and charts and contracts and all that sort of stuff, you could embed them using a vision language model such as Llama, NemoTron, EmbedVL-1B, V2, uh, into a multimodal image and text space embedding space, and then you could search over your entire documentation with, uh, not only text search but visual search. So that's the cool part about multimodal embeddings. But it is 3 hours in and it's time for us to do some push-ups and squats. Who's joining in? Uh, let me know in the chat if you're doing some push-ups and squats. We do 10, 10 push-ups and squats every hour because we're building machine learning and muscle. There we go. We'll add that to the count. So let's get this dataset uploaded to Hugging Face. Uh, shadow lilac. I don't get it. Isn't this circular logic? Upload the PDF of a recipe and it gets you the exact same recipe from a preset DB? Uh, no. So you could do that, you could do recipe matching if you wanted to, but we are like— you could do— this is query rewriting, right? Like, so say, um, now repeat for Um, sardines, sour cream, right? So you had sardines and sour cream. So there we go. So now we'll will retrieve from our database recipes with sardines and sour cream. Or if you're in the shop and you take a photo of cabbage, you're like, I want to make something with cabbage, and then it will retrieve all the cabbage recipes. Or if you're building a generation pipeline, it will, uh, write you a recipe based on others. Did I ever try the Deepseek OCR model? I'm toying with it for advanced PDF ingestion technical docs. No, but I should actually create like an OCR. There are some great OCR models lately. Is OCR here? Deepseek OCR, I personally haven't tried it. Like, well, I've tried it on an example document, but it worked quite well. Uh, Nanonets OCR. Dolphin. There's also Dolphin, which was ByteDance, I believe. There's Paddle Paddle OCR, and there's Dockling. Yeah, Dockling. Uh, yes, I'm going to encode the PDFs as images. I'm go— you could encode them as text and images. The thing is, I'm kind of building a dataset. I, I have the advantage here of, um, like, I have the ground truth. So, but that's, that's why we're doing this, right? Because I have the ground truth, I kind of know what it should be. So In practice, though, you may not have— how do I say— you may not have the ground truth text. You may just have PDFs. So one option is to just embed the PDF as an image, or another option is to both extract the text from the PDF, embed that, and then save it under a unique ID, and then embed the PDF itself as an image. So you have two. They both link back to the same ID. They're the same resource, except one is embedded under text one is embedded under image. So that's what we might do is create an embedding database of the text version of the recipe and an embedding database of the image version of the data of the recipe and see, see how they compare in terms of retrieval. Why not use Mistral OCR3 to convert them into markdown? We actually don't need to. Convert anything into Markdown for now because we have, uh, we already have them in JSON, so we could go straight from JSON to Markdown. So if I choose one from here, pickled banana peppers, yum. Then if we just go to NemoTron turn this recipe into markdown. So this is running locally on the NVIDIA DGX Spark. I wonder how I turn reasoning. Wouldn't it always be more accurate to extract the text even without any OCR, just the raw text, and only embed that? See, you would think that, but what if your figure has images? That's why we're doing the whole— that's why we're doing multimodal, right? Because you can do OCR on text, but not OCR doesn't necessarily work very well for images or graphics. So if we read back the models, right? So in real applications, data is not just text. It lives in PDFs with charts, scanned contracts, tables, screenshots, and slide decks. So OCRing these type of things isn't always the best, right? We've all tried. If you OCR a table, won't work, uh, won't necessarily work as well as potentially an image or a screenshot, for example. So that's why we're doing both. We're doing both text only and vision retrieval. And actually, like, the best result would be just straight up just search. But I want to practice image search as well. It'll make sense once we build the pipeline, then we can compare the return results, right? So we're processing here, we're creating 10,000 images. We don't know if we— unless we try. Food image uploads won't work. Don't worry, I got some ideas to make them work. Um, and then we want to get all the images. Are they PNGs? Yep. Now, why didn't this— oh, there we go, there's our markdown. Beautiful. I actually think if it's— if they've used SigLIP2 as the image embedder, we may have some natural images as remaining in the latent space. We don't know that for sure though. Oh no. Nice. Now this, this could obviously be improved, right? But this is just to try out. I have no idea whether this will work, but we're going to try. We need the output JSON. Okay, let's Yeah, ET Videos, I think you're exactly right. I'm going to put down here in the, uh, ex— or data extension, could be layout creation. Spend more time making a few more diverse layouts. Right now, ours are pretty basic. In the future, we could— so right now we're just trying to get the workflow down pat. We could, um, make it much more diverse in terms of recipe layout. E.g., take inspiration from, from many different recipe layouts. So let's now— we have to To create a Hugging Face dataset, we need JSON-L in this format with a folder of images. We need So we need a JSON-L in there. We'll just make it as simple as possible. Um. I need the name, right? Name. There we go. Oseket, hello, how are you? Did you end up launching that nutrition app we talked about? Is this a new course, Daniel? It could be a new course. Multimodal rag. This is some pretty new stuff. I mean, these models were only released the other day. January 6th. So what's that, not even 2 weeks ago. And yes, Oseket, um, the app is available here. This is Nutrify, very simple home page, but the app is where most of the functionality is. So you can download that on the App Store right now and try it out. So it works quite well. I actually use it almost every day. And I'll see if we— do I have an example of it running? Yeah, here's Nutrify running. This is what I had for breakfast the other day. Or for lunch, I can't remember. Oh yeah, it would have been breakfast, 8 AM. So this is going to go to Gemini. Gemini returns structured data and we get detected foods: sardines. You can update that. Pecans, pickles, dates, olive oil. So that is Nutrify in a nutshell. And then of course that gets saved and you can get a summary. And it focuses on Whole Foods, so we specifically don't do barcodes. So I need to now save this to JSON-L. How do I save a list of Python dicts to a.jsonl file. I always get this messed up. So I want to upload this folder of images. I'm going to put a.jsonl in there. Uh, it is done with SwiftUI, so my brother built that all, all with Swift. And thoughts on Gemini reliability? Um, Gemini has been our favorite in terms of usability by far. That's, that's our experience, definitely for multimodal as well. We've never hit a rate limit. Um, I suppose because our inputs and outputs aren't actually that big. Like, our input is a single text prompt with an image, and then, uh, that gets cached, and so we save a bunch on that on inference. And then the outputs are only ever like 500 tokens. We're not doing massive reasoning steps or chains or whatnot, so I haven't— I can't talk to that. But for simple input and output, we— I basically haven't faced a single rate limit yet. Okay, we need Metadata.jsonl. I might actually create a full recipe. You still remember the Tesla diagram of the, the food image engine? Yeah, we've defaulted to using Gemini. One of the, one of the goals for this year is to potentially have an on-device model which basically replicates Gemini's capabilities. But yeah, getting started with Gemini for that kind of workflow is almost too easy these days. Like, it's what, 3, 4 lines of code and then just a bunch of structuring around that. So we have metadata.json, but I want to get a full recipe. So I would like to The full recipe is going to be a dictionary with fields inside it, and then we go item full recipe. Equals full recipe. What do I use for note-taking or productivity these days? I've switched to just random notebooks and Apple Notes, as simple as possible. My approach these days is just spend way more time doing than you do like writing about it, or like text fields or markdown fields, write random things here. So I don't have a huge productivity system other than physical notebooks, and then a bunch of random notes scattered over several projects. And then of course, my notes are like code now, like as in, I, because I work on a lot of computer programming stuff, rather than write stuff about it, I test it in code and then save it in code. And then if the idea works, well, then there's code to reproduce it. So that's why we're building this multimodal RAG pipeline in step-by-step code, so that we can test it. And if it tests and we work, I can share it online, if that makes sense. So full recipe is What else do we want? Now I'm just making a field that's the full recipe, so That way we just have one field we can easily access. And then we could embed this whole thing if we wanted to. You just fine-tuned ParliGemma 2 with your dataset. Oh wow, that's awesome! How'd it go? Justin Clark TX asks, is the DGX Spark worth it or just invest in a good desktop machine? Um, honestly, it depends. Like, if you want to build your own desktop, you could probably get a more powerful machine for the cost of a DGX Spark. Um, but if the— so the big benefit of the DGX Spark is the 128GB universal, uh, unified memory. You you're not gonna get an NVIDIA GPU for the price of a DGX Spark with that much memory. So for example, I just downloaded, uh, NemoTron. That is it for raw. Okay, so for memory, the DGX Spark is worth it. For raw compute, it's probably not the best bang for your buck. If you need raw compute I would get like an RTX 4090 or something like that. But for memory, the DGX Spark is probably the best in the market at the moment. So, uh, for example, I can run NemoTron 3 Nano 30B, which is 40GB, um, natively on the DGX Spark. You can't even do that on a 5090. Wow. You get structured output and tried with Pexels food images and it is spot on. So PaliGemma, that's incredibly cool. Do you have a demo of the PaliGemma model? I'd love to try it out. I haven't tried for fine-tuning ParliGemma? Which version did you fine-tune? That's incredibly, uh, cool. So yeah, the TL;DR, um, to Justin, so if you want memory, if you want to be able to run large models, DGX Spark is worth it. If you want raw compute, you would probably better invest that into, uh, like a dedicated GPU, but you'll be cut off from the memory. There's no, like, I think, like a DGX Spark 2. I don't know when this will come out. For me, my perfect GPU would be like 128 gigs of GPU RAM but heaps of compute, which is kind of like, I mean, best of both worlds, right? Cake and eat it too. Probably cost double the price. But if I was to have a dream GPU sent to me by NVIDIA— so if anyone from NVIDIA is watching this, and in fact we have actually had an NVIDIA rep say they would send us this— is NVIDIA RTX 6000. So 96GB of VRAM, pro. That's— that would be like an epic GPU, but the price of that in Australia is $15,000. So we're going to keep streaming until we can get one of those, okay? And just to prove that NVIDIA have said this at some point, whoever was on the NVIDIA developer account, let me just find it. There we go, there's Green Code. So shout out to Green Code. And then we have Atik, NVIDIA developer, can you please send Daniel a Pro 6000 so we can watch him experiment with it? We know some people and we'll work on it. Deal. So don't know, we might get an RTX 6,000. But where were we? Okay, yeah, we're uploading this dataset to Hugging Face, and someone fine-tuned PaliGemma. That's incredible. Oh, PaliGemma 2 10B, did you use LoRA? What variance does Pali Gemma come in? 3B. I think it's time for a Pali Gemma 3. Use PaliGemma 3 with Gemma 3 plus Siglip2. Okay, we need to upload Full recipe. Beautiful. Okay, let's save this to JSONL and now import our image dataset as a Hugging Face dataset. From datasets import Yeah, it's basically a borderline consumer H100. That's my goal. I want to get one of them. So, uh, I was lucky enough for NVIDIA to send me a DGX Spark, but with all your guys' help, uh, and continued streams and videos and whatnot, I'm going to convince them to send me an RTX 6000. So where is my load data set? And in return, I promise to publish many open source models and tutorials. Let's see if this works, ladies and gentlemen. A quad 3 3090 AI rig, that is epic. What did we get wrong here? JSON parser, column description changed from string to number in row 2. Great. The point of Pāli Gemma 1 and 2 was that Gemma 1 and 2 had no vision, but now that seems unnecessary. That's true. Gemma, Gemma 3 has vision. You're right. So maybe PaLM Gemma 3 isn't needed, but I kind of saw PaLM Gemma models as like a model to be fine-tuned. But I guess you can kind of do that with Gemma 3 now. Okay, so we have a corrupted JSON. So let's go through this. Also, some don't have a description. Nan. Okay, if we have a nan in there, what do we need to do? Uh, ML training performance. I haven't actually had an issue with the Spark because I'm doing small data sets. Um, but yeah, I notice it's significantly— I haven't looked, I haven't compared it to any of the benchmarks, but, uh, I've noticed it's significantly slower than an RTX 4090. For the benefit of Spark is that it has an insane amount of memory, so it can hold large models, but the downside is the raw compute power is not quite there. Now why is this still nann? I'm going to replace it to none. Boom! How is this still— oh, because it's loading from that. Change from string. To number in row 6, another thing that we need. Lungs Detoxification 841, I still remember in 2022 I studied for the TensorFlow certification with Daniel Bourke and he taught me so well that I cleared the exam on my very first attempt. That is so awesome. Hey JJ, what's going on? So cool to hear you passed the TensorFlow exam. And Shadow Lilac, yeah, the TensorFlow cert course was so good, basically the only reason my manager trusted me at Google to take over an entire ML project and lead it myself. See, that's Certifications Help, and then Google went and shut it down. Maybe they'll bring out the Jack Cert next, you know. So we need to find— again, one of the problems with, uh, creating your own dataset is you're going to run into many issues. Random reviews not as list, or is it they're empty? Will it change from— Random reviews, what? So maybe it can't handle a list. Is it hard to switch to Jax? I personally haven't used JAX much. Uh, request for me: please explain all the code you wrote in the last 20 minutes at the end of this stream. Great idea. Have I tried Function Jammer fine-tuning? No, but that's definitely something we could try. I'm an inspiring AI/ML engineer. I'd really appreciate if you could help me improve. How can I help you improve? Best way to improve is to practice. That's what I'm doing right now. Okay, this is good. I'll be right back. Yeah, the problem with JAX is, you're right, Shadow Lilac is saying that there's about 8 frameworks for JAX, and yeah, they're almost entirely undocumented, whereas PyTorch just is like PyTorch, and then you have Torch Lightning. But I, whatchamacallit, have just stuck to raw PyTorch or Transformers. Thanks for your amazing PyTorch tutorial. Thank you so much, I really appreciate it. I'm glad you like it. There we go. Gemini Flash is fast. Is our dataset working now? Oh no, we're still getting errors. Ah, because I need to make the full recipe path. Excuse me. That's where I need that. I see Nan and I think Nan as well. Hey, we got a dataset! Beautiful. Do we have a PIL image? There we go. Okay, let's now— we're going to upload that to Hugging Face in a minute. But we're going to do 10 push-ups and 10 squats and then get our database dataset to Hugging Face. Hello, Asim. Welcome back. Okay, update the squat count. So What should we call the dataset? Recipe images 10K. Yeah, that'll do. We can upload the README as well. So now in tomorrow's stream, we can Because we could also— yeah, we kind of need to upload our script and stuff like that to Hugging Face as well. So we're adding a README to our dataset. Do we have the dataset up here? Yes, we do. Beautiful. That should load soon. Beautiful. Have I tried or experiment— experimented with Karpathy's NanoChat lately? No, I haven't. Have you? Is it good? Let's upload our dataset readme. So— You can train— pre-train a decoder transformer in 9 days on the DGX Spark with GPT-2-ish capabilities. That's really cool. Okay, so our dataset is created. I think that's a successful start. We've done that. So if someone wants a quick summary of what we did, was step 1, we downloaded some recipes from Kaggle. Um, at food.com— well, from food.com. We downloaded this dataset and then we sampled some recipes from that. Mainly we got 10,000 samples because our goal is multimodal RAG, so we wanted to create some synthetic images as well as text pairs. Now this is of course a play dataset, but it's meant to simulate an actual workflow, and I wanted to create it from scratch so that you could view examples of how to create your own datasets. So if you are working in a business or something like that, we've just gone from zero to dataset in what is it, 4 hours and not really even knowing what we wanted to create. We just created it from scratch. And this is something that you could definitely do in your workplace or your own projects. And we filtered 10K recipes with 1 to 20 ingredients with a minimum of 5 reviews. So we're using reviews as a proxy count. So then we had JSON recipes and we got Claude to help us write some visualization code. We could, of course, improve this with some extensions later on. I've got these later here, but this is for a practice, right? We're not exactly trying to make the most robust RAG system here. We're just seeing if multimodal RAG can work with the dataset that we've got. And this is a great way to practice, right? So we save the recipes to JSON and then we use Claude to— or we got Claude to generate us a script that just turns it into images. So all we do is we go JSON of recipe, turn it into a rendered PDF and image. So now in the next stream, what we're going to do is embed the images and text using NVIDIA's small yet mighty VL embedding models, and then we're going to build a RAG system based on vision embedding and on text embedding, and we're going to compare the pair. So if we go down, we then prepared the dataset to upload to Hugging Face. We had to make sure, because datasets are Apache Arrow, so we can't have any NaNs. So in Python, I always think of naan bread, shout out to, um, JT Videos, or ET Videos, sorry. In— we had to convert them. We had to make sure that they're either None, as in Python's None. It can't be NaN or an empty string. And so what we've got now is a JSONL file with just all of our recipe metadata and PIL images. So we've got our folder of images over here, and we've got— here we go, right? These recipes are really plain, but but I've done that on purpose. We could get really creative and make these look a lot better, but for now we just keep it as simple as possible, right? Keep it as simple as possible. We'll upgrade it maybe in a future stream if we need to. Um, we filter out this, we turn it into a Hugging Face dataset by just going through the documentation, creating an image dataset. There we go. Create an image dataset. And then finally, we've finished off by pushing that to the hub. And now we can load our dataset. If we go dataset train, let's get sample number 42. Why not? Because that's fun. And then let's look at the image, which is that. So artichoke hearts wrapped in bacon. So the ideal workflow here would be if we had— if we searched for artichoke hearts wrapped in bacon, or it would come back with this recipe. And then we could build a retrieval augmented— augmented generation system and it will return this. But we could just get this recipe back, we don't necessarily have to generate anything from that. So that will be the next stream is not only will we do text-based matching for embedding, we'll do visual-based matching. So if you want to tune in, I'll be here tomorrow live streaming, maybe a bit earlier than I did start today. But if anyone has any, uh, questions, please feel free to leave a comment on the video. Otherwise, thank you all for tuning in. I'm gonna go for a walk and have some lunch, but, uh, I'll see you in the next stream. The current training, pre-training for GPT-2 is 1.7 minutes. That is insane. I'll see you all tomorrow. Hey.