Local VLM fine-tuning on the NVIDIA DGX Spark - Part 1 - Creating a VLM dataset

Here we go, testing. Are we connected? I'm just going to check, make sure it works on my device. Let's go over here. You gotta be joking me. Nah, I like it. It's on another screen. Let's make sure the audio works. Okay, there we go. Beautiful. Okay, so if you're joining the chat, please say hello and let me know. But otherwise, we're going to start today's stream. So we have some goals for this week. So the end of the week, we want to make a tutorial on fine-tuning a VLM for food and drink image tagging slash structured data extraction. Um, now why would we create such a thing like this? Well, if we wanted to, we might want to get a— this is the app my brother and I work on— we might want to get a large food data set and extract some information from that so that we can make a model that can take a photo of food and then extract what different foods are in there. Now, right now, Nutrify, that app, depends on Gemini. So a big goal for the year is to rip out the Gemini dependence on Nutrify and create on-device models to, to do the right task or to do the same task. Now we're going to run all of this locally, so we're going to fine-tune on the NVIDIA DGX Spark, which is just in front of me here. And we've got some rules, which is 10 push-ups, 10 squats every hour, or when a model trains/data uploads. And of course, this is part of the Machine Learning and Muscle Cooking Show Streams. Now, today is Monday, the 12th of January, 2026 in Australia. And our goal for today is to make a dataset for VLM fine-tuning. So the type of dataset that we need is going to be image and text pairs, but we're going to see that throughout the stream. And then finally, we want to upload that dataset to Hugging Face. Of course, if you go to— here's the one we recently did on LLMs, but now we're going to do it with VLMs, which is of course vision language models. So we want to end up with a dataset like this by the end of today. Now the reason why we upload it to Hugging Face is so that we can one, reuse it ourselves and we can two, share it with other people if they want to use it, for example. So we've got food, not food. Oh, these are images as well. Sweet. So we could expand off this, but I've got some real life images that I've taken. So we're going to use that mostly as our labeled set. And then for the food images, we have images from Food 101, but we'll discuss that in the stream. So let's go, I'll just get out of this, um, let's go to here. So I'll make a new file and we'll go call this dataset and dataset goals. Get the, uh, image plus text pairs dataset uploaded to Hugging Face. So, um, dataset type is going to be images plus text, uh, perfect for VLM. And then we're also going to, um, let's go dataset size, 1K food images plus 1K not food images. Now why do we want this? Well, we want our model to be able to distinguish between what's food and what's not food. So maybe we don't actually need, uh, 1K not food images. Maybe we just do 500 or something like that. So then total size will be 1,500 image plus text pairs to fine-tune our small model. Now this— oh, I want to rename this. Rename. So this would be good for— note, this should be a good size for a, um, let's just say prototype or V1 prototype model. But for a better model we'd probably want 2 to 5x this size for a bit more robustness. Plenty of diversity equals bonus. So where are our food images going to come from? They are going to come from Food 101. Which is an open source food dataset, plus, um, let me just write this, main source, e.g., this is 101,000 images, total images, uh, but we're going to take 10x images per class equals 1,000 total. Bonus: upload your own food images for more real-world, um, usage. But this can come later. Um, not food images, main source random internet images. Also, uh, we want— we have manually— let me write this up here, sorry. VLM dataset goals. Goal: food extract VLM model. So this is what we need: manually taken images and then bonus could use synthetically generated images here to build out robustness. And we could also do that here. There's bonus 1 and bonus 2, um, could also use synthetic images here for robustness. Okay, wonderful. Let's check the stream chat. Yo, let me know if the chat's working, otherwise I'm going to get into just building the dataset. So like all good machine learning cooking shows, I've prepared prepared some things earlier. So if we need some data, we've got Food 101. So, and I've got Not Food 1K Samples. These are just random internet images and I have 1K Samples Manual. These are manual images that I've taken or found myself with iPhone camera or something like that, because we want our model to be able to distinguish food and not food, right? And we want it from real life images. Usually real life images are always best for now, but synthetic data is getting very good to the point where it can definitely bolster for cases that, um, real life images don't cover. Um, and then we have mixed, which is empty at the moment. So what I'm going to do is to begin with, I'm going to create a subset. So this All data train has 101 food classes, but there's 100,000 images here and that's going to be too large for our use case. So what I'm going to do is first and foremost, I'm going to filter out 1,000 or so images from the food dataset and I'm going to filter out about 500 or so not food images into the mix. They will be manual and internet-based images and then we're going to collate them into a single folder so we can have food and not food images. And then we'll also create a JSON that we can upload into Hugging Face into something like this. So this is our end goal, is to upload a food extract vision type dataset just like this. So let's— let me just show you as well. If we go to TRL docs, which is Transformer Reinforcement Learning, this is what we're using. We need, uh, dataset formats. The dataset format we're going to be using is, if we scroll down here, yeah, Vision datasets. So this is all we really need. That's the beauty of VLMs, right? And people are saying hello. Yo yo yo, glad to catch the stream. Good morning from India. Hey Alex, what's up? So I'm streaming a little bit later today because I had some things on this morning, but Australian time zone is not the greatest to line up with the rest of the world. But that's okay, we'll work with what we can. So a vision dataset, what do we need? So let's have a look. Some trainers also support fine-tuning vision language models using image-text pairs. Yeah, that's what we need. In this scenario, it's recommended to use a conversational format as each model handles image placeholders in text differently. A conversational vision dataset differs from standard conversational datasets in two key ways. The dataset must contain key images with the image data as its— as lists of PIL images or image with a single PIL image. The content field in messages must be a list of dictionaries where each dictionary specifies the type of data, image or text. So textual dataset content, what color is the sky, vision dataset. So this is what we need to do. Okay. And we want our image to go in and then our model is going to produce text, right? So rejected, chosen, prompt, images. Okay. I think we can do this. So let's go. First and foremost, we're going to create a viewer like this. So One, get 1.5K samples, 1,000 food plus 500 not food images into a single folder. To create labels in JSON conversational format. For the images, 3. Okay, sounds easy enough. Now I'll just show you what our food— some of our food images look like. So this is data from Food 101. If you haven't seen that, let's go Food 101 Hugging Face. Essentially what we see here is food images collected from the internet with a single class name. However, we want to expand this and maybe that's a project in the future. We could expand this to multi, multi-label, as in this is a beignet or beignet, however you say it, and icing sugar. So there's two actual foods here. That's what we want our VLM to be able to do. Like in this case, we've got a crepe or a beignet, icing sugar, and maybe this is cream. We want to expand it to be able to do multiple outputs. And so let me show you what this looks like. I've actually prepared this earlier with Quen v3 8b or VL, sorry, Quen 3 VL. So here we go. This is the original Food 101 class name, hummus. And what we're going to do is have an output. This is generated already by Quen 3 VL 8b. And so what we can do, we want our model to output something along the lines of dish, dish title. And then food items. So we're going to tag it with individual food items: hummus base, olive oil, paprika, chopped parsley, garlic, cilantro, relish, which is that there. Okay. And then there's no drinks in here. So if we do it again, we want— what do we get here? So wedge of blue cheese, white cheese, and soft cheese. Soft cheese, white cheese, blue cheese. Okay, so that's cheese plate. Now, why would we want to do this? Salad. So the overall name here is salad. And then we have lettuce, tomato, cucumber, black olive, tomato. Why did I say it like that? Feta cheese, grilled chicken. Is there grilled chicken? I can't really see that. Maybe you can, but that's okay. We don't want the most outlandish results to begin with. We want to just see if we can get this to work. Fried chicken with gravy. Caramelized onion, carrot, celery, potato, pearl onion. Okay, so this is our inputs and outputs. This is done with Quen 3 VL-AB with prompting. I want to do it with just a small vision language model so that we could eventually deploy it on device. Now, why would we want to— as I said, the initial goal is to— I have an app that me and my brother make called called Nutrify. And where did I put the video? Here we go. So this is our app, right? My brother and I— this is what I had for breakfast. And you take a photo of it and it uploads it to Gemini. Right. And it outputs that you get a title and you get structured data here with sardines and pecans, pickles, dates, etc. You can change what it actually is to correct it. Olive oil and salt. So our goal is to rip out the Gemini dependence and to put a local model on device and do this. So that's the long-term goal. But to begin with, we need to make sure we can actually do it with a small dataset. So that's our goal. Okay. Are you vibe coding some feet? Not sure what that is. How do you handle dataset versioning management? I've seen a lot of people using dataset version control. Curious how you go about it. So right now for Nutrify, our dataset is version managed on Weights and Biases as well as Google Storage buckets. So Weights and Biases just, just watch the buckets for changes. And if I was to do it going forward, I would probably use something like Hugging Face Datasets or something like that, which is what we're actually going to do in this project. We'll see how Hugging Face Datasets works. So I personally haven't really done the largest scale of projects of VLMs on Hugging Face datasets, but that's why we're doing this stream, right? So at the moment, yeah, our current setup is very simple. Google Storage, Weights and Biases watches for changes on Google Storage. I haven't used data version control before, or DVC. Okay, we get the point. Ice cream scoop, banana slice. Sauce, dark, fried element. All right, so we need to go get our path to the images and the food labels. So if we go print number of images is going to be Let's go all_images equals all_image_paths. So these are nested, so I think it's 3 levels. So it'd be data/train className image. Oh, by the way, we're running this locally on the DJX Spark. Is this going to work? Okay, it should work. Maybe I have to reset the Spark. Let's just try and get the labels first and see if there's actually something wrong. Why is this not working? String has no attribute read. Oh, excuse me, we don't actually need that, do we? What? Oh. Derp. JSON decode error. Hmm, extra data line 2 column 1 char 373. So we might need to figure out a way So let's see if we can see what this looks like. So because these are all generated, right, there might not be the right output. So Okay, that's not very helpful. Let's go to Claude. Some have the drink options as empty array and some don't at all. That's a great point. So let's go. Beautiful. Thank you. Why not train on current usage data but anonymized? Yeah, definitely. We could definitely do that in the future. Dan Hager says, sorry, I just got here. It's not clear to me the idea of adding the model to the device. In the device. So What do we do with the add the model to device? So run the model on device equals no data required to be sent to Gemini. Can just use the device's compute power. Inference is free. And potentially much faster. So that's a big, that's a big goal. Okay, so we've got our food labels here, we've got the data path, the image path, and then we've got the output YAML. So why did I do YAML? Because I just felt like trying out YAML for, for inference. I could have done JSON, but the problem with YAML is, as we found, is that some some items don't have all the right fields. But that's okay because we can define an output spec that for future we can use. So let's go and inspect some random samples. So food_df We'll create a DataFrame out of this. pd.read_csv— oh, sorry, um, DataFrame. There we go, beautiful. So we have YAML, we need to next Let's make a— while we've got food labels, make an We want some few, uh, a few new columns in food labels. Let's expand, uh, item name. Equals, um, item image path. We'll get the name and then item We don't want that. What is it? We want— not parent. Oh yeah, that's it. There we go. Beautiful. Okay, so what we're doing is we're enriching our dataset with a bit more information. So when someone goes to Hugging Face, they can try out our dataset. They know what's going on. So let's add these to item Item. There we go. Okay. And then I'll put the image source there. Why not? We're making a dataset, we want to make it as self-explanatory as possible. Okay, so next let's have a look at some of our YAML. And we want— if we go here and we go Food 101 class name. Oh, what have we done here? Oh, excuse me. We need to do that for everything. There we go. Okay, so we're going to get 10 of these each. So next is convert So let's first have a look at few just random samples as we go along. So there we go. Beautiful. So we want to minify this. So we've got drink items, pancake, pancake. Maybe we just turned it into Something like this. And now let's make this short. Does that make sense? I wonder which is going to be best for our model to output. This is— we're in the realm of like decoder. So if we compare this Alright, 34 tokens. That's not too bad versus 50 tokens. So maybe we just start with this and then later on if we needed to condense it, we could do the short version. Now I need to work out how to convert YAML to JSON. Stack Overflow. I haven't been on Stack Overflow in a while. Wow, I can just easily do that. Sweet. That's going to be a string. Nice. Okay, YAML to JSON, easy. Okay, so now we need to convert this, our sample JSON, into this. So, um, now remember, this is synthetically generated, okay? So we're going to convert it into our style. I'll keep the title simple. Pancakes. Maybe in the future we could— let's write something down here. Future: make image titles more elaborate for the whole dish. Um, for now we'll use short JSON, the full JSON for simplicity. But in the future we might use the short version. So I'm just going to keep it nice and simple by doing output is going to be dishes pancakes, uh, that's going to be the image title. The food items will be a set of all the unique items here, and the drink items will be orange juice. And then if there's no food item, which will be the not food stuff, we can use that later on, and they won't have any, any title or any food items or any drink items. Serge, how we going? Yeah, Alex, you're so right. Python to JSON, or YAML to JSON, so easy. Okay, let's go create helper function to convert to our desired structure. So def convert. So our input is going to be something like this. And how would we convert that to this. Might just put this there. Zoom out for a second so I can see. But I'm going to turn this into a set, then back into a list. That's already— that should already be a list. Uh, inputDrinkItems. And this input's going to be JSON. So, um, if some of them might not have the key, else, and then we'll go Simple is food 1, because we know all of these are food. Image title, uh, and then food items. What type is this? Oh, is this just a dictionary to begin with? Oh sweet, we can just put the sample YAML straight in there. Beautiful. Okay. That's all we need. So we can now create our structure because this is what we want our food model to output. We want it to output exactly that. And then for non-food items, we want it to output an empty version of this. So let me just go not food items output goal equals not food JSON example. Beautiful. So now that we've got our function here, let's convert our food labels. I'm just going to create a new list. A little bit of double handling here, but I don't mind that. I'm getting familiar with the dataset because once we make this dataset, right, like, we'll have it forever. So it's okay to be familiar with it. We can update it later on. I mean, the updates can be much quicker once we've— once we're familiarized with it. Image source, that's okay. And now convert the YAML. So the input is going to be the image, and then we go, um, what could we call this? We want food label out, maybe. We'll just call it output_label.json. Now let's see if we get any errors. Maybe we get TQDM. This should be fast, right? I like, I like watching progress bars. Input equals the YAML, and then we go Do you have a hack to get free GPU computes for training something like 50 param model? Yes, Google Colab. You can actually run the notebook I did the other day, uh, which is a 270 mil parameter model in Google Colab. Didn't actually need to do that, did I? Create a new variable. Actually, what am I doing here? I don't need all of these. So we're going to try and beat Quen's inference time, right? But for a similar result. That's what we want. We want to distill our big model of 8 billion parameters into a smaller model. Okay, how does this look? That's right, I need to convert this to Well, it works quite well. Unhashable type dictionary. Okay, so we get some items that don't work. That's okay. What I might do is only append the items that do work. That makes sense. And we're going to accept, cuz Quen 3 VL AB wasn't perfect, right? So we might put the whole thing in try-except. Now we could fix this with an LLM, right? We could just go So we get a fair few errors. That's okay. I'm going to— but we get many more of them. I'm not going to print out the error. So out of 101,000 samples, let's see how many we get with errors. And these could also be fixed in the future, of course, with another LLM or just some programmatic fixes. But it looks like we're getting far less errors than we do samples that are in a good, a good state. Yeah, look at that. We only have 350 errors out of the whole thing. So that's good. Now we want 1,000 samples from here. Beautiful. Okay, so we're starting to get— this is the ideal output we'd want for our model. Why is this coffee? Okay, we've got something wrong here. Just have to restart the notebook. Hmm, my SSH has been quite off lately. Maybe I won't print out so much so it doesn't have to So let me reconnect. Hello, Ratio Held, how are you? I'm not sure why. So I'm having not the greatest time with this SSH connection at the time, and I don't know whether that's the Spark or whether that's me and my internet connection. Okay, there we go. That's what we want. Croque Madame, food items. What is a— let's now compare our label JSONs with random images so that we make sure that it's good. Maybe I'll make one more field here. Train. Image path, because now I can use the full image path. View random samples. I want the class name. And then I want the image name. There we go, lovely. Okay, let's now put the side by side So we want the image title to be a string. Okay, we can do that in the future. For now, uh, now we do it like this. There we go. Okay, we need to do one more thing. This is good, we're making good progress. So once we've done the food image and text pairs, we can collate those with the not food image and text pairs and then upload it to Hugging Face as a dataset, and then we'll be ready to go. There we go. Tomato soup as a string. Wonderful. What do we get? Image title: edamame. And then the tags are wasabi powder and edamame pod, with drink items none. And what do we get here? Box of assorted donuts, glazed donut, long john donut, ring donut, braided donut. Beautiful. Then we get strawberry shortcake dessert, cake layer, mint leaf, strawberry whipped cream. Oh, this is a great one. Lobster roll, sweet potato fries, coleslaw. Okay, we need a space in between those. In the future, we'll format our image title to be better. But if we go down while that's running— hot dog bun, lobster meat, sweet potato fry, coleslaw, pickle slice, drink items, cola. So that's pretty good. Hot dog bun, lobster meat, sweet potato fry, coleslaw, pickle slice, drink items, cola. Now again, that's not our model yet, but this is— we're getting the image and text pairs of what we want. I always like to view— we're not going to do 100 this time, but when I'm working with a new dataset or creating a new dataset, I always like to view 100 different items. So grilled steak. And what do we got? Herb garnish, mushroom, steak slice, onion. Wonderful. Now this is seafood paella, paella, green peas, chorizo. Yep, you can see that. Rice, mussels, see those. Red bell pepper, lemon wedge, shrimp. Drink items, none. Stone pot rice, seafood rice. Layer dessert. Okay, that's quite a long one. Holy smokes. Okay, so that's given. So this is what— where we can make— this is a beautiful thing about making our dataset better is we can put in the things that we want. Chicken with gravy, brown rice, corn, brown rice, grain, gravy, chicken breast, corn kernel, mushroom. Mushroom there, beautiful. No drinks visible. Oh, this is a nice one. Cheese platter, food items, goat cheese wedge. Blackberries, red grapes, strawberry. Yep. Hard cheese slice, blue cheese wedge, orange dried apricots, honey. Honey might be in the middle there. How many items do we get? 1, 2, 3. We get any drinks? No. So 1, 2, 3, 4, 5, 6, 7, 8. So we're missing one there. Goat wedge cheese. I'm going to say that that's that. Blackberries. Yep. Red grapes. Yep. Strawberries. Yep. Hard cheese slice. Oh, have I got that twice? Maybe there. Blue cheese wedge. Orange dried apricots. What did we miss? Honey. We missed one of the cheeses. Anyway, make a playlist for LLMs. You mean on my, my channel? Is that what you mean? I can definitely do that. So, um, let's now— we've got our updated DataFrame. Let's save this to file. What do you mean when you say, um, an LLM series? Do you mean, uh, like what we're doing now, training models, or do you mean building LLMs? Uh, depends what you want to see. So not that I— we can do either or, but at the moment I'm focused on creating fine-tuning series of LLMs so that we can create our own models to use. But let me know in the chat of anything in particular you'd like to see. So now we've got our food labels updated, right? We can import those if we need. And now we want to I'm going to fine-tune VLMs. Uh, Adam asked, have I fine-tuned, um, VLMs? Not yet. I'm going to, but by the end of this week we will have. So that's our goal for this week in the stream, is to— today we're making a dataset for VLM fine-tuning. And we're going to publish that dataset to Hugging Face. I'm off for Tuesday having party day with my wife, and then from Wednesday and Thursday we're going to use the dataset that we've created today and we're going to fine-tune a small VLM. And then on Thursday we're going to create a demo from that fine-tuned small VLM and we're going to have that all available publicly. And then on Friday I'll turn that into an end-to-end tutorial just like we've worked with before. Just like last week's video. Very fundamental thing about LLM, like what is LLM, how it really works, the technical thing from the very start. Okay, so you want to see— you want to build an LLM architecture and you want to train one from scratch. I'm gonna make a playlist. Yeah, so the series I've got planned out at the moment Is series, um, LLM fine-tuning done? VLM fine-tuning doing, um, synthetic data creation, e.g., with GPT-OSS. 120B. And then later, so this could potentially be ideas building LLMs plus tokenizers from scratch, um What is an LLM? Uh, build apps with LLMs, etc. So that's, that's the series ideas in a nutshell. But yeah, any— always leave a comment on one of the videos or a comment in the stream if there's anything in particular you'd like to see. Otherwise, um, oh, ideas is deploying LLMs to small apps, e.g., on-device, such as MLX. So that's the current big goal. This ties in with the current big goal, um, replace Gemini on Nutrify with a local VLM. But we're a little bit off that. We need to fine-tune a small VLM first, so create a small sample of the data. I need this df, food_updated_df. Okay, so let's get I'm gonna write a tip here. So does that make sense? You want diversity in your training data. Instead of getting 1,000 images from one class, we are getting, uh, 10 images from 101 classes. Oh, thank you for buying the PyTorch course. And you— Alex is saying, yeah, that series you typed out would be good. Okay, that sounds good. Well, we can work on that. We can work on that in the future. Whether it be a stream or a video. So that's the kind of the way I want to take the channel in 2026 is stream and then create dedicated videos, if that makes sense. And so I'm streaming on YouTube now. I don't know whether it's better to stream on Twitch or something like that, but I think I'm going to just stick to YouTube because I already have a channel on YouTube, if that makes sense. So Let's go. Um, wonderful. I'm gonna randomly loop through this, actually. So we get some random. Beautiful. That's what we want. And what's our— we should have 10 for all of those. Lovely. That's what we want. So we're really getting towards the tail end here of creating a sampled dataset from Food 101. Awesome. So we get a good diversity of different samples here. And then if we have a look at the class names, we should get 10 of each. That's what we want. Okay, now we want next, um, match the images to the Food 101, uh, 1K samples to combine the not food, um, with the appropriate output label. Uh, then we want upload all samples. To Hugging Face. Okay, that's what we're going to work on next. I'm just going to go to the bathroom and I'll be back in— get a drink of water. I'll be back in a few minutes. Go keynote. And we're back. So we've got our subset of samples, nice tidy 1,000 samples. Next, we need to get or match the images, and then we need to combine the not food. Or actually, we might as well just do it in one step, shouldn't we? Uh, I might add down here prefix for the model to add to training. Something like that, but we won't do that just yet. So We've got our small sample of food items from Food101. Let's now create our not food subset. We do this so our model learns to distinguish between food and not food items. For example, a car or a cleaning product, we don't want our model identifying it as food. Nice and simple. So— We need this structure. So we've got some manual images here. I'm going to copy that path and then some random images from the internet. And then path to not food images mixed. So this is 1K manually taken images. Good morning, EK Shunya, how are you, my friend? This is, um, 1K manually taken not food images, and this is 1K internet not food images, and this is going to be a mix of both of the above, 250 random samples from each for 500 total images. Okay, so then our overall data sets are going to be 1,000 food images and 500 not food images. First time catching me live. Where are you? Where are you tuning in from? EK Shunya. I'm trying to work out the best time to stream. So it's 1:20 PM where I live in Brisbane. But for a lot of people I know, Australia is not the best time zone. So I'm trying to play around with different streaming times. Himalayas! Oh my gosh, that is cool. Nice to see you. What time is it there? Now, are these all JPEG? Beautiful. Now what we're doing is we're curating a dataset here, obviously, right? We want it to do food, not food. But the whole premise of LLMs is generally to do everything in one hit. Does that make sense? So you just, you don't curate it. Yeah, well, you do, you want quality samples, but you train on as much data as possible. But because our models, the underlying model, we'll get to this in the next stream, has obviously been trained on a lot, uh, maybe not obviously, has been trained on a lot of data to scene with. What we're doing with this is we're basically guiding that model, which has seen most of the, most of the stuff on the internet probably, to our particular task, which is food and not food, right? So paths, we want, this is going to be internet, not food. And this is going to be mixed. Beautiful. Oh, actually, I think this might need to be that. There we go. So we've got about 1,000 or so of each. Now what I want to do is combine 250 random samples from here, 250 random samples from here into here. Now, why randomly sample from manual plus internet? No particular reason except for the fact we're trying to keep our dataset quite small, e.g. 1.5K images. For the future, or for production-level models We could definitely increase this perhaps 5 to 10x in size or more. The main goal of scaling would be to cover more and more edge cases in the training data. 9 AM. We should also have one paper reading session followed by implementation, experimentation. That's a great idea. Are there any papers that you'd like to see. So let's— yeah, this is really live. Smoothie the Cat asks if this is really live. Yeah, ask me to do whatever you like, within reason obviously, or type something. And I'll prove to you that it's live. Ready? Hi, Smoothie the cat. This is live from Brisbane, Australia. MHC. What is MHC? Austin, Texas. Hello, or howdy, should I say. 9:25 PM. Yeah, I forgot it's a Sunday in— we're in Australia, it's Monday morning. We are— or Monday afternoon, sorry. We're ahead of you guys. By, uh, about 17 or so hours. It always clashes whenever I have to do a meeting with, uh, the US. Um, my, my time zone doesn't line up very well. Okay, we got some paths and we're going to sample them. This should be 500. If it's not, we've got an error. 500. Okay, so now we need to Have a series where we start from the legend papers and then move to the latest in 10 weeks. That's a cool series. So you mean like start from like the Transformer paper and then go from there? Is that what you mean? So we now have food images mixed. Now we need to turn them into JSON. We've only got paths. Okay, these are our And what structure do we need it in? We need it in this structure. So this is where our output label— we could make this really condensed. We just put that— we could put that just as 0, I think, but we'll just make it consistent. That's what we're experimenting with, right? What is the best input and output for our model? Beautiful. So given this image, we want to output is food is 0. Adama Karts is asking, can we use TRL for VLM fine tuning? And yes, we can. That's exactly what we're going to use. So where's TRL? This is actually what I'm doing right now, is creating a dataset for TRL for vision. Do we have a guide here? Here we go. Vision models. Beautiful. This is cool. So there we go. Dataset. This is what we're working towards. We'll, we'll get this done by the end of the week, right? It's using PaliGemma2. Wow. So Sergio contributed this, so thank you, Sergio. We got this. This is also another one doing Quen via Quen 2 7B VL, so we definitely can, but we need to create our dataset. EK Shunya, have you used PaliGemma before? I wonder what PaliGemma is like compared to Gemma 3. So for structured data extraction, that is. So let's merge this with Food 1K samples. This should be about 1,500. Oh no, that only went up to Beautiful. So now we have 1,000 samples. Of food and 1,000— or 500 samples of not food. Okay, so now I believe we need to What I'm going to do, actually, I'm going to turn this into a DataFrame and then turn it back. Yeah, so now we get Nanz, and that's okay. Beautiful. There we go. Nan. So the ones with that aren't food are going to get Maybe I just add a simple column as well. If item Okay, that's what we want. So now we need a way to put this on the Hugging Face. So I believe I need to add the image key. Let's find Hugging Face datasets. We'll get to the docs. See you later, EK Shunya. Thanks for tuning in. JJ, hello, how are you? So I need to So let's create a vision dataset. Hey, so I want some labels and images. Maybe I need my images in a single folder. That would make sense. Filename. Step 1. 2. 5, we're going to So we're going to move all of our target images into Oh, copy. I want to Hi, how would I start learning ML and AI today if I was starting with no CS background? Well, I don't have a CS background. So I would just go through all of the freely available online courses, probably go to huggingface.com/learn or something like that. Watch all of my videos, or Andrew Ng's deep learning and then start to work on your own projects. So of course it's going to take a while, but that's the whole process of learning. What you'll see is a lot of people are using sort of AI generators and whatnot, whatnot, which are very helpful, don't get me wrong, but they are probably that good at using them because they have spent the time learning the underlying fundamental skills. And so I think there's still no replacement for just going through the effort of learning something new. And it just takes practice. And if you want to get better at coding, you just have to keep coding. And if you want to get better at math, you have to keep practicing math. And if you want to get better at like what I'm doing, practicing livestreaming, creating datasets, you have to create more custom datasets, you have to train more models. So look at this, we've been going for nearly 2.5 hours now and— oh, actually we've missed out on some push-ups and squats. So I'm going to do those in a second. But we're going to— yeah, that's the way I would learn is I would go through, pick one resource, maybe buy a textbook. AI Engineering by Chip Hewitt, great book. Where is it? AI Engineering. This book. So great book. Maybe read through that and then set yourself, okay, I want to— and also this Designing Machine Learning Systems. So read those two books over a year and set yourself a good 6-month project of like, okay, here's an app that I like. I want to try and reproduce that myself. That's what I would start working on. And I would take it slow, right? Learning something new is hard, so it takes practice. So we need to do some push-ups and we need to do some squats because that was our rules. We've broken it, we got too into the vibe. So 10 push-ups, 10 squats. So because I haven't done them for 2 hours, I'm going to do 20 push-ups and 20 squats. E.T. asks, g'day, g'day, how are we? E.T. asks, did I pick the 1,500 image count for no specific— for a specific reason? No, I didn't. Um, I just guesstimated, right? We'll, we'll need that. Where do we go up here? This is what we did. Um, I wanted to keep the dataset quite small. So why do 1,000 food? Because I think food is a bit more diverse than just general random things, if that makes sense. So I want the model to have a kind of bias to being a little bit better at food. Whereas the not food is basically just predict zero, as in it's not food. Like if I take a photo of the Mac Mini, I don't need it to understand all of the details and intricacies about a Mac Mini. I just need it to know that's not food. Whereas if we get a plate of food, like whatever you had for lunch, for example, there could be 5 or 6 elements on there. So I'd like the model to give a little bit more weight to that, trying to predict each of those elements. That's my hunch. I don't know whether it will work. Um, we'll find out, I guess, in the net once we've created the dataset, then we go to try and fine-tune a model. But in practice, this is for like sort of an experimental model. In practice, you would probably want, uh, yeah, a 5 to 10x times larger dataset, and then potentially even more depending on how much coverage you want. So that's why I did it, and you can read a bit more here. But I need to do some push-ups and squats. 20 push-ups. We need, uh, squats time. So my legs are a bit sore because I went running this morning with my friend. Beautiful. So there we go. Let's get, um, push out count. So what do we have here? Greg says, I'm working on a project, a cigar scanner. That's cool. Oh, it's gone. Um, cigar scanner, that's definitely something that very aligned to what we're doing with fine-tuning a VLM. So instead of food, you could train it to extract details from cigars. I love cigars. You are currently Android and iOS, but the market has been rough. Yeah, my brother's experiencing that at the moment a little bit. iOS, trying to get a job somewhere, but not— they always go, oh, you need more experience, you know what I mean? But it's like, well, I've built multiple apps. What do you— what else do you want me to do? Okay, we need to move our images. Image name. So we've got— this is our food images and this is our Internet, and this is our manual. So we need to pile these all into here. So if it's in Food 101, we'll do it that Now, is it train or test? Do I have that? Yes, I do. We need That looks good to me. That looks good to me as well. Now we just need one more for the internet. This is what a lot of machine learning engineering is, by the way. It's from collating data from the right sources and making sure they're quality, and then training a model on that, evaluating the model, seeing if it performs to your liking, and then, uh, improving the data set over time. And then of course you can tweak the model and whatnot, but for me I personally use mostly pre-built models and fine-tune them. But if you're a researcher or whatnot, you are always looking for new architectures. I take things that are already proven to work on similar problems and customize them for target problems. So right now, of course, we're spending a lot of time crafting this dataset. But then once we've proven that this experiment can work, we can start to really scale up the, the data loop, right? It's called a data engine, Tesla's data engine. So that's what we're going to work on. But for food extraction. For now, we're keeping it small, just 1,000 samples. In the future, we might move to 10,000 samples, and then after that we might go to 100,000 samples. But we need to verify that it works for the smaller dataset first. So now we need path to notfoodinternet and item image name, new path equals path. So I want— oh no, sorry. And then path to— beautiful! That's exactly what we want, right? So now what we can also do is go item.filename, because we need the filename key, equals item.image_name, right? We've got a, we've got a redundant key, but we want to now copy to source equals original dest equals new Path. All right, and we need, um, import— this should work. Copy them across. And now let's check. Here we go, work setting up to cigar detection. What's— if you tell me what, what the exact problem is with your cigar scanner, I can help recommend a bit more, uh, information. If you just need to detect where a cigar is in an image, yeah, you would use object detection. But if you need to lift information from that cigar, such as the label or something like that, you probably need either a VLM or something to do with text extraction or something like that. So let's now— how many images do we have in here? Wonderful. Okay, Food Extract 1K Vision. We have images in a folder. Revolutionary. I know. And it should be about the same amount as labels that we have. Wonderful. Okay, so now let's have a look at our labels. So have we done our steps? Yes, we have. And we've also added this. Filename for each row. And now we need to turn the file plus metadata into a Hugging Face dataset. Now I believe we're going to need this inside here, so we can do that as a JSON-L, I believe. Have I tried adapters for your use case? Can you be a bit more, um, specific? Adapters as in LoRa adapters? If it's LoRa, that's what we're going to be trying to do. So metadata CSV Jason L. Okay. I'm going to do JSON-L, or maybe, or Parquet. I can do Parquet. Miguela, how are you, my friend? We're currently creating a dataset for our VLM. So let's ask Cloud. How do we save a JSON as a JSON-L? So right now our list of dictionaries is— looks like this, right? We want to save this to a JSONL. Let's do that. So with open, but I want to save it in food_extract_1k. Path to food extract. So path to save JSON-L labels equals— and this is going to be metadata.json-l. Uh, my JJ asks, has my DJX Spark overheated? No, I haven't had it any, um, any overheated yet. We'll soon find out. I've been training. I trained an LLM the other day. I fine-tuned a small LLM. But, um, Maguela asks, just what I need, thanks for doing this. Are we allowed to ask some things about ML? Of course. Feel free to ask away. I'll do my best to answer questions as, as I'm coding along. So with open that as write as f, and now we'll go here. And then Cloud is saying for item in data, just going to do this. Don't copy from AI. You can rewrite it for item in dicks. We're going to go f.write). Plus, now this is obviously very hacky code that we've got, but we're doing it on purpose because we're experimenting, trying to figure out if it even works, and then we can make it better. You kind of have to spend some time doing that. Do we have our JSON-L file here? Scroll down all the way to the bottom of the images. metadata.json-l. Yes, lovely! Do we get the file name? Yes, we do. Okay. Ladies and gentlemen, I think we may be able to turn this into a Hugging Face dataset. So this will be a cool moment if we can turn this into a Hugging Face dataset. What are we at? 2.5 hours to make a dataset. That's not too bad, you know. And of course we can scale this up quite quickly. So turn our dataset into a Hugging Face dataset. So let me just write up here what we've done. I need to go 3, actually, is save the labels plus metadata to a file in the same directory as the images. For import the data as a Hugging Face dataset and make sure it works. I think that's it. So now we've done this, might move these over to here. Let's see if we can load our dataset. We've now got a folder of images plus metadata. So I think it'll be load or image folder. I think it might be as simple as this. Oh no, what did we get wrong? Change from string to number in row 140. Ah, I see. So for the ones that are not food, we may need to give them a default type. Yeah, here, nan. So Does that make sense? So we have some samples which are strings and some which are NaN. Yeah, there we go. So in this case, we have NaN, NaN, NaN, NaN. Let's just see what Cloud thinks. My go is maybe we just do it as an empty string. I think we should use My bet is empty string. What's your bet? Let's see what Opus thinks. None. Okay, let's change it to none. Beautiful. Oh, we still get some nans, don't we? Nan. That's one option. And now we'll go— see, as I said, a lot of machine learning engineering is getting your data in the right format. Beautiful. Aye, look at that. We now officially have a Hugging Face dataset. That is cool. That worked. Thank you, Cloud. Okay, empty string was wrong. You should have to convert it into None. So I'll write a little note up here. Note: converting the NaN samples fields to None so they can be handled by Arrow in hugging face datasets. Beautiful. And then note, make sure we have no, um, in the dataset, otherwise these will error when using hugging face datasets. Oh, actually, now let's look at our dataset train 0. What do we get? Hey, we get it! We get a pill image here, right? And then we get some information, so we can now have a look at the image. Lovely. Okay, and then let's go 0, and we want the output in this case to be output info target vlm_output is going to be this. Uh, I need a single quote there, don't I? And then let's show the image. Wonderful. So in this case, the target VLM output would be, is food. Yes, it is food. Cheese plate. And then we've got toast, herb garnish, dipping sauce, cheese wedge. Lovely. So how about we now upload this to Hugging Face? So that's going to be— that is step 4. Now I need to push this to the Hugging Face Hub. We've done this, so let's see how we can do that. Push to Hub. I think it may be just that easy. We put in the repo name. I think I can delete the version from here so that this link will just work whatever the version it is. Yes, we can. Okay, so let's get our dataset. I'm going to call this foodextract1kvision, and then commit message equals initial commit done live on stream. All right, who's ready to push our dataset to the hub? In 3, 2, 1, let's do it! So it's going to wrap up everything and it's going to upload some files. New data upload. There we go. So our dataset's 280 megabytes, which is not that big in terms of machine learning datasets these days. But if all works, we should have a shareable link that not only can I check it out, but you'll also be able to check it out as well. So I think it'll upload by default. This will upload to your username slash repo ID, e.g. There we go. Commit. And let's have a look at the dataset card. Should be available soon. 1.5K rows. How cool is that? Now of course we could have a README and whatnot, but then we get data train Parquet. So I'm gonna like that. Oh, I'm not logged in, am I? Oh wait, am I? There we go, a image. So let's see, can you guys see this in the chat? That is so cool. Edit, edit dataset card, and I'm going to put the README Apache license image text-to-text. There we go. I'm gonna run it. It worked! Beautiful dataset to practice fine-tuning VLM models. Goal: extract food, drink items from an image in a structured format. Input: output JSON structure. Now let's just add a little bit of info to the README. Where was I? I was in Safari, wasn't I? Oh my gosh, I'm in formatting hell here, people. Now let's get another actual example. Now of course, these are our just ground truths, right? We don't have Okay, I'm pretty happy with that. Let's put a link to Food 101 on Hugging Face and then Quen 3 VL. 8B, my favorite model at the moment. Let's update the README. README. Hey, Matthew, how are you? There we go. So that's what we want. We now officially have a dataset we can reuse. So if we go to Colab. Let's see how quick we can download this dataset. So, um, food extract 1K vision. Playground. And then we're going to go here. And then let's just run this. ds train 0. Obviously, we've got to wait for Colab to boot up. Granted access to my Hugging Face token. But this is going to download 280 megabytes. That was incredibly quick. There we go. We can reuse this dataset over and over again now. Image. Beautiful. Okay, so I think that is a good place to wrap up the stream. Have we done what we wanted to do? So we've done that. Yes, we have. And then next, have we done this? Upload the dataset to Hugging Face. And yes, we have, because we have the link. If we go to my profile, Food Vision Extract. We can put that there. Okay, so tomorrow I'm not going to be here because I'm going to do Party Day with my wife. Twice a year we have one day where we plan for each other and we go out and just have fun together, essentially, which is not a birthday, not an event of some sort. It's just— it's called Party Day. So I have no idea what she's planning, but I'm excited. On Wednesday, we're going to use the dataset above to fine-tune a small VLM, so Gemma 3 or something, to reproduce our task. So, um, see example notebook here. So let's go to Gemma docs Google. This is the notebook I'm going to be building off. So we want Vision Tune using Hugging Face Transformers in QLoRA. So if you want to go and try this out for yourself, this is what I'm going to try, but we're going to try it on our, our own dataset, right? I think this is using Amazon products. Yeah, there we go. Amazon product descriptions. I'm going to try that with, um, our food extract task. And then afterward, we want to see if we can, if we can fine-tune it, get that uploaded. Upload the fine-tuned VLM model to Hugging Face, and then create a demo for the fine-tuned small VLM. Upload the fine-tuned VLM to Hugging Face. And then I want to have an off-stream tutorial for doing this step by step. Okay, now it is getting to the end of the hour. I think it's time for us to do 10 push-ups and 10 squats, and then we'll finish it off. Hey, finish off the stream. But that, that was the goal for today, was to get a dataset that we can fine-tune our VLM. So our VLM will want to— we want to input this image, and this is what we want it to generate. Okay, that's our goal. So we've already done this with QuenVL-8B, but we want to do it with a smaller model. So Gemma 4B or something like that. Okay. Greg says, I'm hooked and will continue learning. Been following you since you made the Nutrify app, and I thought you disappeared for a moment. Well, I did. I disappeared trying to build the Nutrify app, and we did. But what we want to do for Nutrify is right now we rely on Gemini. Let me just show you the demo video. This was me taking a photo of my breakfast with Nutrify, right? So food detect, that's food, not food image classification. We take a photo of the food. This goes to Gemini, right? So right now this loading screen is going to Gemini. But what I want to do is replace the Gemini requirement with an on-device model. So then we get some outputs here, and my brother has made this look beautiful, right? Sardines, pecans, pickles, dates, right? Olive oil. But if we go back to the image, it got that exactly right, right? Sardines, there's some salt there, pecans, pickles, dates. That was my breakfast. And so Of course, this loading screen is going to Gemini, but I would prefer to run this model on device. I think that would be really cool if we could do that. So that's what we're trying to work towards right now. We're starting small with just 1,000 samples, but we can start to scale it up if we find that it works right. And then it's one thing to get it running on a Hugging Face demo, but it's another thing to deploy it to an app. The reason why we use Gemini is because it's— you kind of call it with one line of code, right? You pass a prompt, you pass the image, Gemini sends back JSON. But if we can get to the stage where we just fine-tune our own model, we can deploy it. All of a sudden we get inference for free. So that's really cool, right? Whereas we have to pay, we have to pay to use Gemini to get this sort of capability. So that's, that's what we're working towards with Food Extract 1K. We start with a small dataset. If it works, we scale it up from there. So with that being said, thank you all for tuning in. If you want to see the next stream, it'll be— so it's Monday in Australia time. It'll be around in 2 days' time. So I'm not going to be here tomorrow, but I'll be here on the day after that. So where is our chat? All right. Oh yeah, we got to finish off. We got, we got to do 10 push-ups and 10 squats to finish the stream. Everyone join in wherever you are. This is Machine Learning and Muscle. If we're fine-tuning our models, we got to be fine-tuning our bodies, okay? My brother just came in, probably just like, what the hell are you doing? Um, but thank you all for tuning in. I will see you in the next stream. We are going to use our dataset to fine-tune a VLM to see if we can replicate this workflow. So that's our goal for the next one— create a food extract fine-tuned VLM. And then the stream after that, we'll deploy that to Hugging Face. Thanks for tuning in, and I'll see you next time.