Local VLM fine-tuning on the NVIDIA DGX Spark - Part 2 - Fine-tuning a model with LoRA

Let's go live! Oh, we're connecting. Ladies and gentlemen, welcome to the stream. Probably have to say this again because I just want to make sure. Tuning in. Let me just see if this works on another device. If you're in the chat, please say hello. That way we can all get started. Where's my phone? Today we want to fine-tune a VLM. Yo, okay, sounds like it's working. I'm gonna pop out the chat. So let's go into— I'll just close Messages. Let's go to stream.md, and here's what we're going to do. We are going to— how do I fix this? So on Monday we created a dataset, food extract vision. The goal of this is to fine-tune a VLM, right? So here's where we're up to. It is Wednesday the 14th of January, 2026, and we want to fine-tune a small VLM Gemma 3 could be 4B, I believe, or small VLM to reproduce the task of what we're doing. So we want to extract structured data from food images. So if we go to our dataset, this is what we created in the last stream. Most important step of basically any machine learning project, create a dataset because now you have something to work with. And so we have input images. Is this going to show me? Maybe I have to refresh this. So we have input images. Let's have a look. There we go. Beautiful. And we have a structured data output here. So is food? Yes, it is. Caprese salad. We want to predict food items. Onion ring, tomato slice, basil leaf, mozzarella cheese. That's what we want. Our model to be able to reproduce. But we used Qwen3 VL8B, 8 billion parameters, to do this. And so we want to be able to do it with a smaller model that is purely fine-tuned for, um, focused on food images. But we also have, uh, 1,000 food images but also 500 not food images. So where we use this is if you are building an app like Nutrify and you wanted to fill, uh, filter a large-scale dataset, you could use this model to, uh, inference over that large-scale dataset. But let me show you a few examples. I'll go to this— Gemini just had an update to the file input methods. Maybe we'll have a look at that in a second. But let's give an example of where this might be used in a practical sense. So I'm just going to jump I want to get out of some things here. Here's my chat. Yo, Alex, thank you for joining in. Ashik, top of the morning, my friend. It is 10:20 AM, a little bit later today than I wanted to start, but I had to do some things because I had yesterday off. But that's all right. So where would a VLM be used? So this is an example. This is just released. Today or yesterday. Well, today for me because Australia is a day ahead. Medjema 1.5. So Medjema is a VLM. It can also do audio. So VLM is vision language model, but Medjema can now be used as a developer tool. So what we can do is— do they have a demo of Medjema here? Medjema collection. So Medjema 1.5, this just got released for B parameters. That's pretty cool. So it can take in pathology images. Let's have a look at that pathology images example. So something like this, right? We can pass that into Medjema and then it can— let's view these side by side, actually extrapolate data from that. Dermatology, Medical Text and Records, and now new is speech via Med-ASR. So if we go into Med-ASR, Google just released this. I'm going to like that model. So is there a demo here? Doc-ASR, Med-ASR. Let's see how good this model is. So we're going to record. Oh, do I need to auth this? Ah, okay, we can just imagine that it's a pretty good model for recognizing text, right? You can also fine-tune it in Google Colab. So here we go. Wow. So it's better than Whisper v3 Large and even better than Gemini 2.5 Pro. That's incredible. So, uh, Medjema, this is an example of where a VLM could be used. Now, Yes, you can use Gemini or ChatGPT for something like this, but of course in the health space world you want to preserve privacy, right? People, people don't want to upload their images to ChatGPT. Maybe some people do, some people don't mind, but a lot of people don't want their data just uploaded to some random database. So the beauty of having a model like MedGemma 1.54B is that it can run on a small device. I mean, you could get MedJAMmer to run on an iPhone. Maybe that's a topic for a future stream. But so we could put an input in here of a prompt. So you are analyzing a continuous, continuous block of CT slices from the abdomen. The patient is being evaluated for potential liver tumors based on the visual evidence in the slices provided above. Is there evidence of a tumor in the liver? You must first explain your reasoning, uh, specifically noting any specific, specific suspicious features. Excuse me. That goes into MedGEMMA 1.5. AI response: based on the provided CT images, there is evidence of a large irregular mass in the right lobe of the liver. This mass exhibits significant heterogeneous enhancement, which areas— with areas of both hyperdensity and hypodensity. The mass appears to be causing some distortion of the liver parenchyma and may be associated with some degree of intrahepatic biliary dilation. See, I can't pronounce these words. Maybe Med Gemma ASR would be able to recognize those. And so the AI response here— I haven't read this, by the way, this has only just come out today. I'm going through this live. Um, this is verified by a board-certified thoracic radiologist. I agree the pneumonia has worsened on the second image compared to the first image. I agree with the reasoning and description of the lung opacities. However, A region of relative luency in the mid-right lung, which could represent either cavitation or merely a region of lung spared by pneumonia, is not mentioned. Fine-tuning may improve results. So this shows how MedJamba 1.54b can be used to interpret a longitudinal— so longitudinal pair of x-rays, like over time— of chest x-rays, along with commentary by a board-certified thoracic radiologist on the quality of the output. Note that MedGemma is not intended, intended to be used without appropriate validation, adaptation, and/or making meaningful modifications by developer for their specific use case. So we've had an improvement on text only. And we've got MedASR, so an open model for medical automated speech recognition. So much better here. On medical-specific speech. And so where can we use this? Well, MedJEMMA 1.5, medical images and/or text, output is text. So it can interpret medical images and it can do compute-efficient text or modal, modal, modal, multimodal medical reasoning. So this is where something like a Kaggle competition comes in, where the beauty of a Kaggle competition is you get people all over the world that can use MedJEMMA and these open-source models for for creating a specific use case for a workflow, right? And so this is cool. You can actually win real-life money. And I think this would be a great example of like working on machine learning project with an open-source model that could eventually be turned into a company. Great timeline. Maybe that's a topic for a future stream is we work on this. This is a competition I would think about entering, you know. And then Google obviously have, um, pre-trained OpenWeights. But okay, we're getting ahead of ourselves. MedGemma is an example of a VLM for medical-specific tasks. We want to create our VLM called Food Extract, uh, for food-specific tasks. Does that make sense? So that is our goal for today. If we go to the stream goals We want to use the dataset that we created here to fine-tune our own version of Gemma. Apparent— uh, we could fine-tune Gemma, we could fine-tune another open-source VLM to reproduce the task of food extraction, and then we can upload the fine-tuned VLM to Hugging Face. So if we go to Google, right, this is what we want, but for our specific use case. They've obviously uploaded MedGemma somewhere. Here we go, MedGemma 1.5. Give that a like, and I want to acknowledge the license so I can use that later on. Um, they have created MedGemma, which is a customized version of Gemma for medical. Now we want to create FoodGemma, which is a customized version of Gemma for food. So let's see how we might do that. And we've got a guide here on Google's Docs. This is what we're going to be following. So we want to fine-tune Gemma for vision tasks using Hugging Face Transformers and QLoRA. I haven't actually fine-tuned a VLM before, so I'm excited to see how this goes. With, with everything going right, by the end of this stream we'll have a fine-tuned Gemma model for, uh, food tag extraction. Who's ready? And yeah, this is an update to the Gemini API. This is the reading stuff, but we'll, we'll read— well, maybe we do some Gemini exploration in a future stream, but for now we want to focus on local models. Oh, Google have a Health AI Developer Foundations collection. MedGemma, MedSigLib. That's cool. MedGemma 1.5, and there's MedASR. Okay, so fine-tune Gemma for vision. Let's do it. Oh, there's the last chat. Here we go. Hey, Zawa. So yeah, would for sure do, uh, be open to a Medjammer stream. That sounds awesome. Yeah, so let's keep that in the future, um, for a Medjammer stream. I want to get this workflow done and then let's put in here, um, Medjammer stream, and then, uh, or I've got one— NemoTron 3 will probably be next week, and then yeah, Medjammer might be after that. So let's think about that. So this, this week is VLM fine-tuning, so we want to— I'm going to put this over that side and get out of that. And I want to get VS Code over here. So we've created our dataset. We did that in the last video, create VLM dataset. And now we're going to create a new notebook called 01, um, fine-tune VLM. Beautiful. Can you still see the code? Okay, my face is over a little bit of it, but that should be fine. Import Transformers and import trl. Import torch, and let's make sure torch.cuda is available. We are running on the NVIDIA DGX Spark. Beautiful. So this is our Spark, what we're running here, and I'll just put the docs put a markdown here. Let's go fine-tuning a VLM for structured food tag extraction. Goal: input food or not food. Output is going to be, um, structured data for whether or not the image is food or not, plus, um, food tags, food and drink tags. And then we'll go resources, docs example. So our main goal is to get this docs example running but on our own custom dataset. And that's probably how I'd recommend, uh, you learn a lot of the different things, is find a working docs example but then customize it to your own dataset. So device equals— wonderful. Hey, IX, how are you? Oh, she can't send. Should I turn timestamps on? Okay, there we go. Not gonna lie, I fell off the grind sometimes, and sometimes seeing you go live consistently is inspiring, bro. Let's get it. Oh, thank you very much. Seriously, you just got a 32B Deepseek running on two Arc A770s under Linux with 32 contexts. That is epic, Dead Cat. Okay, let's do— I need to— if you can't share links in the chat, maybe I need to— that's a setting I can update in YouTube Studio. But let me know if that's something you'd want to see in the future of, um, getting links shared in the chat. That's probably helpful. So, set up to, uh, environment. What do we need? PEFT. We need PEFT. I think that might come with, with TRL. So PEFT stands for Parameter Efficient Fine-Tuning. Let's write down some definitions. Because this is a tutorial, right? I'm figuring out things as we go too. Well, sorry, it's a preliminary tutorial. Parameter efficient fine-tuning, um, only train a subset of the model parameters. Plus hopefully still get good results. And then next, Laura is, um, low rank adaptation, which is, uh, sorry, adaptation. Can I spell this morning? No, I can't. Laura equals a form of, um, PEFT. And then we've got the link to LoRA here, LoRA paper, low-rank adaptation. And then we have QLoRA, which is quantized LoRA, which is even smaller. Um, so fine-tune an adapter. Oh yeah, so that's another thing. You basically fine-tune an adapter model for your original model and apply— optionally, um, apply the adapter to the original model. And then QLoRA is Um, I believe train with quantized weights, even smaller than LoRA, for even more parameter-efficient fine-tuning. Okay, I think that may be enough of what we need to begin with. Oh, and actually there's a great blog post by Docs example. And then we go, um, Laura without regret. So here we go, from Hugging Face and also from Thinking Machines. Now basically, LoRA without regret is a blog post from Thinking Machines, which is an AI company. And they basically say you can use LoRA on LLMs. I don't believe this is for VLMs, but we're going to figure it out, aren't we? They say you can use LoRA on LLMs and get almost the same results as fully fine-tuning. And so the benefits of LoRA is, um, benefits of LoRA is equal— you can train multiple adapters only and run those at different times depending on the required use case of your model, right? So instead of training a single model for many specific tasks, you could just train 10 LoRAs. And because adapters are quite small, you just have a base model and then you train the adapters to do specific tasks. And this is actually done with Apple's Foundation Models. Yeah, Foundation Models Framework adapters. So this is one of the most, I guess, tasks or use cases that you can do right now at a very large scale. So Apple Intelligence runs off Apple's foundation model, but you can teach the on-device model. So every iPhone has an on-device model with, I guess, newer iPhones, an on-device language model, new skills specific to your app using a custom adapter. So this toolkit contains a Python training workflow and utilities to package adapters for use with the foundation models framework. So the really cool thing is that you can just train the adapter and then deploy that to device and customize. Yeah, here we go. They're using, they're using parameter efficient fine tuning here. So let's go here. Example of LoRA in the wild. Okay. So this is all well and good, but our goal is to figure this out with firsthand knowledge whether we can actually do this or not. So let's go. Can I import— do I need to import PEFT? Let's try. I might not even have PEFT. Okay, let's install PEFT, which again is parameter-efficient fine-tuning. Maybe we put it here. So Hugging Face is on an absolute tear lately. Where's PEFT? There we go. Okay, beautiful. We're going to install PEFT. I've been using UV a lot lately. UV is epic, works really fast. Can I just do this without restarting? Oh yes, I can. Lovely. So now we have PEFT. So what else do we need? We We got perf— do we have bits and bytes? So import bits and bytes from transformers. Can we import it from transformers? And we're not going to copy any of the code here, by the way. We're going to write all of this from scratch. Okay, I should be able to work it. Let's just see if we can go through this tutorial on Google AI's Docs, but for our own dataset. Yeah. So see here, this is load by Phil Schmid. So Phil Schmid, I believe, made this tutorial. So thank you to Phil. We're going to replace this with our food extract dataset over here. So let's just go from the top. We should have our developer environment set up using device We don't need to use Google Colab. Well, you could run this in Google Colab. So I think we need to structure our data something like this. Let's load our dataset. System message. Do we need a system message? Maybe we do. Let's try it without a user prompt because we just want simple image in, extraction out. So I'm going to load the dataset that we've created earlier. This is public, by the way. So it's going to download. That's incredibly quick. Thank you, Hugging Face, for that. And let's have a look at the first sample. Number 0. So this is, yeah, exactly what we want to output. We want to output this. And let's have a look. Do we have image— oh no, we could just display the image, can't we? Print example sample. And then we go example sample That's the full data sample that we have. Example input, which is our example sample image. And then we have example output is example sample, um, output label JSON, because we're doing structured data extraction, right? Our output is already structured. So print, let's go example output, new line. Beautiful. And then we go, let's just go example input. This should be an image. So simple as that, right? Now, how do we structure this to be in the format that we need? So we have, uh, messages, system, content, type, text. Text is that, and then type, role, user content, text, and image. Math asks, do I use Weights and Biases? Yes, I do. And do I use Google Storage? Yes, I do. Bring your own bucket. So that's exactly what I do. Weights and Biases, but then I just link Weights and Biases to a Google Storage bucket. So do we actually need this? Let's go to Um, TRL, then Community Tutorials. Do we have VLM? Small VLM. Okay, let's see what this looks like. Images, sample image. Okay, we can come back to this one. But I think maybe we just go— maybe we just trust the system and just copy. I'm going to copy this verbose, basically everything. And then we'll see if we can reduce complexity. So example input. Let's now go here, format data for use with a VLM. Our VLM input requires our data to be formatted in a conversational style, so System message. Let's go system— oh, we'll create it like this. System message equals go, you are an expert food and drink Image Extractor. You provide structured data to visual inputs, um, classifying them as edible food, drink, or not. As well as titling the image with a simple food drink related caption. Finally, you extract any and all visible food drink items to lists. All right, something simple like that. We can— that's, that's text. We can always update this in the future, but for now we'll keep it as that. So user prompt with image input as well as desired output. Extract, classify the input image. Into food or not. And if edible food or drink items are present, extract those to a list. If no food or drink items are visible return empty list. Only return valid JSON in the following form. So we're writing all of this from scratch on purpose, right? Because We want to be familiar. Int 0 or 1 based on whether food drinks are present. 0 equals No foods visible. 1 equals foods visible. Image title: string short food-related title for what? Okay, let's, let's run with this, see how far we get. And if it doesn't work, we can update it and come back later. I just want to start as simple as possible. So classify the the given input image into food or not, and if edible food or drinks are, are present, extract those to a list. If no food or drink items are visible, return empty list. So we need system user. When we don't actually have a product name or a category name. In our case, we just have image and text outputs. Okay, this is good. Do we want the image? Usually people say put the image first. So I'm going to try to mess this up and just put the image first. or change this up, not mess it up. Come on, Daniel. So messages, and then we want this as a list, and then we want a dictionary in here, and we'll go role: system, and then the content is going to be type: text. We want to see— let's see if we can get our model running by default here, right, first. And what I mean by that is that we want a non-fine-tuned model so we can get a baseline output. So the role here will be user and the content will be type— I'm going to put the image first, as I said. Image, and then we'll go sample. And what is our sample called? Image. Beautiful. This is why we spent some time creating a dataset the other day, because now we're up to content we want in here. No, this is going to be text, excuse me. And then text will be user prompt, and then we want our model to give back. So see how we go. Let's write this down. And this is setting the scene. So the system prompt, and then message 1 is user input. Image plus text pair, image plus, oh, plus prompt pair. And then we have here message 2, which is user, and this is system. And then we have this, which is also user. Oh, sorry, this is the model talking back is ideal model output, e.g., our structured data format. So let's go here. Role is going to be assistant, and then we're going to put in content type text, and then text— the ideal— this is our model's ideal output is output label JSON. So we want our model to, given this image, output this JSON. Okay, so we'll put this down here, this field. This is what we created the other day. And we go sample. Okay, so that's not too bad actually. So with system, we set the scene, right? I'm not sure whether this should be up here. In the system prompt or in the user prompt. But for now, we'll just, we'll just see how this goes, of course. So return format_data. Wonderful. So if we give our sample to here, formatted_sample, sample equals— oh, excuse me. There we go. So the input is: you are an expert food and drink image extractor. You provide structured data to visual inputs, classifying them as edible food or drink or not, as well as titling the image with a simple food or drink related caption. Finally, you extract all visible food or drinks items to lists. And then we have structured data input. Oh, sorry, image input and a prompt input. And then we have the assistant. We want our ideal output to be this. Very nice. So we've got a— we've got our first formatted sample. So what do we need from this format sample for datasets? So process vision info. So let's go, um, updated dataset or dataset processed. Equals format_data sample for sample in dataset. Dataset processed 0. Beautiful. Messages format each item in the original dataset keeping the PIL image type. If we use map image may, um, note get converted to bytes. No, I might put a TK here as well. TK means I come back to it and check it. Not 100% sure if this is true. Let's find out. So let's find out. We can always re-download So if we go to our other notebook, we have a format_sample function. While this is running, I'm just gonna go to the bathroom quickly. Be back. I wonder why map is taking so long here. Usually this should be quick. It's not like it's a, it's a tough map to do. Maybe because the image type. We don't get messages. I'll come back to this. Okay, so let's say we have our dataset formatted. Now we need a model. I want to try use our sample first on— should we try Gemma 4 or Gemma 3? Sorry, Gemma, Gemma 3 4B. Google Gemma 3. That's— so that's Gemma 3, 270 mil. Maybe Gemma 3N. Let's try Gemma 3N, see if we can get that to work. And let's try the small one. Okay, this is what we need. Gematria N. Yeah, that's correct, Alex. The map function applies that to the whole dataset. But for now, we just, we just want this, I believe. So I think I should be able to pass this into— or actually, I only want to pass in the user and see what our model outputs. So we're going to see what if we pass just this. So here, messages, and then if we go up to 2— yeah, there we go. Let's create a pipeline. Today we're going to try and fine-tune Gemma. Let's call this our model ID, and in the future we can try to do a smaller model. And then dtype is just Spelled pipeline wrong. So this is going to download Gemma3n. That might take a few minutes, but that's okay. And then we can get ready for the next one. Let's see how this goes. So model loaded. Let's see, how long does this take to do? Oh my gosh, that is fantastic! That's what we want. No, we want one, and then we want content, and then we want They want zero. And then we want image. Okay. Cheese and crackers with dipping sauce, cheese crackers. Okay, so it is okay. At least it can return the structure is food, which is what we want. But we want it to extract all food items. So we've got the cheese and the crackers, but it missed the dipping sauce and salad. Here in the text. Our ideal output is— let's go here. Where's our example sample? Okay, so we're close, but we're not exactly to where we want to be. Let's now work on fine-tuning. And we get a question. Hi, mate, how long can you focus for a duration? I can usually only— usually only focus for 1.5 hours, then I need to rest. Yeah, probably about the same for me. 1.5 hours, and then I rest, and then I come back to another 1.5 hours. If I get 3 of those in a day, that's usually a pretty good day. But what are our rules? We're coming up on the hour and we've got a little timeline here. Let's go. Next is improve base model to be more aligned with our ideal outputs. So what we might do is load our model like this, see if we can use bits and bytes. So load in 4-bit. Okay, I wonder if we can be do this. Let's go here. Because then we want to do LoRa config, LoRa, SFT, collate. Okay. We'll come down to that in a second, but we're up to here. Okay, time for push-up times. Westside Detroit, I do remember you. How is your company? Have you made any new scientific discoveries or made progress on your family's app? You finally got me into AI. No scientific discoveries, but our app is still going. Nutrify, it's running, it's downloading on the App Store. We just released a 2.0 version and you can go try this out right now. So if you go to Nutrify and then look up the, the blog post here, you'll be able to see how it works. But one of our goals is to be able to run this. So this is using Gemini extracting food information, but I'd like to be able to use an on-device model. So that's kind of why we're fine-tuning a VLM. But it's 1 hour, it's time to do 10 push-ups and 10 squats. Who's joining in? Put in the chat if you're joining in. We're doing 10 push-ups and 10 squats. 10 squats. Beautiful, let's do it. We're just copying this for base, right? Or verbose, right? And we can customize it later on. I just want to get this working on my own custom dataset, and then we can really step it up a notch. Now this is the one I haven't used before, bits and bytes config. This will be interesting. Hey, well done on joining in on the push-ups and squats every hour, you know. Let's go to our counter. That was on Monday, now we're on Wednesday. +10. This is machine learning and muscle, right? Load in 4-bit is true. I don't know if we need to do this, But I'm going to do it, and then I'll probably try without it. B&B. I think this is, uh, replaced with just D-type as well. Moment of truth, let's load our model. Typo. Oh no. Okay, I think this should be fine. I thought we might have to do some kuda. Hey, you haven't worked out consistently in almost a year. We got to step those games up, right? Those gains. Oh no, error. Let's try just pip. Let's restart the notebook. I'm just going to redefine the model ID. So we need to research what this LoRa config does. There we go. So I've got our Gemma model loaded with the bits and bytes config. Let's see if we can do some inference on that. We'll go back up and load our dataset. What parameters here? So this is a good finding. What matters for LoRA? For supervised fine-tuning on small to medium-sized instruction tuning and reasoning datasets, LoRA performs on the same the same as full fine-tuning. There we go. We get the 4 outputs that we want: cheese, crackers, sauce, cilantro. Well, this is— I would say that's bread. So this is what we want. Rank. So they swept over the rank here. What's the default LoRa config? Let's find that out. Pfft. Rank 8. Let's just try the default, or very close to the default. I want to print all of these. How would I loop through this class? Let's ask Claude. Attributes— there's going to be an easy way to, to loop through attributes, surely. Oh, can I just turn this into a dictionary? Vase. Cool. Here's our default config. Rank 8. Laura Alpha. What's the Laura Alpha? They say alpha here. Optimal learning rate for full fine-tuning is lower, a factor of 10. Hyperparameters. They use a alpha of 32, whereas the default is 8. That's okay, just going to clean up some tabs. Okay, let's try these settings. Or actually, let's try and make sure we can run the model. On our example sample. Yo, Australia, that's where I'm from. Uh, Australia is very hot at the moment. It is, uh, 30-something degrees today, I believe. Well, if it's not 30 degrees, it's very humid. So let's go processor, apply chat template. Messages. I want to go, uh, example. We're going to add generation. C++ math is all you need to be very good at machine. Okay, that is perfect. Let's pass this to our model now. So our input is about 500 tokens, and now we'll do inference. Is this gonna work? What did we get back? Does it need a URL as the input? Maybe. Let's just get an image with a URL. Um. What token is that? So we're getting— clearly getting not an ideal output here. Image soft token. Maybe I need to turn it into bytes first. Let's try to base64 encode it and see what happens. Oh no, we've created a copy. That's okay. Does this work on a URL? Okay, cool, it does. Uh, VLM stands for Vision Language Model. Wow, that has saved it way back to here. I need to fix that. So, something is happening on the image tokens here. It's not getting encoded correctly. Yeah, definitely. We'll definitely have models running on locally devices. In fact, this is running locally on device right now. So inputs, apply chat template. System, text, user, image. This is correct. So why isn't my model here? Maybe I've got something wrong. Let's see what Claude says. We're just gonna dump this in and see what happens. Pipeline works, so we know it should work, but for now we're just getting empty outputs. I have a suspicion it's something to do with these soft tokens, but I don't know. Let's figure it out together. Hey, what do the issues say here? Okay, I don't think this is any different to what You know what, I've had problems in the past with inference mode. Let me try this down here with no no-grad. Okay, now let me try it with just the input IDs. No, that's actually still the same thing, isn't it? Apply chat template is— let's just show you. Going to turn it into that. It's like basically applying the preprocessor. Because this is what the model is going to expect. Let's just try what Claude said. Still getting the same output here. Device error here. Yes, see, we have this 262145. Image soft token. Why do we have that? That's the end of the image, and the start of image should be that token. So we're digging deep into the tokenizer here. Kevin McAllister, hello! We are trying to fine-tune a VLM, but for some reason I'm not getting an output. How I'd like, and I'm trying to figure out what's going on. So the pipeline works. This, as far as I can tell, should work, but again, it's not. So that's why we're doing some live troubleshooting. Let's just read through this again. Add generation token. We got an image. This is putting everything to dtype here. Model.generate. Maybe I need this, Gemma3N. What model is this using? No, this is using the same model. What Tim version do we have? Okay, we should be well and truly on the way there. Apply chat template, self.history. Yeah, add generation prompt. Yeah. Do they have a system prompt here? Okay, let's take out my system prompt. No way, ladies and gentlemen! Has this worked? We won't know until we know. Oh, okay, so the pipeline definitely does some stuff behind the scenes, but at least we get the correct output. Well, we get a output. But I think we might need this in bfloat16. Because now if we decode, at least we're getting an output now. And what order did they put the So the pipeline is definitely behind the scenes. Oh, is it because our model is in— ah, let's not load it in because we loaded it in bits and bytes before. Let's load it in normal precision and see what happens. See, this is the power of going through samples. We are slowly but surely figuring out errors in what we're working with. Little DJX Spark is going great. Oh my gosh, we got it! We got it! Cheese and crackers with sauce. That's exactly what we wanted. Now why did it get that wrong? Well, because our dtype was wrong. The input dtype was different to the dtype that the model was expecting. So something to note— note, if you are loading a quantized model make sure the inputs are quantized as well, otherwise you will get errors. So our next step is to investigate this a little bit further, but we came through the classic TK note This is one of the classic ML problems— input DTypes being different to the DTypes your model has trained on. Note, um, if we are running this way, format the system prompt to either be removed or folded into the user prompt. So I'm going to write some notes down here so that we can notes. Make sure model inputs are the same dtype as the model loaded, e.g., if model is quantized, inputs should be quantized too, or if model is If we're not using transformers.pipeline Be aware. So let's go. We've got a lot of stuff going on at the moment here. And is it hour 2? Nearly, 3 more minutes till hour 2. So we've got it working just without the quantization bit. So let's go back up here. This is another resource. So where did Q Laura come into fat play? Wonder if we can change with just Laura and not Q Laura. I'm going to try with just Laura and then we can do Q Laura. Actually, no, I'm gonna do Q Laura first and then try Laura. All right, I believe it's time for some push-ups and squats. Put in the chat if you're joining in. We got 10 push-ups and 10 squats. 10 squats. Beautiful. I'll be back. I'll go to the bathroom and be back. Let's update the counts. Sorry, just had to check some messages. Someone's trying to buy my old camera. Hey, Westside Detroiter, you're crushing it! So let's now set up our LoRa config. Now the docks on Laura. And this is the default config, but we're just going to copy this and see what happens. Now the R is the rank. I'm going to keep the default 8 and see— we're just seeing what happens with the exact defaults, and then we can try change things. Oh, they're using 16. I'm going to put some notes here. These guys got any new blog posts? Because their blog is really good. Vision input, you can fine-tune on Tinker. We're going to fine-tune locally. So target_modules is going to be all linear. I'm not exactly sure what that means, but we check the docs. Okay. All linear. What's our default task type? Do we need that? I don't know if we need a task type, but I'm going to just do it anyway. And then So modules to save, list of modules apart from adapter layers to be set as trainable. Okay, so I think we're going to train the embedding tokens along here. So let's restart. And I'm just going to go from the top, basically. So we've done a fair bit of good troubleshooting for, uh, both loading a model, running it on one of our example samples, loading a model with LoRA— or, um, sorry, quantization. And then seeing that quantization in its current form doesn't give us good outputs. So that's actually a good thing. That's some information that we want to improve upon. So now we're setting up our LoRA config so that hopefully when we train our model in a quantized fashion, the next version of the model, because it's been fine-tuned in a— on, uh, in— when it's— so when we first quantize it, it's not used to being quantized, basically. So the fine-tuning is going to get it used to being quantized so that hopefully our next lot of outputs are better and more tailored to our dataset. So we need to set up an SFT config. So this is loading in Quantized. Let's just train it for one epoch, eh? So we know that our outputs at the moment are going to be poor. Yeah, so that is our default output at the moment because our model's quantized, but our— it hasn't been trained with quantization. So now we're going to use a LoRA adapter to fix this, or hopefully fix this. We'll just train for one epoch and see what happens. Let's see if we can train with a, a batch size of 4 because we have quite a good GPU here on our local device. I want to log every step and save every epoch learning rate. Again, this is all experimental. BF16 equals true. Now I believe this is based on the paper, QLoRA paper. Yeah, I haven't read the QLoRA paper. I should read that. Most of these settings. We're not going to push to the hub, we'll do that later on. I'm actually sure what a lot of these do, but we'll get it working and then we'll discover more. So now we need to make— so we've got our arguments here. Good. This is going to be passed to SFT trainer. So what do we need for this? Create a data collator to encode text and image pairs. Mask tokens for not being used in loss computation. Where's our list of tokens? 144. Now I'm not sure why they put the end of the image as -100, but let's create Our fields are the same, so we should be able to copy this. Ah, I see. That's where this function comes from. I believe that's what Process Vision Info will do. So let's go back up to here. So next— Beginning of image token. The labels are input IDs. And we mask the padding tokens and image tokens in the loss computation. After you quit your ML job at that consulting place, how did you end up being a full-time business owner? Courses, yeah. So that was the majority of my income for a long time was courses. I've since done some more consulting, since published several apps, getting revenue from those, and entering Kaggle competitions, winning those. So YouTube, of course. So several different levels of income at the moment. But I've been thinking about applying for a job somewhere like a tech company. And getting back at the game so that I can practice my skills a bit more. Of course, building, uh, real-life projects is probably the best way to do that, so I'm doing— trying to balance it all up, you know. I'm just trying to figure out why. Convert. Ah, I see. So the base model input here would be— let's see what happens here— processor For me, the labels are actually just this. I wonder if we didn't match or max. Let's see what happens when we try to run sft-trainer without this. Ah, we don't have messages here, do we? We're going to check out what this dataset looks like because I think we can actually simplify this. I don't know, but that's what we're going to try. What is I? Cool. Yeah, that's right. Small example first, and then we get full fine-tuning. Oh, this is cool. That is cool. So only 16% of our model is trainable. It's still a lot of parameters in my eyes. Maybe if we change this— still 16%. Now that is cool. No, I haven't had lunch. Haven't had breakfast either, just a cup of tea and a cup of coffee. I probably should eat something soon, but we're on a roll here. We're gonna get this working. Okay, I'm gonna go with these pa— uh parameters and see where we get to. But the reason I was looking in this notebook is because I think we should be able to Cool. Let's shake up our dataset. So we want to shuffle this so it's random. How do you shuffle list? Random shuffle. We're just creating a train split, test split. I want to see if I can get this to work without doing all the processing stuff. Bam! So if we have a look at a random training sample now, we should have Beautiful. I want this to be reproducible. There's some syntax I'm missing here. Maybe I'll just do it with NumPy. Jayden W says— ay, you says meet you at Sandgate Maccas. I live close to there, that's where my office is. Well, I live in Sandgate. Jayden, you from Australia? Where do you live? Beautiful. This is what we're trying to do now. We've got a val dataset and we've got a train dataset. So let's see if we can just straight up train our model now. I actually don't know whether this will work. We're training quite a few parameters here, so I'm not entirely sure. Let's see what happens. Hey, fine-tuning a VLM, do we get it right? No, we don't. Okay, we don't have TensorBoard. Oh no. Okay, we need input IDs. What did they do here differently though? They're only using— I need a cup of tea for this. So on this one, they don't format the dataset. Right, if we look at this, just image and text pairs, I think we can get it similar to this. Watching your PyTorch course at the moment. Cool, how's it going? Hope you're enjoying it. So Role user. Yes, image text. Maybe I gotta restart this because I think— where's my sft-config? Hey, glad to hear that you like it. Let's read this error. Do I need to train the PEFT model here? You know what, I'm going back to this. Let's preprocess our— this is not that hard. So we get the image token ID, which is this token. We're nearly at 3 hours, by the way, so we're going to Do some more push-ups and squats then. Processor. So this is Beginning of image token. Now mask this token processor, which is the pad token ID 0. What dataset am I using for fine-tuning? I'm using Food Extract 1K Vision. So the goal is to have a food image input and then extract structured data from, uh, from the image, as in what foods are there, is it food or not, etc., etc. So it's only 1,500 samples, but it's a real-life dataset that I've custom made myself. And the goal is to— we've got some labels by Quen3 VL8B, but the goal is to fine-tune a smaller model to do the specific output that we're looking for. Oh, and it's 3 hours in. Let's do some push-ups. 10 push-ups, 10 squats. Who's joining in? Put your name in the chat. We're going to get our counts up. Let's get some squats. All right, let's update the counts. Now I'm not actually sure why they do this here, why we mask the zeros, but All right, we're training. So we had a collate function here. Let's see if our model— oh, we're getting quite— there we go. Oh good, training's working. Okay, apparently this is going to take 30 minutes. At least the loss is going down. Now loss is going up. Now I need to figure out why this is. Yeah, I think I've added too many PEFs. Let me figure out why I can't see. We're going to look at VRAM. Let's run this little script. Uh, Brian Another asks, any chance you've compared training speed with something like a 5090? I've been tempted to get a GB10, but it seems like it'd be far slower assuming the model fits in memory. So I I don't have a 5090, I have a 4090. Uh, I haven't yet run them head to head, but after I finish this VLM tutorial, or I guess after I get it working on the GV10, because that's what I'm working on now, uh, I'll try it on the RTX 4090 and see how it goes. I'm not sure why I can't see the GPU memory, so I don't think I really want to wait. Half an hour. So maybe we change some settings. The loss is going down though, this is good. See, if we wait 25 minutes and it doesn't work, that's not fun. You reckon— who reckons we should leave it? Put in the chat, vote, or leave it, or stop and try something else. We could leave it and then read some other articles. I'm gonna write down here next of what to try. Namaste. Okay, the loss is driving, so let's let it run. All right, if you guys are happy to wait for them, I'm happy to wait for it. That's 23 minutes. If the loss— okay, here's the deal. If we get to, say, 25 in and the loss starts going up like that, then maybe we, we try again. But we'll just leave it as it is for now. If it makes you feel any better, my current run has about a month or two left on training. Okay, Brian, that is a very big, uh, fine-tuning run. I'm not sure why we can't see the, uh Memory. Anand, I'm working on that, on LLMs fine-tuning. You can go here. This is the version 1 of the, uh, upcoming LLM fine-tuning tutorial that I'm working on. And you can just go to my YouTube if you just want to watch it. So I've already got an LLM fine-tuning on YouTube, um, but if you want an official course, I'm going to turn this into an official course soon. My plan is to have a VLM and LLM project up here within the next month or so. So then I'll record the course videos for those simultaneously so that they can both be edited and uploaded at the same time. So the loss is still going down. Westside Detroit, uh, thanks for tuning in. All the best with study, my friend. You got this. I'm cheering you on all the way from Australia. We're getting some warnings. Let's copy these warnings, so We know what to look at. Oh my gosh, we copied so much. If this works, I'll be quite shocked actually. It's usually not the first try that Why did that copy so much? The loss is still going down. We're only training for one epoch as well. Beautiful thing, watching that loss curve go down. Let's put this over here and over here. Let's, let's check the huggingface.com models page and see if there are any new models. Quen VL3 embedding, wouldn't mind trying that. GLM image, cool. I haven't seen this one before. Exactly right. So the loss is going down with the current stats of— this is what we're working with, 16% of trainable parameters. And you can probably actually get much less because in the SmallVLM docs, or up here They used— can we get the number of Laura? So they train under 1% of parameters. We're doing 16%. Oh, loss blew up a little bit. It dropped right down and then it blew back up, so we might be in a little jagged curve. But we're almost halfway training our first VLM, cuz then what can we do next? Save the model. Anand asked, can we expect production-ready ML? I mean, MLOps. I mean, how to set up a nice pipeline so it won't suck in production. Can you give me a specific example? Like, what would you like to see? Like like just setting up, because, because the models we've built in many of the courses can be set up for production use case. Let's see if this works. Hannibal, how are we? Loss is still going down, people. This is good. We could be overfitting though. Cuz what are we using here for our Laura? Yeah. I'm very happy too that the loss is going down like this. We're gonna, we're gonna ride it right to the end and we'll see what happens, right? We can also plot the loss curves if we do get some eval loss. Because we have— we might be able to plot some loss curves after this. This is really live. Yep. Smoothie the cat, I think you asked that yesterday. Now let's go back to the Hugging Face model page. Medjema. How long we got left? 8 minutes. Let's see if we can run Medjammer. On Google Colab. So the loss is fluctuating at the moment below, just below 1, which is good, I think. Oh, can I get a H100? If I can get a H100, this will be cool. What do I think of the Deepseek V4 rumors? I'm not sure. What are the rumors at the moment? My gosh. Look at these videos. There's a lot of clickbait AI videos these days, and we want Chinese New Year would make sense for Deepseek to release it. 17th of February. Did we get a H100? Ah. We didn't get a H100. Let's load Gemma or Med Gemma. What image are we inferencing on? So the loss is pretty good at the moment. We're trying out Med Gemma. I must be the only person on earth who hasn't used Claude Code yet. Oh, fine-tuning is going to be done in 2 minutes, ladies and gentlemen. We are fine-tuning our first Wow, here we go. Wow, the image is big. Okay, I have no idea what I'm looking at when I look at an x-ray, except obviously that's a rib cage, right? What's happening here? What's Medjema said about this? Overall appearance. The image shows the chest cavity including the lungs, heart, ribs, and other structures. Lungs appear clear with no obvious signs of consolidation like pneumonia or large masses. The lung markings are visible, including normal lung tissue. Heart— can you even see the heart? Is that the heart there? Or maybe there? Diaphragm visible, no obvious abnormalities. Cool. Let's go, um, BCC skin cancer images. Let's replace this image. Oops. I don't want to describe this x-ray. I wanted to— this image appears to be a close-up photograph of skin, not an x-ray. It shows a slightly red Right, so we're just going to do an open-ended question here. Describe the appearance of this skin photograph. What is visible? Return 3 questions I should ask my doctor about it. Oh no, I went back. Oh no, there we go. Appearance: the photograph shows a close-up of skin with a noticeable red, slightly raised lesion The legion appears to have a somewhat irregular border. Essentially, there's more red visible. Red Legion. Oh, this is cool! What is the nature of this legion? Sweet! Okay, Med Gem is pretty good. Claude also released a medical model. A fine-tuning— look at that, we finished on that! Do we get training loss? That is incredible. Look at that loss curve. We better save the model, right? Okay, moment of truth. Do we have a saved model over here? Checkpoint? Yeah, let's save the model before That's a beautiful loss curve there. So now we're going to Test the model. So we've got our model saved, which is the most important part. So the fine-tuning— oh, just want to put out some time there— took about 30 minutes. And that was 14th of January. Put that loss curve in the Louvre next to Mona Lisa. You're exactly right, uh Alex, that is the one of the most perfect loss curves you'll ever see. So let's now From pre-trained loaded model. We want to merge the LoRA and base model and save. Let's see if this works. Oh no. There's my adapter model. So what did we actually get here? Let's see if we can load it like this. Load it. So the adapter path is going to be Hey Alex, thanks for tuning in, my friend. All the best, I'll see you in the next one. So we've saved an LM head. I think we may have saved the wrong parameters here. That's okay though. I've got an idea. Let's load Gemma 3 into here and then see what it looks like. I'll be back, just gotta go to the bathroom. Uh, token here, I'm streaming regularly. Yeah, I want to try and stream more this year, so Check out my channel. I've already done 3 or 4 live streams already. This might be the 4th live stream of the year. I'll be back in a minute. Uh, not always the same time hours I'll be streaming. Um, I haven't got a schedule at the moment. If people think having a schedule would be good, let me know, but otherwise I'm kind of just doing it ad hoc at the moment. Model. Okay. LM head. Ah, there we go. Whereas I might need to— and embeddings, embed tokens. I see. Whereas currently, lm head— oh no, wait. Let's just check the model, the original model. Okay, so their gemmetry uses Now let's compare these architectures. Anand, what was your question? So what we've done is we've saved an adapter on— modules to save, there we go. But I don't know if our model has an LM head. Oh, it does. Let's see what Claude thinks. Let's try another way to load the adapter. Anand, if you could repeat your question, that would be fantastic. That would help me answer it. Default. Okay, so if you want to deploy your model, you have two options. You can deploy it on device, or you can deploy it via API. And if you want to integrate new data into the model, you need to figure out what data is coming into your model, save it into a database, and then retrain that model on that updated new data and then redeploy it. So you can deploy models on, say, a hosted GPU, or again, you could deploy it on device. For example, in Nutrify We have a model that runs on device to identify different foods, and then we send an image to Gemini to get back inference from that. So there's two different ways of hosting a model in one app. And of course, we log images that are sent to the API and to the database so that we can update the models over time. And to load new data, it's just like loading data the original time, except you have more samples or different samples this time. So I think I shouldn't— I double saved my model or adapter. With adapter name default. Maybe I have to reload this model. How's that happening while you're training on new data? So the old model is just like a file, like it could be called model 1, and then you have model 2, which you train on, uh, or create from scratch if you wanted to, and then you deploy model 2 if it's better than model 1. So we're trying to load in our trained model. And it's 4 hours, so it's time to do 10 push-ups, 10 squats. Who's joining in? Okey doke. Now we get the wrong adapter. Let's update the count. Zimbabwe, hello, hello. Best practices to avoid billings from production Production deploy the model on device. Key error. We may have to retrain. So we saved some corrupted adapters, I believe so. There we go. Thank you, Cloud. We have now loaded our adapter successfully. Okay, so the error is— error: do not stack LoRa fine-tunes. And Laura. So— Let's now test our model. Hey, go back here. We need our dataset back. I'm just gonna load this. Now let's get our Val dataset. Beautiful. So equals, and we want to get messages. And then we want to get one, then the ideal model output would be this. Beautiful. So let's now pass this to our model. Just going to append vowel to all of these, or prefix. Maybe this needs to be a list. Yep. I want to put this on— hmm. Very hacky notebook at the moment, but we're just trying to get this working. Woohoo! Is food desserts baklava nuts? It's starting to work. Let's try with another sample. Okay, takoyaki. Oh no, we disconnected. My poor DJI Spark. Got some dandruff. Let's— oh, did we reconnect? Oh no. Ah, classic. Let me restart. VS Code, right as we were about to connect. Okay, we should be able to get started fairly quickly again. What's happened here, ladies and gentlemen? We're back. Let me know in the chat if you can see what's going on. Okay, this is a good one to test. Wow, carrots, beets, feta cheese, chickpeas, green sauce. That's pretty good. What about another one? Macaron assortment. Yellow macaron, white macaron, brown macaron. Macaron. This will be interesting. Looks like our fine-tuned model is working. Creamy soup with dumplings and green onions. Soup, dumplings, green onions. That's, that's pretty close. Like, it's not perfect. Tuna tartare and chips. Tuna tartare. Because what is going into the model here? So I think this is working, and our model is outputting JSON. You notice here, let's see if we can find a not food. Here we go. This should be not food. There we go. Oh, that's beautiful. Okay, let's, let's try it on a food image that I know for a fact has not been in the dataset. So this image is what I had for dinner a few nights ago. Has not been in the train dataset. Remember, our model has only been trained on Food 101 images. Okay, I want to really make this image a bit smaller. I think we can go even more. This is a big image. Maybe even one more. Okay, we're going to put in this image. So def format_sample_to_message, input image. So we need it in this format. So the input image is going to be this one. Just make sure that the image is correct. Okay, let's now pass this And we'll just change this. Dinner plate, beef, sweet potatoes. Oh my gosh, it worked! That is perfect. Wow, that's what we wanted. We fine-tuned it, Gemma. We fine-tuned Gemma 3N. Holy smokes. Now let's compare that to the original model, right? And notice how this is structured JSON. I'm going to take the exact— let's load the original Gemma model. Got a lot of extra code here, but that's okay. All right, so we won't, we won't apply our adapter. Does the DGX Spark have So here we're doing original model— ah, excuse me. Let's compare original model to our new model. Hopefully they both fit in memory. But notice how we're doing original model here, we're not doing the loaded adapter. Okay, so it actually— it did what we wanted to begin with, actually. Hmm, maybe our LoRa adapter isn't loaded. Original model. Have we got some leakage here somewhere? Let's try it with another image. Oh wait. Load adapter. Let's make sure— how do I tell if my model has loaded an adapter? Okay, it's kind of identifying that as KitKat. Let's see if it gets this. So that's actually takoyaki, but it's very close to being correct. So maybe we Ah, we don't have What's going on here? So, loaded model. I don't know if my loaded model had— let's, let's collate a few of these things. I want to make sure we're inferencing here with— okay, we've run out of space. Let's fix a few things up. Default adapter. See ya, Azimuddin. Thanks for sticking around. You were really helpful with all your support through this stream. I really appreciate it. Hmm, let's try this. I think our adapter hasn't been saved correctly. And if that's the case, we might have to go back and retrain it. Hey, what you doing? Oh, your PyTorch and TensorFlow tutorials helped me crack the coding round and get a job. Congratulations, my friend, that is amazing. Let's test this. Hello, Anja and, and Jani. Are you working with VLM? Yes, I am. I'm working with VLM right now. Let's try work with our loaded model. How does it go? Beautiful. So I don't actually know if our model is I need to run some inference over more images. We're going to get the exact same outputs as before because our random seed is So I think the problem I would actually like our model to do is The input prompt, I think, is where most of our tokens get taken up. What would I choose to work with? Core ML/DL building models or build GenAI applications? Both, probably. Yeah, 171 tokens are going in there. Let's get some test images. Wait, maybe I haven't merged our model, that's why. You know what, I think we have to nuke it and start again. Because I think right from the start we caused ourselves a lot of issues. I'm going to delete this, restart the notebook. We caused ourselves a lot of issues Trying to override. Our original LoRA config. Now I wouldn't mind in the future— let's write down here Hey, building a GenAI application is perfectly fine. We messed up when we were doing our LoRa config, so let's Let's see what happens. When we change it from this to be So we've only got 12 million trainable parameters. Why don't we see what happens with that? 5 hours in. Let's do some push-ups and squats. Let me know in the chat if you're joining in. We're doing 10 push-ups, 10 squats. And Jana, yep, this will be on my channel. After I finish it. Oh no, what's happened here now? Come on, little DGX Spark. I think I'm loading too many models. What just happened here? So I'm not going to load the pipeline this time. 5 hours in, we wanted to get a model uploaded, but we didn't. Okay, we're just loading one model now. Let's see what happens. Before, we got an error, so we're only training 0.2% of parameters. So what if I update this? Ah, I see. You can't rerun this. Let's go back. Reload the model. Our problem is our base model might actually be so too good, but it's not really. That's what we get with the quantized input. Okay, we only have 0.4% of parameters that are trainable. So let's see what happens. Assim, we're trying to retrain a model this time. Because I think I messed up. Um, this time— when was this? About 3 hours ago, so that's 12 PM. Training loss is zero. Okay. Okay, this time the training loss is not going too well. Pretty sure this issue now is Virianna Main, congrats on your spark. Okay, we're going to cancel this run. What does this say about— I'm just gonna do straight up Laura. Instead of Q, Laura. Can everyone still see the screen? Hey, you started following me last week, thanks for tuning back in. The problem with what we might be working on right now is that our base model is too good. So I want to try and fine-tune an even smaller model. The process failed before because I was running— I believe it was running too many instances of LoRa. So for example, we did one instance of LoRa and then one instance over the top, and then it got overrun. So now we're in that deep troubleshooting phase of trying to just make sure we have one LoRa input. And then eventually, something I keep noting is I'd like to remove the text prompt. So just have straight image in and structured data out. So there we go, we get a good output. How many image tokens are we using? So these are our tokens in. That's the start of the image. That's the end of image. Now I want to find out how many are between these two. How do I find how many elements, uh, between two, um indexes in our tensor. E.g., I want to find how many elements are between the start and end image tokens here. How many image tokens are we using? 256. That makes sense. So we could halve the number of input tokens by removing the prompt. Okay, I want to update this to be— let's delete the model. And then make this the PEFT config. We got this. You got rejected 3 times this week alone? From what? ML engineer position. Oh, that sucks. Getting rejected always sucks. So we're trying another experiment here. We're just going to keep running experiments till we get it. I might even try small VLM, right? Because I want to make sure that we're actually getting improvements in the model. Now this one's at 348. All right, let's see how we go this time. We're training more parameters this time. Hey, we get some training loss! That's good. Beautiful, that's what we want. Okay. loss is going down. Now, how long did their model train for? See, this is the problem when you publish notebooks that don't have outputs. You don't know how long So this is 2 billion parameters, but I would like to see what's ours. Gemma 3, this is 5 billion parameters versus Claude launched new medical AI. That's really cool. Okay, good, the loss is going down versus Let's look at just sheer footprint. About half the size. Small VLM. And small VLM too. So we could potentially do it on I think we try it next on small VLM 500ml. This is what we're going to try next. If we still get errors here, we're going to go through this notebook. Where's the Claude Medical AI? Oh cool, Claudian Healthcare introducing Labs. See, the problem with these tutorials is that they don't show the final setup, like in input and output. As in, here's what it was before and here's what it is after fine-tuning. So we're getting training loss going down here, but the hard part is knowing whether or not this is actually working. So again, our base model was already probably too good. Whereas with— is there any demos here? Let's try small VLM too. Let's see if this works. Hey Erasely, how are you? Or Erasyl, Erasyl, if I can say that correctly. Uh, startup is going good. It's over here. So we just launched an update, but right now we're running on Gemini, but we'd like to change it to be on device, so that's why I'm Numb to words. Okay, I'm trying to fine-tune some VLMs. Hello, I'm Shorty Yubrick. Do we get an output? Yes, we did. Oh no! The DJI Spark. I think I lost internet for a bit. Am I still live in the chat? I'm fine-tuning a VLM for those who are wondering. Or trying to. Now I think my DJI Spark just disconnected, or my internet just disconnected, one of the two. I've still got internet and I'm still connected apparently. Okay, my DJI Spark seems to Okay, we're back online now. Is this runnable? Okay, start again. What if we changed our model ID instead of Gemma to B? Small VLM too, and then we can try Fast VLM in the future. I do need to install Flash. Let's see how small VLM2 does. Now this model is— let's get the footprint. Oh, excuse me, is the screen back on? I just had to fix some stuff. The footprint of this model is 2 gigabytes versus what we were trying to do with Gemma 3e2b— 3ne2b very big mouthful, is about 10 gigabytes. So it's 5x less. Excuse me, I need to update our model ID here. Let's try small VLM too. First we got to download the model, of course. So we're nearly 6 hours in and we faced a lot of errors. Most of them are my fault. But I think my Spark might need to be restarted. Yeah, there we go. Look at that, it keeps disconnecting. That's annoying. I'm gonna restart the Spark. I actually think that the Spark is still training something because I can hear the fans going off. So I— why is that happening? Due to internet? It could be internet, it could be overheating. Um, what I might do is see if I can get it restarted, and because I've got to go soon anyway. So unfortunately, this stream ends with, uh, us not getting a model trained and uploading to Hugging Face. That was our goal at the start of this stream, and we can't even get our goal up because we can't connect to the Spark at the moment. Let's just find out one more time. Yes, see, we're getting devices disconnected. Oh wait, are we back? Okay, Asim, I think you might have been right, overheating. But it is that time again. 6 hours in, we need to do some push-ups and squats, and then we'll try one more time to get SmallVLM 2 running. And if not, I probably have to go get ready for jiu-jitsu soon. So then we'll also plan of what we want to do in the next stream. Okay, push-ups and squats, who's joining? We are back, ladies and gentlemen. One last try and then we'll have to come back to tomorrow. But that's the beauty of coding and programming, right? Nothing tried, nothing gained. Wow, it must have cleaned itself right up because it's running fast. This is where we aired out before. Look at that. That's some fast downloading. Okay, we're back. Oh no, num2words, what package is that? Okay, I've got numb to words now. My Spark just needed a little break. I'm gonna restart it after this stream. Oh no, is this happening again? Usually when this happens, the SSH connection is broken. Yes, see, look, it's just completely unresponsive now. I thought we were on, we were so close. Every error See, now we're not even connected. I'm sorry everyone, one of the most annoying things is when hardware Yeah, device is not running. Can I restart it from here? I'm going to try one more time. Yeah, you're right, Asim, I should give it a break. I'm going to try one more time, and if it doesn't work, then we'll give it a break. There's still something running here. Hello from Taiwan! Hello from Australia! Can we get just a small VLM 2 output? Okay, good. We're loading. We've got a model and a processor. Beautiful. Oh my gosh, what happened here? So I think it might have to be a slightly different input. For small VLM2. Oh no. Why didn't that work? Let's change the message, see if it can actually do any kind of output. So we got small VLM2 working up here. Input prompt. So Okay, this is good. Now let's try it with a different image. So Gemma too was able to Oh, sorry, Gemma 3 was able to one-shot this image. Let's see how small VLM 2 goes. Should be fine with that. Let's do a simpler input prompt. Just to make sure it's actually working. Potato, beetroot, pizza, meat, piece of broccoli. Okay. So it's okay but not great, right? Let's see if we can train it to be better. This is good. Before, Gemma 2 or Gemma 3 was already too good. So our model's just returning an empty list here. Let's now I'll just write a note. Okay, what if we just trained 11 million parameters? Model ID. Okay, so we don't need all of this jazz. I think our labels are actually going to be Hmm, okay, this might not be as straightforward as I thought it would be. We might have to prepare it in a different way. Let me just check if we need a collation function here. Zeedop. I don't know if this will work. Okay, we do need a collation function. That's all right. But the thing is, we need to change this for— what happens if we just leave this exactly how it is? Yeah, I thought so. Boi token. So that's where we're going to go into I would really like to get this done, but I do have to go soon. Okay, let's get— so we've got the batch input. No, you're so right, Asim. You are right. Thank you. I should take a rest. We're going to come back tomorrow. We're training with 2.2 parameters— 2.2%, sorry. Oh, we got some training happening. Oh wow, is this working? Potentially. But we can't sit here and wait for an hour. But that is a low training loss. Okay, well, we've seen over here how SmallVLM doesn't perform too well on our— if we just go return visible food and drink items in JSON, what happens here? Okay, that's pretty good. And then we want use the format is food. 0 or 1 for if food is visible. Um, food items list of visible Food items, and then finally drink items. List of visible drink items. On it goes. Okay, so see, the original assistant is not that great. Original version of SmallVLM 2. versus the Gemma models were already pretty good. So look, we're starting to make some progress already. This is good. So I'm probably going to end the stream very shortly, and let me just see what I need to— let's write a plan for tomorrow, hey? Okay, so we kind of failed at doing what we wanted to do, but we got— at least we learned a lot today. We didn't get a model uploaded to Hugging Face, which is what we wanted to do today. But we discovered that the model we were trying, Gemma, Gemma 3N, was 10 gigabytes. And now we're using a model that's 10x less— oh, sorry, 5x less in size and 5x less in number of parameters. So I'm going to leave this training for the next hour or so, and then we'll see how it goes. But we can steadily see that the loss is decreasing. So, and this is a good model to test because the default outputs aren't that great, whereas Gemma3N, the default outputs were already good. So it was hard to compare whether the model actually improved or not. So that's what we'll be trying to do in the next stream is making SmallVLM 2 robust, and then uploading it to Hugging Face. So hopefully it can do our task of extracting structured data. So I'm going to let this train. For the next hour. Sorry, you won't all get to be able to see it, but I'll, um, I'll show you the results in the next stream, see how it goes. And then I'll also probably do some little bit of troubleshooting behind the scenes to see how I can improve the workflow for future use cases. But thank you all so much for tuning in. Hopefully in tomorrow's it'll be a bit more successful. But as I always say, um, there's no losses, only learnings, you know, something trite like that. Anyway, I'm gonna go get ready for jiu-jitsu. Uh, I appreciate it, Asim, for tuning in, and everyone else who's out there. And let's, uh, let's keep going and fine-tune our own small VLMs. Hey, I'll see you next time.