Local VLM fine-tuning on the NVIDIA DGX Spark - Part 2.5 - Training the LLM part only

Connecting. Let's do testing 1, 2, 3. Who's in the chat? Potentially I do YouTube and Twitch. Maybe we can do that in a future stream. Pop out the chat. Just going to make sure that this works. Talking. Can you hear me talking? Just going to make sure that this works. Okay, looks like it does. Okay, if you're joining into the chat, say hello so we know that some people are here. Today we want to fine-tune our VLM. Yesterday we spent a fair few hours troubleshooting, getting it ready, but we have some good avenues to try today. So let's come over here. Now these are my experiments I've been running. Waze out, chat. Okay, there's our dataset. There's our goal. All right, here's what we're doing. So finish fine-tuning small VLM to extract structured data from images. Running experiment for full fine-tuning. So I ran some experiments overnight. So for full fine-tuning doesn't quite work. Um, then LoRA works okay but not as well as we'd like. And then I found the SmallDocling paper, which we'll have a look Works. SmallDockling paper freezes the vision encoder SigLib and only trains the LLM parts of, um, small-llm-2. 256 mil. Tried this with small VLM to 500 mil, and it seemed to work well with our data scale. So let's try this on the DJX Spark. So if we go right up here, I'm going to clean up this notebook. Oh, let me show you the paper. So this is SmallDocling, which you can go here, and UltraCompact. So this is from March last year, about a year ago. And what they did was document understanding. So you put in pages and images, vision encoder, visual embeddings, and then textual embeddings, LLM doc tag outs, convert to doc, doc, convert to Docling. So we want the same thing, but we want this to be food images, and then vision encoder, and we want to extract tags out of that. But it could be anything, right? This is the beauty of structured data. So there's some backstory here. But what I was looking at, I read this paper when it came out. Was model training. They got a different dataset to what we want. Yeah, look at the dataset size that they put in there. So we're using 1,000 images versus— this is what, 2 million, maybe 10 million samples? I know, probably 20 million samples. So if we go down to training, they get some pretty good results here, right? SmallDocling compared to Granat 2.5 VL— or sorry, Qwen 2.5 VL. That is a good model, but SmallDocling at 256 million parameters. So what's that? 7 billion divided by 256 mil, 28x smaller, thereabouts. Experiments implementation details. So this is what I looked at. Training. This is the beauty of reading papers, right, is you find little tidbits of, of help and something that you can try on your own problems. So training was conducted using 64 NVIDIA A100 GPUs. I don't have that, that's a lot of GPUs. And they took 38 hours per epoch, but they have much more data than what we have. We're doing on a small experiment. The model was trained for a total of 4 epochs using the AdamW optimizer with a learning rate of 2 times 10 to the -4 and 2 times 10 to the -6. So this is the Vision Encoder was frozen. So if we look for frozen, um, they did have— I think it might be under architecture.

Here we go.

Precisely, we adopt SmallVLM-256MIL, a compact variant of SmallVLM-2.2B model inspired by the architecture of IDFX3, which employs the shape-optimized SigLIP encoder as its visual backbone. It applies an aggressive pixel shuffle strategy to compress visual features, reduces the number of image hidden states and introduces new special tokens to separate sub-images and improve token efficiency. So if we find architecture— there we go. Architecture is in Figure 1. SmallVLM relies on SigLIP Base Patch 512, so 93 million parameters in the vision encoder compared to the 2B— 2.2B version of the same model. Its training data was rebalanced to emphasize document understanding and image captioning, combining the Cauldron Doc— Docmatics datasets with the addition of MathWriting. It uses lightweight variant of SmallLM-2 family, 130 Mil, 35 mil, as the vision backbone and employs a radical pixel shuffle method, compresses each 512 image into 64 visual tokens. Last but not least, token efficiency was also improved in SmallVLM 256 mil by increasing the pixel ratio, pixel rotation ratio, to 400— 4096 pixels per token. Okay, and then one more. There's something about freezing the vision architecture here and training. Treefog asks, am I using Claude code? No, I'm not. I'm using straight up just normal VS Code. Yeah, here we go. To align the element LLM part, we freeze the vision encoder and train only the remaining network to adapt it to the new output format which it hasn't seen before. I like that. Let's write this down here. I might duplicate this notebook, actually. Small VLM. This is the small, small Dockling paper. You can read it. I'll put it in the chat. So this is where they used a VLM for structured data extraction, and it's a relatively— it's a small one, which is what I like. Nice. So this is the method we're going to use. We'll probably just do the first one to begin with and see how that goes. Oh no, what's up with my spark? The connection on this hasn't been that great lately, if I'm being honest. So what datasets do they use? What tools do people usually use to create scientific diagrams and papers? Did you know? So diagrams like this, you could use almost anything. Like, you could use Keynote, you could use, uh, Photoshop. Diagrams like this, um Let's ask Gemini. So for their training data— yeah, pre-training datasets. Layout, tables. Charts, code, equations. Figma, Adobe Illustrator. Figma is probably a good one. Microsoft PowerPoint. Okay, let's, let's do step 1. We're going to freeze our Vision Encoder. And then we might even see later on if we can prompt optimize it. I wonder if we can do small VLM. I'm going to try this, or 500, and see if we can get it to work with the tiny one. And then if it doesn't work with the tiny one, we'll do it with the big one. I'm using Hugging Face right now with downloaded data. So the dataset I'm using is downloaded locally, but it's also hosted. So someone asked, can Hugging Face do streaming data? It can do streaming data, but I'm using this dataset. It's running locally. It's about 1,000 food images. And 500 not food images. But you can see this at Food Extract 1K Vision. Okay, so it works. Let's have a look at the image. Oh, we've already got it. So right now our model is saying no food or drink items are present in the image, which is not correct, right? So let's load the model. Our goal today is to get a shareable model and demo up to Hugging Face. No food or drink items are present in the image, right? That's not what we want. This is the image we've put in. Whereas we want our model to output the right foods. So we've seen that our model right now doesn't output what we want. And so before I was using LoRA, but I think SmallVLM2 256M is small enough to fully fine-tune. And basically, again, we're replicating this paper here which fine-tuned small VLM-2 on 256 mil on documents, but we're doing food images. So in a previous notebook we did— I'm going to put this as— this is going to be 256 mil food extract vision. So we can use a little bit of an increased batch size here. They used— I'm just copying the hyperparameters from the paper. Gradient clipping. So they use the AdamW optimizer, which is what we've got here. They used a learning rate of 2e Yeah. And a warmup ratio of 0.03. And gradient clipping to 1.0, so we're doing that as well. So I ran this on my RTX 4090 as well. And right now our code is very hacky, but that's okay because we are experimenting. So we're going to recreate our model here, check the dtype. Beautiful. And we're going to train it on our dataset. Not gonna use P.E.F.T. Oh, I want to freeze. I've got some code over here which helps us count the number of parameters. So let's see how many parameters. How many we got? So right now all of our model is trainable. Which is great. Oh, maybe we can do this. Now we're going to— yo, Abdouj! No idea what you do as a machine learning engineer. I just enjoy when I follow you and I wish to work like you one day. Have a good day. Oh, thank you very much. You have a good day too. Where's everyone watching from? Shoutouts in the chat. I'm coding from Brisbane, Australia. So next we want to freeze the vision encoder. So if we have a look at our model's backbone, we have a vision model, which is what we want to freeze. We have a connector, and then we have a text model. So we want to train the text model to output our certain format, and we want to output the— or train the LLM head to output our certain format. So let's freeze the vision encoder. In model.model. We got Indonesia, Thailand, beautiful places. So we can freeze it by going requires_grad equals false of gradients in the vision model. And then we go after freezing. Algeria, Peru, India. Oh wow, thanks for tuning in, guys. There we go, so we've frozen the vision encoder. Beautiful. So see how this is in line with— and they used— how many epochs did they do? I'm gonna do one training epoch and just see what happens. But they did four, so maybe we do four. Let's try two. I'll try one, see how it goes, because we wanted to optimize this for quickness. See how we go after one epoch. Hey, Egypt, hello! Ready? Frozen Vision Encoder. Let's see what happens. 1,000 samples. See how the memory is increasing here. Okay, it's going to take about 15 minutes. This was in the AM. Let's speak to Claude. B200 Blackwell can do this too. Yeah, I'm running this locally on, uh, an NVIDIA DGX Spark. So this is my setup here. I'm running this all locally just here. Okay, so I just make a new trainer. I do teach on Udemy. And Foundalog, yes, that's, that's me. I teach AI and machine learning on Udemy. You'll find my PyTorch courses, TensorFlow course, and beginner-friendly AI machine learning course. Right now we're working on Hugging Face Transformers though. Okay, so we can potentially create a new workflow. Wow, that's so cool. So you crazy, you join the, uh that you started the course and then you got on YouTube and joined this random live stream and it's me. That is, that is crazy. The internet makes it a small world. Where you, where you tuning in from? To output our desired structure, e.g., JSON form. The algo knows, algorithm knows. New York City, what's up? Uh, we are building small VLM too. So one of my goals with Nutrify, which is an app that my brother and I have built, is— so this is Nutrify running on my lunch the other day. And you take a photo of food using a vision language model, or this happens on device, and then it goes to Gemini. That's that loading wheel, right? So then it outputs this food title, etc., etc. Sardines, pecans, pickles, etc. But that requires internet connection and we have to send it to Gemini and then we got to pay for tokens. So my goal is to replicate this workflow but with our own custom model. So we're using a VLM to do that, and a VLM is a vision language model. So right now we're fine-tuning small VLM, which is— where is that? Here, which is an open source model from Hugging Face, to— on a small dataset to experiment whether we can do it or not locally. So we're only using 1,000 images, so very, very small dataset in terms of modern machine learning, that is. But hopefully we can get some good results to begin with as an experimentation, and then we can scale up and improve it over time. So that's where we're training here. So, so far on Monday, we created a dataset to work with. Tuesday I had off because I was partying with my wife. And then Wednesday we've completed that. So we ran into. Yeah, the app's available to download. You can go to nutrify.app. So my brother and I, yeah, just, just us two, we made this. So it's like a Pokédex for food if you ever played Pokémon. It focuses on Whole Foods because that's— I studied nutrition. And I mean, Whole Foods, I think people really know Whole Foods are the best, right? So it doesn't scan barcodes, it just scans like actual images. So yeah, this is the app that we built in Brisbane, Australia. You can go get it on the App Store and try it out. If you try it out, let me know. I'd love to hear about it. And in fact, Josh might be able to send me a free, uh, a free month code for everyone who's watching the stream. So yesterday we ran into a fair few problems, uh, with fine-tuning, but nonetheless we learned a lot. And figured out a few ideas for next steps. So that's where we're at. Oh, you lost 30 pounds? That's outstanding. So the tech stack that we made it with, it is SwiftUI. And oh, Josh is here. He might not be able to post links. I'm gonna add him as a moderator. That's my brother there, Developer Josh. He's actually sitting right next to me. So we got to do push-ups and squats every hour, right? 10 push-ups, 10 squats every hour. So yeah, today's project is we're trying to use the recipe from SmallDuckling to customize our small VLM, and we're nearly done here on this first epoch. I think it's sft-config. You're web and mobile dev. Hey, that's awesome. Getting into AI so you can build AI-powered apps more. Good on you. AI's definitely gotten way better in the last few years, as everyone kind of knows, right? When I first started, it was, it was pretty good, but it took a lot more effort than it takes today. You can get some great results almost immediately now. Nearly finished training. So this is— took about 15 minutes. And now we're eval. So what advice do you have after completing your course? Well, that's very kind of you to offer to buy all the Udemy courses. Um, you're a self-taught web dev, so you probably have, uh, all of the modern skills that you need. So my courses are more focused on, I guess, creating your own models, um, and, uh, like training models and just, just general concepts of machine learning. But if you're looking to sort of build like a a generative AI app, I'd say you could do it in a two, two-pronged approach is use your web dev skills and the existing APIs. Like for example, we use the Gemini API in Nutrify. You could use my courses to build a foundation in ML and AI, but then you can on the side use your web dev skills to make like a small app that uses, say, the Gemini API to analyze a photo or something like that. Um, so it could be as simple as recipe tracker dot app or whatever, and then you take a photo of the recipe and then use Gemini to output that into JSON. So that would be a cool project to not only build the fundamentals in generative AI, so how does data come in and out, but then my courses will help you really build the fundamentals of ML in terms of running stuff in Python creating your own datasets, evaluating models, all that sort of jazz. So it's kind of two different skill sets. The web dev and generative AI are very application focused, whereas mine is sort of what I'm doing now. If you want to be able to do what I'm doing now, that's what my courses all help out with. Definitely all of this stuff. If you want to build web apps and generative AI, I wouldn't say that's what my courses are tailored towards. Maybe in the future. I mean, Josh and I might potentially make a course at some stage of how would you build an app like Nutrify, for example. So that's just where I'm at. Yes. And my brother, everything you can feel and touch in the mobile app is by my brother. He's a really good iOS dev. And then the machine learning slash icons and stuff like that and all the data. That's what I handle. Okay, so the model, that's actually really good. 90, 98% token accuracy. Let's now evaluate it on— and that took about 20 minutes. ET Videos, how are you, my friend? Thanks for tuning in. So right now we've just trained a model that looks like it's performing pretty well with a Frozen Vision encoder. Let's look at our loss straight down. We could actually probably keep training it for longer, uh, but for now let's just save it. And what do we get? 256 mil V1. That's what we want. Let's test our model and compare it to the original model. Quick question from Do you dump all into a database for memory? How does that work? Can you give me a bit more context? What do you mean by that for memory? I'm not sure what this is. Let's see what our model outputs given this image. So this is the original, and this is the new one. Oh my gosh, that's better, but still overgenerating. What if I change— I want to do sample. That was quick. Oh no, sorry, that wasn't a generative output. There we go. So our model is doing what we want it to do. No food, no food. That's wonderful. Steak. So we've got that wrong. Now maybe we do a fine-tune stage 2. Drossy, what's up everyone? Can you make money by just making models and selling them, but just like making your own API? Yes, you can. But right now the marketplace for models is kind of overrun by the big dogs like Google. So where small models fit in is not necessarily for API, but it's more so for business practices. So for example, I was hired by recently by a wildlife company, Conservation. They were wanting to educate people about a certain type of animal. and I had to build computer vision models to run them on a mobile phone. So take a photo in the wild and then analyze it. And so it had to work offline. So the model had to run online— offline, sorry. And so I built them a custom model for that and they, they paid me for that. But in terms of selling models as an API, generally these days you're going to go start with an OpenAI or a Gemini or a Claude or something like that. Or a pre-trained model on Hugging Face. The custom model space is more so within a business. So for example, Nutrify, we have an incentive to create our own food vision models so that we can run them on our own in our own app rather than relying on the Gemini API. So as for memories, So memories, um, LLMs, yeah, forget. So memories, you always just want to— you can either one, put it into the context window, which is if you're using ChatGPT or Gemini, the context window is anything you put in here. So this is context. And then say you had a rule, um, only reply to me with succinct dot points. You could save this in, say, a config file, and the model is just going to refer to that config file every time it talks to you. So what people pay me for my ML services, well, it depends on what the project is. But for the recent project, I was charging an hourly rate. And it was, it was pretty well paying. Nutrify uses SwiftUI rather than Flutter or React Native. So the model is going okay. It's not as good as It's not as good as a small VLM. Yeah, this should be ramen, but this is pretty good. Uh, pre-AI, you stored stuff in SQL. Does this fit in the AI world? Yeah, exactly. You can still use SQL. So, so for example, um, the database actually layer doesn't really change that much. It's just when you want to use certain data or not. That's the hard part. similar to a database, right? It's like LLMs work best when you give them the data they need. So let's unfreeze the vision encoder. You can definitely still use an SQL database. You just know modern LLMs are actually good at tool calling, which you could have a tool of like query database. So if you had some information in a database but wasn't in the context window, If someone wrote something about like, say, hey, how many bookings or how much free space do you have at your hotel? Maybe the model calls a tool such as Check Hotel Vacancy and then replies with the information from the database. Maybe we could do some streams on agents and building something like that later on. Let's unfreeze the vision encoder. And see if it provides any improvements to our model. Now, what was our loss here? Okay, I think our model could benefit from another epoch or two, but for now we'll just— we might leave it training for a while after the— after we do get this next one going. So let's go here. Model. What do we have? Oh, we don't have a model, surely. Checkpoint. Oh, pipeline. Do we have a— can we get, uh, ft-pipe? Okay, our model's in here. So now our model is unfrozen. Hey, enjoy the gym, Founder Log. Nice to meet you, my friend, and Um, all the best, hey, in the world of ML. I might see you in a future stream, and if you've got any questions about, uh, my courses or whatnot, feel free to send me an email. You should be able to find it on my website. Oh, do I have a ute, Land Cruiser type truck? I would love one. But they are one of the most expensive trucks you can get in Australia. So now we're going to fine-tune our whole model. Let's see what happens. Ratio held. Hello. So what I might do as well is simultaneously train another model on the RTX 4090 for Small VLM, 256 mil. But we go— this is good. We go from outputting the wrong output to outputting structured data of what we're after. Not perfect, but we're only using 1,000 samples, right? Drossy says, really enjoy this type of content. Thank you very much. I really appreciate that. So we're loading. This is stage 2 after it finishes training. I don't know how this is going to look, but that's what we're all about, right? Experimenting. So maybe we copy this code down here. And run it with Stage 2. I'll be right back. Ashik, what's going on, my man? Thanks for tuning in. Just go on the bathroom, I'll be back. You guys watch the model training for me. Now we're 1 hour in, and you know what that means. We're going to be doing 10 push-ups and 10 squats. Who's joining in? Put your hand up in the chat if you're doing 10 push-ups and 10 squats. 10 squats. Beautiful. So let's see. ET, well done, my friend. Stream. I'm going to— this is stage 2. Let's go to the RTX. So we're going to simultaneously train another model on the RTX 4090 because why not. But we're going to train it for a few more epochs. So over here we only did one epoch. On this one we're going to train it for 4 epochs and see how far we get. So we're going to do the same thing. Freeze the vision model. Done. Ashik, you've been coding for 36 hours. What the hell are you doing? Infinity GG is done. Good work, everyone. This is ML and Muscle. There we go. So we've frozen the vision module on here. 86 million parameters are frozen. Now, I had a good training recipe here before. Yeah, so no food or drink items are present in the image. That model is not what we want. We don't need LoRA. We're not— we're doing a full fine-tuning here except for division parameters. So here, yeah, we're going to do— let's go 256 mil v1. We'll see how we get, how far we get with 4 epochs. Now this is on an RTX 4090, so we should be, we should be racing along here. It's a faster GPU. That's my takeaway so far on the NVIDIA DGX. It is good for tinkering with AI things, but if you need some horsepower— and the memory allows us to run much bigger models— but if you need some horsepower, you're going to need a bigger model. Uh, sorry, a faster GPU. Such as— let's just make sure we've got the right dtype— such as an RTX 4090 or 5090 or even an RTX Pro 6000. That's what I'm going for. So yeah, there we go, frozen vision model. So over here, this took 32 minutes to train. And notice how the training loss continually went down. Let's see what happens with 256 million parameters. This is 500. Let's do it with small vlm2 256. So this is going over here on the RTX 4090. And this is going over here on the GB10. This is how we experiment. Hello, I am Brisbane too. Ben Lung. Hello everyone. Smoothie the Cat. Cool, so nice to hear someone from Brisbane as well. I mean, it's awesome that everyone's tuning in from wherever you are, but You kind of got to have a soft spot for your hometown, right? So now we're just going to sit back and wait for some models to train. We've got 2 minutes left over here, and we got about 20 minutes left over here. Now, if we— I know that the, the SmallVLM 500 mil works pretty well. That's a spoiler alert. So if neither of these models work pretty well, my goal is I'm definitely uploading whatever we get today to Hugging Face. And then I'm going to create a demo, cuz tomorrow I need to create a fine-tuning video, which will be interesting. Will the RTX 4090 catch up to the NVIDIA DGX Spark. Let's have a look at the Hugging Face models page. I haven't done that today. Always like looking up what new models are available. Neematron speech streaming, that's cool. MedGemma, that's a really nice one. There's a Kaggle competition Yeah, this is like a series of— Sheik, we're trying to do a bunch of things with local models, but I think it's about time we did another day in the life video, you know. Uni video, what is this? That's crazy. Cool. Oh, beautiful! So the training loss has gone significantly down. No, why did you do that, Spark? I might have to take a screenshot of this so we know that hopefully the checkpoint saved. That is annoying. My poor little Spark. Pushing that to the absolute limit. Okay, so the Spark is offline. I've, I've actually had a terrible time lately with the Spark. I don't know whether it's a connection issue. See, this is the problem, right? You could spend hours— okay, now it's not even— did it turn off? I don't know. The RTX is going good though, nearly finished its first epoch. Maybe I overheated it. I know, nice. Let's see how it goes on the next epoch. So where'd we get to? Did we get a checkpoint for that? Stage 2. Yes, we did. Lovely. Okay, thank goodness, because Where's stage 1? We halved our training loss. No, a third. Okay. I think we're going to have to restart it. Can we load in a pipeline? We need torch. Okay, we're going to need the checkpoint directory. Macaron, macaron, apple. Great. Okay, so our model works. How's our training going over here? Nearly finished a second epoch. Beautiful. So let's just output the content and zero and text. Bibua. What's bibua? Where did strawberry come from? Paprika. Chocolate cheesecake, marshmallow, mint. That's not correct. Okay, good. It's good at not food, that's for sure. Alright, so our model works well, but not well enough. So crème brûlée.

Sushi.

Maybe we also figure out a way in the future to fine-tune So I wanted to do food extract because that's a simple one to do, but we could replicate Apple's visual intelligence. So let's create a demo. So the input is obviously just going to be the image. I'd like to count how many tokens are going into a money, uh, model. I wonder if I could just retrain a model without any input prompt. That's an experiment next— is next. Extract foods without What did you find MLX to be faster on your test in training? Have I tried Claude Code or HF Skills to train models? No, I haven't. Have you? So we need given an input. So we need to format it as a message. What did you find MLX to be faster? Faster than what? Let's go. Our input's going to be an image, and then we need to do it as message template. Yeah, we can do some MLX streams in the future. Big fan of your online Torch course for, for an epoch in a range model doc train. Oh, thank you very much. So our model's going well over here. Training loss is going down.

Let's look up Gradio's docs.

So we need a pill image. So we want our pill image to come in here. Because in our app, we're going to have an original pipeline and a— or fine-tuned model. That's what we want. And then— Nice. Okay, let's see how this model went. Broth tofu. Hey, that's not bad. Fried noodles, red chili, green onion, fried noodles, sesame seeds, chili flakes. That's pretty good. Not food. That's what we want. Fried chicken wings, red sauce, fried chicken wing, fried chicken wing, water. It kind of got that. Not perfect. Not food. Bread, cucumber slice, green onion. Got that one wrong. Ravioli, sliced pork, spaghetti noodles, green peas, fried shallots. So it's at least outputting now structured data. It probably should have been Pad Thai. You're right. This is from the fine-tuned model. That's an idea we could do next. So it thinks that there's no food items in here. That's where our model goes wrong. So we need to train it on some more real-life images, that's for sure. Let's restart and go back to training. Small VLM 500 mil, and that'll be the model that we upload. Behind the scenes, I might actually make an upgraded vision dataset. With some real life images rather than just Food 101 images. Okay, let's run this and we'll test our demo over here.

Beautiful.

Let's try it on an image. Might resize the image, actually. What'd we get wrong here? We've lost our ability to view demos. Come on now. Eh, there we go. Not perfect, of course. But it's a good start. That's what we want, structured data. That's with our fine-tuned model. So let's put a label here.

A lot of description.

So this is a fully fine-tuned version. Well, not fully fine-tuned. I want to print the input prompt. Hello Daniel, have you already used a fine-tuned model with DSPY? No, I haven't. Beautiful. It doesn't get it right, of course, but we need to How do you escape tick in Markdown? In the next video, we might try extensions. Could be Beautiful, that's training nice and well. Oh, Daniel Bourke's book on fine-tuning, and now I'm finding ways to improve the performance of fine-tuned SLMs with DSPy. What's that book? I think he might have sent me that book actually. Is it a new book? I love this. 100% human writing. This looks like a great book. I should read it. He did send it to me. I'll have to check my emails, but I think I did get actually emailed a copy of this. Maybe we read it. We've just kind of been hacking around with, uh Trying it out. Whoa, I'm getting laggy. Is the stream dropping out? Let me know if you can still see me. I think my My internet is lagging. Testing 1, 2, 3, are we back? Oh, we just passed the second hour. You know what that means. Who's doing 10 push-ups and 10 squats? Let's do it. 10 push-ups, 10 squats, done. Okay, this is what we want to get to next. We've done the demo. Wow. A basic one. Oh, thank you very much, Grouse. Have you been watching them live or you're watching the replays? And if anyone's in the chat, can you tell me how the stream is going at the moment? I'm getting some warnings from YouTube that We're not going fast enough. After this model is done here, we're going to upload it to Hugging Face. Okay, apparently YouTube is back online. You branch, how are you? Oh, Stepfun releases a new model, VL. Let's try it out. See, this is what we have to go through when we're waiting for our models to train. Beautiful. Our model's nearly finished training. Okay, let's save our model. There we go. And now let's see how it performs just before we update it. Upload it, sorry. Soup. Okay, the model doesn't go too bad. But how does it go on our custom image? Yeah, it still outputs quite a fair bit from that image. I think this is where I'm going to write some notes on what we have to improve. Just watch your TensorFlow video, part 1, 5 hours done. Best tutorial we've ever had so far. That's very kind of you to say, so thank you so much. So our model's working okay, not, not as well as we'd like, but let's now upload it to Hugging Face. Okay, I think I need to authenticate my HF. Yo, Alex, what's going on? We got a small model working, albeit it's good at structure but not as good as I'd like at— um, so this is our demo. So this is a non-fine-tuned model, right? Original model. Now, if we upload an image like, say, this, right, the input prompt to the original model comes out with no food or drinks. And this is breakfast plate with fruit and rice topping, apple and carrot, pineapple, drink items, water, rice base. So not perfect, but at least it's starting to adhere to what we'd like. So if we try another image, where's something that's not of food? Maybe we just go. So now our model, if there is no food or drink, is outputting pretty well. There we go. So we tried to get the base model to output this, but it's not. So the fine-tuned model sticks to our structure, but we need to improve its perception abilities. So maybe that's a topic for a future stream, but for now I want to get something uploaded here, and I think I have to authenticate my RTX 4090 again. Does that make sense? Let's authenticate this. I'm just going to put my head only. Hey, it's my face! Just need to do some login stuff here, so that's why it's my face only. And we're back. Okay, we're uploading a model to Hugging Face now. Let's see if this model goes to Hugging Face. We've got our model here, and then we can create a demo. And try that out. So this is what's currently being uploaded. You gotta like your own models, right? Beautiful. So now we should be able to create a demo. I'll update the README soon too. So it's going to have to download the original model. Maybe I do this up here. There we go. Cucumber slices, chicken breasts. So not perfect. So we've got our model. Drosy, how are you? Big fan of your work. Thank you very much, that's so kind. Let's upload, or let's create a demo. The Spark crashed again. What is going on? Okay, good news is we can upload our demo quite well. Let's— I don't think we need anything else. We'll create a demo folder, food extract vision. So we got our app.py there. So we need a README. We'll update the README in a minute. Oh, maybe we have an example here. Let's upload our Food Vision Extract. There we go. Should be building now, but let's add a GPU to the background and then we can update Now if we look at the files— Raid mate. Let's have a look at the logs and see if we can get it working now. Downloadin', downloadin', downloadin'. Application startup. Beautiful. It's going to download the models. What do we get? We need torch vision. Getting close. This is another thing, I'm just waiting for applications to build. We got Torch. There we go. Oh, few more builds. I'll be right back. How's our app going? Please do a stream on PEFT. What? We've been doing everything. This whole, whole series has been PEFT. Well, this actually, we were on PEFT and then we've changed to be, uh, full fine-tune because we found that in the paper, a recent paper, that that was better than PEFT. For this specific use case. In the future, we can definitely look at PEFT. Here's our app. Beautiful. I was doing Laura and Q Laura all yesterday. Okay, our demo is live. Let's upload an image.

Error.

What we get here? I'm going to add some examples here too, by the way. I see. There we go. We need to create a folder of examples. Can we get a job at Nutrify as a junior engineer? If we were making money, I would consider it. We're not making enough money for, uh, hiring people. I'm just going to create some examples. Are you just streaming Monday to Friday? Just streaming whenever I really feel like it, to be honest. Obviously mostly through the week though. We're gonna use some tandoori chicken. I need them as a list of lists. And then we need one more. Uh, no, we're not hiring any free interns as well. I need one more example. There we go, that can do. So now we're uploading the examples. Let's see how this goes. Once we get this to work— oh, we're into the third hour. Who's ready to do 10 push-ups? And 10 squats. Put your hand up in the chat if you're joining in. 10 push-ups, 10 squats. Let's do it. Much better. Here we go. So we need to fix that. Beautiful. Next steps would be— so there we go. We did what we were saying. We did what we said we were going to do. How do you spell repetitive? Repetitive. Ah, Sam, hello! Samir, that is very kind to say. No need to pay me back. You being in this chat is, is I got a lot, I got a lot to learn from you, like doing everything from scratch, not using AI, learning from your mistakes. You've played a major role in my life. Please tell me how I can thank you for all. That's a very nice thing to say, but you being here, as I said, is more than enough. So another update that we should do is fix the repetitive generation. Asim, good news. The fine-tuning, we decided to step past, uh, QLoRA and LoRA. We are going to reserve that for when we fine-tune a larger model. Uh, we have now done food extract vision with small LM or VLM-2 500 mil. And we've got it to work quite well. It doesn't do too well on, say, food images that it hasn't seen before, but it outputs the structure. So we've got the first part almost down pat, as in getting to structured data. Next would be a step of improving the perception of the model. So getting real outputs from it. So we can do structured outputs now. But the— I guess finding out which food is in an image, it still needs more work there. Any advice for studying research papers? They're so dense. Practice. Lots and lots of practice. So I assume if we check this out, So we got the original model, not fine-tuned, outputs that fine-tuned model, we get that, which is beautiful. But then we put in an image like this and we get repetitive generation. So I need to figure out how to fix that. Yeah, you see there, need to fix that somehow. But If we do this one, boom, french fries and french fry. I'm a total beginner in deep learning. If you were starting DL zero from today, uh, if you want to start from zero, I go to learnpytorch.io and then go through the video there. I better put the email on there of the video, but there's a YouTube. I'd go to this video. If you want to learn deep learning from scratch, and then you can read a book alongside that. 3 million views, that's incredible. So that's what I do. And then, or you could read— there's the full course there on Zero to Mastery as well. This one. If you want to learn deep learning from zero, go through that. And then also go through Andrej Karpathy's video. So there you go. There's there's probably 3 months worth of work. Neural networks and deep learning, my video on PyTorch, and Andrej Karpathy's LLMs video series. But we have officially fine-tuned a VLM. That's pretty cool. We need to make it work better, but— so tomorrow I'm going to turn this into a tutorial. And we've got some notable extensions of what we want to do next. So I think they're the 3 big ones. I might do a— that's what we could be— we could do is queue Laura/Laura on Gemma 4B for having no input prompt requirement. So let's tick off some things. Tick, tick. We got our demo up here, and that is done, ladies and gentlemen. We have done it. Okay, so thank you everyone for tuning in. If there's any streams you'd like to see in the future, please leave a comment. Otherwise, you all have an amazing day. I've got to go take a walk, get some steps in, and I'll see you all in the next stream, which will probably be next week. So take care, and yeah, stay tuned for a video for a VLM fine-tuning tutorial. I'll refine all the code that we've been doing on the past streams this week, um, and I'll turn that into an end-to-end tutorial. So we'll see in the next one. Keep machine learning.