Local LLM fine-tuning on the NVIDIA DGX Spark - Part 1

Testing. Let's see if this works. I just need to make sure this live stream works. Testing. Let's see if this works. Okay, beautiful. Let me know if you can hear me in the chat. It's been a while since I've done a live stream, so going to be a bit rusty. This live stream works. Okay, I think I can hear the audio on my phone. That should be okay. So ladies and gentlemen, welcome to some local LLM fine-tuning. We want to— we We want to get an LLM, we want to fine-tune an LLM for a specific task. All right. So of course you can use all the big dogs, APIs and that sort of stuff, but maybe you want a little one. So let's do it in here. We're going to do supervised fine-tuning and let me make sure. There we go. Okay, beautiful. So we could use Google Colab, which I actually already have done in the past. But we're going to use, uh, I can't show you, but I have one of these NVIDIA DGX Spark. NVIDIA sent me one of these, so we're going to play around with it, see if it's any good. They actually didn't tell me what to say, so I'm just going to tell you if it's good or not. It's pretty expensive, it's $3,999 USD, but they kind of were so nice to send me one for free, so we're going to see if it's any good. So Google Colab, I've tested that this workflow works in Google Colab. Now we're going to test it if it works local. So I've set up the stream. Okay. I'm going to get better at these live streams too. So be sure to always put stuff in the chat of anything that you want to see. I want to do more live streaming in 2026. But don't tell anyone your goals, just do them. So get some LLM fine tuning code going. So we're going to do sft equals supervised fine-tuning, but we'll get to what that is later on. All right, um, today's goals: fine-tune a SLM, small language model. Okay, pop out the chat. Okay, yo, testing the chat. Where's everyone tuning in from? I'm from Brisbane, Australia. Cool, I can get the chat over here. I'm a real streamer now. Okay. So this is— see down here, I don't know if you can see that, but it's got SSH there. It's running on the NVIDIA DGX Spark. So if we do this, this is a segmentation model, EOMT, which is a just a transformer encoder that is a segmentation model. So it's actually quite fast at segmentation, which is really cool. So then here we go, we do it on another image, but this is not what we're doing. I'm just showing you that this code is running on— there's a photo of me and my wife. Wonderful. Okay, so segmentation model running on here. This is on the NVIDIA DGX Spark. Let's have a look. Iran, we got Iran, lovely. We got New York. Is it a big latency? What do you mean by that, Ondra? Is it— is the stream lagging? Is it— let me know if the stream's lagging. I haven't done one of these in a while, so we're just gonna— this is a working out the kinks stream until I get my setup fully done. But let's start a new notebook and we'll do it in— why not just in Code, right? Um, let's go fine-tune-llm.ipy. Okay, we could of course do it in a script, but I'm going to do it in— so we need a couple of libraries. Let's see if we've got Transformers. Maybe I zoom in a little bit more. We don't want ai for now. Import trl. Do we have trl? We don't have TRL. So TRL equals Transformers Reinforcement Learning. Okay, so this is the library we're going to be using for our fine-tuning. So Transformers is going to give us the model, TRL is going to help us do the fine-tuning. So let's go, um, TRL GitHub. There we go. I'm not signed into GitHub. Maybe I need to, um, one second. I'm just gonna sign into GitHub. I need to get used to doing credentials and stuff like that without my leaking my passwords to the public. That would be not fun. Okay, signed into GitHub. We can go back. Okay, there we go. So TRL is Transformers Reinforcement Learning. Okay. And so why would you want to fine-tune your own LLM? Because, well, for one, maybe you don't want an LLM. Maybe you want a small LLM, which is what we're going to work on. We're going to use Gemma 3 270 mil. So let's go into here and put that there. So markdown, I'm going to go Fine-tune a small language model. Why fine-tune? Because we want small model for a specific task, e.g., don't want to pay API credits or leak our data online, right? If we have our own model, we can run it anywhere and everywhere we like. Okay, so let's go here and we're going to go Gemma 3 270 mil instruction tuned. So this is the model we're going to do fine-tune, which is Google's Gemma 3, but it's a 270mm model. So if we go here, why won't— why would we want this? More zoom would be better. What about this? Is this a good level of zoom, this, this amount? Let me know. I am still training jiu-jitsu. I trained jiu-jitsu last night. We got greetings from Russia, Boston, Canada. What's going on? Nice to see everyone here. I trained jiu-jitsu last night. So, um, how to fine-tune an LLM model. We are going to do supervised fine-tuning, which is also known as SFT, which is Basically, SFT equals give samples of inputs and outputs. So for example, input equals hello, my name is Daniel Bourke. If our goal was to extract names then the output is Daniel Bourke. Does that make sense? Input, output. Okay, beautiful. But we need TRL. So let's install that. Conda activate ai, and then we're going to go uv pip install TRL. I've been using UV. Anyone use UV before? It's fun. It's fast. Okay, do we have— let's restart this notebook. Boom. We have TRL. That's wonderful. Okay, now a few things we need, because this is a machine learning cooking show. Maybe that's what I could stream, machine learning cooking show. Ingredients, we're going to go: 1, we need a model and we're going to use Gemma 3 270 mil. Uh, 2, we need a dataset and we're going to use a pre-baked dataset to extract food from text. What we're cooking, we're gonna make using PEB. What's up, brother? My brother just got here. We're in an office in Brisbane and we work together. But I'm gonna rudely be streaming while he's going to be trying to work. So what we're cooking is we're going to build a small LM, small language model, to extract food and drink items from text. Why? If we needed to extract— uh, needed to go over a large data set of image captions and filter them for food items. We could then use these filtered captions for a food app. In which case should we fine-tune an LLM instead of RAG? So let's write this here. It's a great question from WLWLA. Fine-tuning LLM versus RAG. So fine-tuning, you would want that, you would want fine-tuning when you have a very specific task, like a well-defined task of you could treat an LLM as a language processing model. So you might fine-tune something to do a very specific task, e.g., structured data extraction. Okay, so say for example, you had a set of emails or something like that and you wanted to— or receipts or something like that and you wanted to extract it to JSON, you would fine-tune an LLM for that. But for RAG, uh, you want to inject custom knowledge into an LLM. So let's write an example for this. So fine-tuning equals, um, an example would be you're an insurance company who gets 10,000 emails a day. And you want to extract structured data directly from these emails to JSON. And an example for RAG would be you're an insurance company wanting to send automatic, automatic responses to people but you want the responses to include information from your own docs. Does that make sense? So fine-tune, you have a very specific task that's defined and you just want to repeat that task 10,000 times a day. Or with RAG, if you are an insurance company and they send— someone sends you an email like, hey, my, um, my fence broke, right? You might send them an email back of information with links to your documentation, right? That's a very simple use case. But that's— whereas with the fine-tuning model, you might extract that email and go, okay, this person, their problem is the fence is broken, their address is this because they put in the form, etc., etc. So these are simple use cases, but that's a kind of an idea of which to use, which one and when. Now we've got our ingredients: model, dataset, 3, we need some training code, 4, eval code, and then 5, demo. So our method is going to be 1, download model, 2, um, download dataset, 3, inspect dataset, and then we're going to go 4, train model on dataset. 5, eval model. 6, launch, create an interactive demo. Okay, 7, the bonus can be make the demo public so people, other people can use it. Okay, welcome to Cooking with Machine Learning. So like all good cooking shows, I've got something that I've prepared earlier here. Now this is— I'm making a course on this, so just stay tuned for that. Oh, we need datasets as well. So download model, we use Transformers for that dataset. We use datasets, Hugging Face. Uh, all these videos, all these videos will be available after the stream. I'm going to— I really want to improve my YouTube stuff this year, so, um, I'll put it on just, uh, I think you can just watch livestream replays on YouTube, so it'll be there available. But then I want to take the best parts of the stream and turn that into an actual video. Not sure how to do that yet, but we're going to figure it out, aren't we? So train model on datasets, we're going to use Hugging Face TRL. Eval model, um, basically just look at a bunch of samples. And creating the interactive demo, we're going to use Hugging Face's Gradio. And then bonus, we can do this on Hugging Face Spaces. Okay, so, and I want all these tutorials to be end-to-end, like there's got to be something working at the end of it. You can only hear through one headphone. I'm so sorry about that. Hmm, I think that may be because I've had that in the past. I need to fix that. I'm going to fix that for the next stream. I'm sorry, but that's a good note. I'm going to write that down. I didn't test with headphones. Next stream, fix headphones, um, mic. So the speakers, fix headphone, fix mic so the sound comes through both headphones. Sorry, it doesn't work on this one. Next time I'll do my test on headphones as well as a speaker. Okay, so we need datasets as well. Beautiful. Okay, so we've got— we're starting to get our ingredients. Let's get our model. Over here. Oh, we could get our data first. Now I'll get the model. That's going to be more fun. So our model name— and by the way, this, this Google Colab over here, this is— I've already worked on this, so it's kind of cheating. We're not exactly doing it live, but oh well. Google, we want Gemma 3 270 mil IT. And I need to turn this into a code cell. Okay, now where does this model come from? Of course, it comes from our friends at Hugging Face. So Google, thank you so much. Model link here. Now, it doesn't really matter what model you use. You could— I'm just using Gemma 270 mil because this is quite a small model, 536 megabytes. There are other models out there like, um, Bayegettron model Hugging Face, which apparently works better. I just haven't tried it. It's a 321 million parameter model. Um, if we go have a look at this image, there we go. So this is MMLU, which is like a LLM benchmark. Gemma 3, 270 mils down here, but we don't exactly need an extremely knowledgeable LLM to do what we're doing. We just need one that's good at text processing, basically. Um, and then Quen 3, 600 mil parameters over here. And then they've released this one, Baguette Tron, uh, which is 321 mil parameters. We could even try Monad at a future point to do it 6x smaller. So that may be an experiment for a future stream, but for now we're going to stick with Gemma 3, 270 mil, because I know that that works. Okay, so let's get Torch and we'll go, um, device CUDA if torch.cuda is available. Else CPU, and then go print using_device device CPU. What? We don't have CUDA available? That's blasphemy, Python. Import torch. torch.cuda.is— false! What has happened here? We might have to reinstall PyTorch here. The last time I used this Spark, CUDA was available. So that's fun. So let's check out Torch version. So there's a problem already. But that's okay. Plus CPU. Why do I have— let's go to PyTorch's website. PyTorch. Linux. Now we'll install PyTorch. Did it work? Maybe you already had it. No. Yo yo yo, everyone joining the stream. Pip uninstall torch-torchvision. So I really thought that I had PyTorch installed with CUDA. Because we want to be on the GPU, don't we? Okay, we've got— for some reason PyTorch wasn't available originally. Well, CUDA, sorry. Thank God, that was a little bit easier than what I thought it was going to be. I thought we were going to be troubleshooting CUDA installations. CUDA, okay, we've got CUDA. Wonderful. So for those who don't know, CUDA is what you need to basically run your models on an NVIDIA GPU. If we don't do that, our models are going to go really slow. No, I'm not going to use Claude code. We're gonna, we're gonna be raw dogging this. We're gonna be coding all of this by hand, right? Because we want to learn. Once we've learned how to do it, then we can do Claude code. Before that, we need to start learning things ourselves, because if we know how to do it, then we can get AIs to do it for us, and then we can inspect it if something goes wrong. I always— does anyone get causal and casual mixed up? I always get those mixed up. Okay, so we're going to set up our model, and model is going to be equal to AutoModel.from_pretrained, and then we're going to do the model name. How do I connect, uh, NVIDIA GPU to Mac? So right now I'm doing SSH, so you can't see it from my screen. But I've got a Mac Mini and an NVIDIA Spark. Maybe I take a photo of this and I can show you. What are we building? We're fine-tuning an LLM. This is my setup. I'm just taking a photo of this. So this is what we got. I'm coding right here. So this is my Mac Mini, which is beautiful, but this is the NVIDIA DGX Spark. So right now I'm writing code on here, and but it's running on here. So see, there's that Spark, uh, FF62.local. Why do we do that? Because this has an inbuilt CUDA GPU. Whereas this is just running macOS. We don't— we can use it, we could use MPS, but, uh, I prefer to use CUDA for these type of workflows. So we're using it on here. We're fine-tuning a small LLM to do structured data extraction. So that's the— what we're going to do. I've got a little keynote here. Fine-tune a small LLM to do structured data extraction. So we come back to code. There we go. Model name, dtype, we're going to set that to auto, and we'll set device map equals auto as well, because device_map put the model on the GPU. Uh, attention implementation— now I'm going to use eager because, um, could use flash attention 2, which is a faster version, but ran into issues. So stick with Eager for now. Okay, let's see if our model loads. So there's the config. This is all going to download automatically from Hugging Face. So there we go, config model.safe tensors, which is downloading there. Okay, beautiful. So we've got our model. Let's see what it looks like. Model. Okay, so we got a geometry model. We have a bunch of layers here, attention, MLP, and then we have an output. So can we just feed it in model? Hello, my name is Daniel Bourke. No. So we need to do some scaffolding around the inputs, right? We get out some errors. Yeah, so problem number one is— who knows what our problem is? Our model requires numbers, aka tokens, as input. We can turn strings into tokens via a tokenizer. Woohoo! So let's get our tokenizer next. And we can do that with the AutoTokenizer method. So we'll just go tokenizer equals AutoTokenizer.from_pretrained, and then we're going to put in the model name. And then we're going to go print out some info. So model on device, model.device, and then print info. And we go model using dtype, model.dtype. So there's our tokenizer. Tokenizer is basically a big dictionary of— there we go. Mapping pieces of words to numerical numbers or numericals, uh, and that way our model can take it. So if we go tokenizer, hello, my name is Daniel Bourke, what happens? There we go, we get input IDs. And now can we just do model We put this in there. We might have to format this model with— or format this with something else as well. Input IDs must be tensor. Hmm. So I think we can go equals true. Hmm, so we want it to return it into tensors. PyTorch? No, we can look up the docs shortly, but why look up the docs when you can just hack around for a bit? So let's just turn this into torch.Tensor. Oh, we might be on here. We need to just put it to the GPU now, to CUDA. And we get another error. I think we might be because we need to use a chat template. Yeah, there we go. Size of tensor 4, non-singleton dimension. Maybe does it need a batch size? One more thing, we're going to try one more thing, and then if this doesn't work, we'll go to the actual docs. There we go! Oh, that is cool, we get an output! Have no idea what this output means, but that's okay. What do we get from that? Outputs, outputs.keys. Lojits. We get some Lojits. Okay, so we need a way to post-process these outputs. Okay, we're making progress here. We've already got, we've got a small LLM running on our own system. So we could just do a pipeline. Yeah, let's do a pipeline. Try the model with a pipeline. So pipeline is— I'm going to move this up here, actually. A pipeline in Hugging Face is the simplest way to get an LLM running or any model running. For that, really. But later on, we might want some higher levels of customization, so we won't use a pipeline. So we go load model and use it as a pipeline. So pipe equals pipeline text generation, and then the model is of course our model, which is Gemma 3 270 mil. And the tokenizer equals our tokenizer. Okay, and now let's check out what our pipe is. Beautiful. So let's put in some information to the pipeline. So we need an input. Prompt equals, and we want pipe tokenizer apply chat template. Now this is an interesting one, and we want our input text. So we saw what came out of that before. If we put, hi, my name is Daniel, we get some Lojits. But now we want— of course, we can't display Lojits to our customers. We need actual text. So let's put in this. We'll go input text tokenize equals false because we're going to have a look at what that looks like in a second. And then we can go add generation prompt true. Now I'm not sure why. What's the VS Code extension to get Python to just auto go to the right line? Let's check out what our input prompt looks like. Oh no, it didn't work. We get a template error. Hmm. Ah, this is because our dataset is not correct, is it? Let's get our dataset and have a look what that looks like. Get dataset. So this is a dataset that I have prepared earlier. You can go to my Hugging Face and see it there. Dataset equals load_dataset. That's basically the most important point in any machine learning pipeline is the models are good enough now that if you have your own dataset, they will work. So a lot of my time is spent on crafting the right datasets. I'm like programming the data for the model to learn. And this is a relatively small dataset, but it should be what we need. Number of samples in the dataset. And we're going to go len dataset_train. But we obviously need single commas there, don't we? So this is downloading from Hugging Face. We got food_extract. Uh, I haven't, I haven't been hands-on with the Mac Studio Ultra. No, we're running this from a Mac Mini to an NVIDIA DGX Spark. So there we go, number of samples in the dataset 1400. So we're going to go def get_random IDX dataset, and then we're going to go returns a random integer. Index based on the number of samples in the dataset. So this is one of my favorite things to do when exploring a new dataset, is to just view samples at random so you get an idea of what's going on in the dataset. We need random So let's get— okay, now Now let's view a random sample. Okay, so a lot going on here, but that's okay. This is when you're working with an LM or any kind of machine learning problem. This is often what your datasets will look like. They're just JSON or something like that, structured data. And we're going to do supervised fine-tuning. So we're going to have a series of inputs and some labels, which are our truth labels. And we want the LLM to produce the truth label based on the sequence. So Let me show you. So the sequence here, and our goal is we're building food extract. So I'll show you this dataset on Hugging Face. Hugging Face, right? So food extract dataset designed for fine-tuning a small LLM, e.g., Gemma 3. To extract structured data from text in a way that replicates a much larger LLM. So what I've done is I've got GPT-OSS-120B to synthetically label about 1,000 or so sequences of text, whether they be about food or not, and to extract different items, food items from that. Now, where would we use this? Well, in my case, building an app such as Nutrify, which is my brother and I's startup, we want to filter large databases for food images or food text. Whereas if you were a company, you might have a specific set of text and you want to extract structured data from that so that you can view it in a database. That's exactly what we're doing here. Now, the benefit of using an LLM for something like this is it can take basically any textual kind of input and then produce structured outputs from that. That's the versatility of an LLM. So if you want to read more about that, you can go there. But this is publicly available data. So our ground truth or our input is going to be our sequence. So example input equals random sample sequence and then our example output equals random sample. Now what I've done is the GPT-OSS-120B label is in JSON. So I want my model to take text input, and I'll just show you the raw output. And then go print info input equals example input, and then print info example output. Okay, so this is what we want our model to do. Say this is our input, this is a raw string from here to here. Now we want the output to be structured data. So this is JSON. So we can go from there, go import, we want our model to output JSON. Import JSON, and then I go JSON.load_s, and we can turn it into structured data. Oh, what's happened here? Um, one second, maybe we could do eval. There we go. So does that make sense? We want to take this input of text and output structured data like this. So we have is food or drink? Yes, it's about food or drink. So this is a hand is holding a glass jar with an orange lid containing a mustard seed-based condiment. The jar's label displays the following nutrition information. Here's the ingredients list. And so the structured data we've extracted from this is the food items: mustard seed-based condiment, mustard seed-based condiment, water, mustard seed, salt, sugar, turmeric, thyme, garlic extract, spice extract, peanut, wheat gluten, etc., etc. Also mentions potential allergens, and there's no drink items. So if we have a look at another example, we have on a rustic wooden share plate, thin ribbons of crisp cucumber and bright green apple are artfully arranged beside a stack of golden glazed donuts Dusted with powdered sugar. A small porcelain sugar holds a splash of amber mirin, etc., etc. And now we have this. It is about food or drink, food items. We have cucumber, apple, donuts. And so this is, this is almost a toy problem, but it is something that we would use at my company to extract large amounts of structured data from a large data database or something like that. Now you could do this for anything. You could do it for insurance claims. You could do it for if you're whatever type of finance business you want to extract company names from something. You can use a small LLM for this. So let's fine tune it to do so and we'll have a look at another random sample just in full. I have a condensed version of this. Now why are we doing a condensed version? So let's go example output. Condensed equals random_sample. And then we're going to go get the condensed version because the condensed version is less output tokens. So when we're fine-tuning an LLM, the less output tokens basically the better, because that is going to be our constraint is how fast— if we want it to be fast, if we can get it to output less tokens, then each token takes up generation time. If we can lower that, then we can speed up our whole process. So, um, all right, so maybe I put some new lines between these so we can see them. There we go. Okay, so online self-sufficient vegetable gardening course in a virtual classroom for one. Is this about food or drink? True. Food items? Vegetable. Drink items? No. Now the condensed item. Notice that the condensed item doesn't have any of the JSON formatting. So we're going to fine-tune our LLM to produce this, which is basically YAML, but then this is easily parsable into JSON, if that makes sense. How does the DGX Spark compare to the custom-built PC? So, um, that's what I'm going to be exploring in future videos. For now, uh, my intuition is that the custom-built PC is going to be faster because it's got an RTX 4090, whereas the DGX Spark is going to be better for basically a one-plug-in AI computer. And if you if you didn't want to build your own custom machine. And it also has much more VRAM, has 128GB of VRAM versus the RTX 4090, which is 24GB of VRAM. Hi from Turkey. Hello from Turkey. Hello from Brisbane. So that's our goal, is to take an input into an LLM and then get this as an output. Now I wonder if we can find— yeah, there we go. Now if we were filtering a large dataset of image captions, we might get some image captions that are just gibberish, basically. 1111111111.tiff or 1111111.jpg or whatever. And in this case, we want it to be food or drink, no tags, foods, drinks. Does that make sense? So this is not about— not about food or drink, not about food or drink. Here we go. A front shot on of Glossy and then we get food or drink. Foods, white yam, char siu pork, seaweed, red emperor fish. So that, that is a perfect example of what we want. Now let's see if our model can do this. What are we getting wrong here? Oh yes, we need to format our dataset. That's next. Format the dataset. Into LLM style inputs outputs. So right now we have examples of, um, string-based inputs and structured outputs. Uh, however, our LLMs generally want things in the format of— it's something like this. I'm going to get the exact format, but I'm trying to use my brain more. So we want user is going to be, hello, my name is Daniel. And then system will be, hi Daniel, I'm an LLM. Right. So we just need it in that type of format. In other words, they want structure around the inputs and outputs rather than just raw information. Okay. Where the system in this case is the model. And where can we find this? Well, that's in the docs, Hugging Face docs. I think I've got a link here somewhere in my, my pre-made notebook. Yeah, here we go. Find different style datasets. So if we go to TRL docs, let's have a look at this. I want to show you the different types of dataset formats. This is very important. Dataset formats and types. I want to put that in here. Oh, what did I do there? Dataset formats and types. Now, if you want to train any of your own LLMs, you should get familiar with this dataset. Sorry, if you want to fine-tune them, you can train them from scratch. A little bit of a different process. But if you want to fine-tune them to do specific tasks, get aware of the dataset formats. And that's often a lot of bit of a lot of the onboarding with any kind of machine learning model is getting your data in the right format. So if you want a custom model, you need to figure out what format you want that data in. So actually, I'll keep this full screen. Green Code, yo, what's up? Yeah, I've heard that the memory bandwidth on the Spark is a disaster. That's this chat comment here from, uh, Madrid Prado. I'm not— sorry, that name is too long for me to try and pronounce. It take me longer. Um, so let's find out that together, hey? Yeah, that's a good comment. Custom PC, if you have a modern GPU, will be much faster at inference. DGX Spark is better at local experiments like the one Daniel is showing. Yeah, so I think that this is a great comment. Can I make this bigger? Yeah, I agree with that. How do I— I can't react to a message on YouTube. Anyway, so that's what we're doing. We're doing a local experiment. That is probably the nutshell. I think the Spark would be great for getting started with things, but I just looked up this bad boy, RTX Pro 6000, and this looks freaking awesome. I mean, it's $14,000 Australian dollars, but that's the, that's the next upgrade I would do. So that's what the goal of this year's stream is, is to get one of these for free somehow, whether NVIDIA send me one or we just get enough revenue from the stream and we buy one of these GPUs. Um, that's, that's the goal, right? And then we can publish much more open source models, but we've got enough to do, to do for now. Anyway, back to dataset formats. So the format of a dataset refers to how the structure— the data is structured. Typically, you'll categorize as either standard or conversational. The type is associated with the specific task the dataset is designed for, such as prompt-only or preference. Each type is characterized by its columns, which vary according to the task shown in the table. Now, a lot of jargon there. I understand it via just inputs and outputs. What inputs do you have? What outputs do you want? That is my Every machine learning problem you get, just what inputs do you have? What outputs do you want? So we want— I believe ours is going to be— is it language modeling or standard? I think, yeah, we want this. We want ours to just reply with language modeling. So we're going to have some text. And then we want our assistant to output some content. I think this is the style we want. So in our case, our content from the user is going to be just a text string and then the content from the model is going to be the formatted structured data that we want to extract. So that is what we want. We want conversational. I believe we're going to see this in practice, so I may be wrong or even just prompt only. No, we don't want that. Prompt completion. Preference. No, I think we want language modeling. Let's, let's go back and we'll just try. So yeah, sample to conversation. Now I wrote this notebook probably 2 months ago, but then I got married and went on holidays, so I'm just getting myself really familiar with it. And then we want messages, and then we're going here, role user, right? So this is what we're going to input. Content sample is going to be our sequence. So remember, LLMs are just tokens in, tokens out. That's all you got to— that's all your framework should be for for modeling stuff. Tokens in, tokens out. So load the sequence from the dataset. And then up next we're going to go role is going to be— this is the system. And then the content is going to be what our labels. So this is GPT. Now I've synthetically labeled all of this with GPT-OS-S120B because it's basically one of the best open source models that are available. So all of this is open source, by the way. Everything I'm working on here, completely open source. The model, the dataset, the samples synthetically generated. So this is the beauty of working with AI in 2026, is that not only can you generate code, you can generate data samples, which obviously— which in the past had been the biggest bottleneck to creating your own models. So now with Hugging Face datasets, let's just see how this works. For example, um, we have our random sample, which is this. Let me get the YouTube chat back up. There we go. I agree with this statement. NVIDIA, can you please send Daniel a Pro 6000 so we can watch him experiment with it? What? NVIDIA developer in here? Yo, okay, I'll— if I do NVIDIA developer, here's the rule: if I do over 100 hours of streaming with, uh, the Pro 6000, then I can keep it. So that's, that's my, that's my deal. I'll do 100 hours of streaming of development all on the Pro 6000. That's my sale. Okay, so now let's get our random sample back. And we go here, a front-on shot of a glossy. So this is a lot, right? And what I've done is I've got image captions and whatnot, UUIDs, all of this is open source. But our main goal here is structured data extraction, text to JSON. So now, but we need it in conversation style. So Sample to compensation. Random sample is role user. Oh, we have a deal! There's, uh, we've seen that. Everyone's seen that in the chat. Let it be known, NVIDIA developer said that's a deal to Pro 6,000. 100 hours of live streaming. Of local AI development. Okay. And we'll get the NemoTron models running and we'll fine-tune some of our own and we'll publish them to, to Hugging Face and then people can benefit from all the work that we do together. So that's pretty cool. But we need it. We keep getting distracted. We need to get this data sample ready. So now all we've got here now is an input and an output messages, role, user. So if you went to ChatGPT, and typed in this, there's a lot there, right? This is a big image caption. So let's just go to Chad, ChatGPT, and we go type in this, right? This is our input. This is all ChatGPT is doing behind the scenes, except it's a much bigger model, right? And I'm paying for this. So let's go. Please extract the following. Please extract the food items and the drink items from this text to JSON text. Right. So this is ChatGPT is doing this, but we want to build a small LLM that can just do it ourselves. So ChatGPT is good at that, we know it's going to be good at that, but right now our little model probably isn't that good. And then what do we get? This is our label, food or drink. Tags, we've got some tags as well: white yam, char siu pork, red emperor fish. What do we get? We get the basically the same output as ChatGPT. Okay. White wine vinegar. Oh, did we get white wine vinegar? We said that was a food item. So that would be maybe a sample where we could improve our— sorry, improve our labels to use white wine vinegar as a drink. But that would be another discussion. Is it a drink or is it a food item? So from one angle, are you going to drink white wine vinegar? Probably not. Are you going to use it in a food ingredient recipe? You probably will. So that's up to you. We're not going to argue about that. We're going to figure out how to train a model to do this. So now we've got a little function to map our samples to conversation style, which again is our data format over here. Now we want to map it to our whole dataset. So let's go map our um, sample_to_conversation_function_to_dataset, and we're going to go dataset, dataset.map, and we're going to do batched equals false. We don't actually need to do this fast because it's going to dataset. And now let's have a look at train because it's, it's our mapping. And I want maybe sample 0. Okay, so now we have our messages and our dataset sample. Beautiful. So next, let's create a train test split. And we're going to go dataset equals dataset train, train test split. And then we're going to go test size equals 0.2, shuffle equals false. And then we're going to go seed equals 42, right? Because 42 is the magic number of the world. And then if we have a look at our dataset now, we have a train and test set. So number one rule— what's the number one rule in machine learning? Always train on the train set and test on the test set. So whenever you see benchmarks of LLMs come out, Often these days, why don't we go to Gemini 3 release blog. So obviously the Gemini 3 models are really good, but if we go to the benchmarks here, this is why we do train and test set. So all of these you would hope are on the results are on the test version of these benchmarks. All right. There's a lot of benchmarks here, but what you can do to hack the benchmarks is create data like the benchmarks, train on that, and then when you evaluate, it's kind of inflating all these numbers. So it's really important to not only test models on benchmarks— these are a good overview— but create your own test sets on your own datasets so that you know when your model performs well on your own dataset, that if you ship that into production, it's going to work pretty well. So like for this one, for example, AIML 2025, that's mathematic problems. So what can happen with these models is they can get really good at solving math problems. Yeah, sure. That's a great— I've never actually seen a use case for LLM solving a math problem. I'm not a mathematician, but if you have lots of example math problems out there, such as the benchmark itself, because these models are trained on the entire internet, potentially examples from that test are in the training data. So number one rule of machine learning, when you're evaluating your model, always evaluate on samples the model hasn't seen, because that is a— not, not a true indicator, but an indicator of how your model might perform in the real world. So we've got a train and test set. This gives us an indication of how will perform in the real world. Okay, what's next? So we've got a train dataset. Can we pass one of these samples into our model yet? I'm getting sick of this, of not passing a sample into our model. So let's just go— I want to put this conversation format and just see what our model says. To "Hi, my name is Daniel." Let's go easy sample. Is this messages? And I want to go content. Hi, my name is Daniel. Oh, actually, maybe that's why it's not working. Input text, add chat template. Now let's get this working. There we go. Okay, so now we've got a way to pass it to our model. Now let's see what our model says to 'Hi, my name is Daniel.' Is it going to say 'Hi' back? Is it a nice model? So we formatted our dataset. Default options, pipe input prompt. Okay, this is the input prompt. Excuse me. So this is our input prompt. What do we got? Hello. Oh, fine-tuning. We're just doing it for fun. MLX. We could do an MLX stream in the future too. That's going to be the, the year. This is the year of local AI. So we're gonna, we're gonna spend this year creating our own models and publishing to Hugging Face for all different kinds of things. Apple devs do fine-tuning stuff. Yeah, they do for sure. We're at a point now where the software and the hardware, like the NVIDIA DGX Spark and the, um, Mac M4s and whatnot, and all the tooling from Hugging Face and whatnot, is getting to a point where there's never been a better time to get into creating your own AIs and, and building them for small things. Benefits of that is you're not sending your data to some big company It's running all on your own hardware. You don't need an internet connection for a lot of different things. And imagine all the use cases where AI like needs to run but with no internet connection. For example, a Tesla self-driving car that can't send data to the cloud because you need it to operate fast. So this year we're going to be learning about how to run all on local AI. That's, that's what I like. Obviously we could do some streams with APIs and stuff like that, but I like running small models and custom small models. So let's go next. This is what we're going to put our input prompt to our model. Now you notice there's some weird things here like BOS, start of turn, user, end of turn, start of turn, model. So this is our conversation type. Okay. So when we put into our model, this BOS is beginning of sentence. So that's— it's going to say, hey, the prompt's starting. So this is the type of data these LLMs have been trained on. Start of turn user. So that's me, the user is putting in this text. And then this is the end of my turn. So these are, these are special tokens, right? So the model has been trained to understand these kind of special terms, tokens. And then we have another token there, which is start of turn for the model. And so the goal of the model now is to go, hmm, okay, given this input, what should I produce as my output? That's exactly what's happening when you use any of these LLMs, ChatGPT, it's doing something like this behind the scene. So let's do the default options. Default outputs equals pipe input prompt. And we're going the long way here, right? I could just get some AI generator to make all of this, but we're going the long way here so that we understand each part of the process. And then later on we can get some AI to just write the whole thing for us. But then if something goes wrong, we can go back and go, hmm, you shouldn't have put that token there. I'm going to fix it. So the default options, let's see the default output. Sorry, default outputs. Hi, my name is Daniel. We got some generated text. And then, hi Daniel, I'm here to help. How can I assist you today? Hi, my name is Daniel. Please reply to me with a machine learning poem. Right, this is running locally on the Spark. This is a small 270 mil parameter model. Look how quick that is. That's incredible. Oh, sorry, I've got the easy sample here. Um, def createEasySample, input is going to be return template equals. And then let's create an easy sample, easy sample. So Look at that, under a second. Please reply to me with a machine learning poem. Okay, so this model probably can't write poems because it's 270,000 parameters, but let's get the generated outputs from this. So we need default outputs and then maybe we go generated text. And then maybe we go, uh, the input prompt and beyond. Excuse me. Generated text. What do we see from this? Ah, that's right, we need— there we go, that is cool. So we just got our first generation from a local LLM. Look at that. Okay, let's read this. My input, by the way, was here. Input text. This is a 270 mil, 500 megabyte LLM file input. Now the poem, I don't care if it's any good. I just care if it can do it. A few years ago, this was impossible to do. Print— I'm just going to do from model name. Maybe we get a new line. It's very important to make things pretty. Let's read the poem. Hi, my name is Daniel. Please reply to me with a machine learning poem. Okay, Daniel, I'm ready. Let's create a machine learning poem. The data streams, a tangled web of patterns, a subtle, silent shrug. Each data point, a question we ask to learn and to grow, to find the right way. From simple data, a seed takes its hold, a learning engine, brave and bold, with algorithms, a clever design to sift the noise, to find the divine. The world of data, a vast and unknown, Where insights bloom, a future we've sown, and in each pattern a new potential bright to make the future a guiding light. At least that rhymes. So let the code begin with gentle grace, a symphony of data, time and space, a machine learning, a helpful quest to understand, to connect, to put to rest. Not, not bad. Like, not great, but not terrible. I don't mind it, right? So that is cool. Now we want to fine tune it to do our certain task. And can we do that? Of course we can. So let's now see what it will do given an input of what we give it. Sorry, of one of our samples. So let's take our example sample, our dataset, and we'll go get random index. Where's my function? So try the model on one of our sequences. What do we have here? We've got the model writing a poem now. Now we need our random sample. And has that been mapped? No, it hasn't yet. And so we'll go get, and then we'll go random_train_sample equals Easy low carb cinnamon roll mousse. Okay, and so what we're going to do is we're going to apply the chat template to messages. So we'll go input prompt and we'll go pipe tokenizer.apply chat template, and the conversation is going to be, um, random train sample, and then we want messages and we want one, I believe, just the first one. We don't want the— or up to one, right? Because if we have a look at what this is— yeah, there we go. That's what we want as our input. So, um, we just want the text because we don't want the model to have the label that we want it to output. So given this message We want our model to produce this content, but let's see what our model does by default. Remember, this is the base model, hasn't been fine-tuned yet. So if we have a look at our input prompt, right, there we go. And we can go tokenize equals false. There we go. So because we're getting a random input every time, this will be random. So I've specifically made this dataset to be full of all different kinds of text. Beautiful. Okay, so now let's run the default model on our input. So input prompt. No, we want default outputs. Equals, uh, pipe, and then we want to go text_inputs equals input_prompt, and then we go max_new_tokens equals 256, default_outputs There we go. So what is this? We get the generated text. Let's get zero, and then we want generated text, and then we want, um, This is going to be a random one every time, by the way. Onwards. Is that the right indexing? Let's see. Excuse me, we might have to go add generation. Prompt equals true. This is a beautiful appetizing picture. Okay, we are, we are rolling here. We are getting, get a random sample. And now view and compare the outputs. So we want to go print input, which is going to be the input prompt. Want a new line. We want things to look pretty, right? And then we're going to get the output. Output is going to be newline, and then we want this. Random. There is actually a My Little Pony in Japan. Yes, there is a My Little Pony movie-themed cafe in Japan. Right. So right now, our— the model is just basically replying to our inputs. And with the cruel work. Okay, I'm ready. Let's dive in the world of cruel work. I'm excited to learn more about this fascinating field. What are you hoping to learn or explore? Right. So Our models, what do we want it to do? Structured data extraction. But of course it's just doing what it's, what it's meant to do. This is a great description of the rustic wooden board, a top-down view of a rustic wooden board. Now, how might we turn this into a food extraction? So could we do it via prompting? Let's try. So this is the default outputs. We take in some text and the model replies. But what are we trying to do? We're trying to do fine-tuning to structured output. Let's try to prompt it because that should be your order, order of operation. Default and then try to prompt the model. Remember, this is only a Gemma 2, Gemma 3, 270 mil. I'm using a small model on purpose. M5 Ultras, they'll be, they'll be cool to see. That's what we've got going on in the chat. No doubt kids in the '80s and '90s had Porsche pics in the room, but today my dream machine is a Mac Studio M5 Ultra. Apple have been really stepping up their game with their Macs, haven't they? So let's try to prompt the model. So we want a model to extract food and drink items from text. By default, the model will just reply to any text input. However, we can try and get our ideal outputs via prompting. So to do that, we are going to take our random sample and we'll inject a little— maybe here in the messages, we'll inject a little instruction of what to do. So here, random sample. But now let's go prompt instruction equals— maybe we do it like this. How could we do it? Given the following text from an image caption, please extract the food and drink items to a list, to a Python list. If there are no food or drink items, return an empty list. Does that make sense? Now we'll give it an example. For example, input text: um, hello, my name is Daniel Bourke. Output: food items, drink items. Example 2: Input text is, um, let's just write what I had for lunch or breakfast. Uh, plate of rice cakes, salmon, cottage cheese, and small cherry tomatoes. With a cup of tea. So the output here is going to be food items, drink items. Okay, so there we go. And now target input text, and then we can go input text, given the following target input text. So you see what we're doing here? We're just giving it a little instruction, and we're trying to get our structured data format out from a prompt. Before, it was just replying with a default output. But now we're actually giving it some structure around it. So how do we turn— how do we get our— maybe we don't need a random sample here. Let's inject our input text, or our instruction, sorry. So the conversation is going to be this. So we want to change this content to that. So content. Let's go def update_input_message_content, and we'll go template_string equals— we'll go prompt instruction. We don't need to do that. And we'll go input. So we'll go original content. So the input will be something like this. Original content will be input, and then new content will be prompt instruction, prompt instruction.replace, and we're going to replace the target input text. With the original content. And we'll see if this works, obviously. And go return new input equals Oh, actually we need original input equals input.copy. This is going to be— we could do this better, but I'm just making it so that it's, it's really visible. Original input, and then we're going to go new input equals— oh, we don't actually need— we could just recreate the input. It's actually not that hard to create, is it? I'll show you what I'm doing in a second. I'm just, just coding it out first. If in doubt, code it out. So new input is going to be list content, new content role is going to be user. Okay. Return new input. Now let's bring our prompt instruction back up here. And we're going to go— that's going to be the original. So print original content. We're gonna need a single one here. Okay, and now let's try our helper function to inject it with some prompt update. Yes, now we've got our instruction there. How good's that? Info: new content with instructions in prompt. There we go. So you see what we've done there? The original content is image caption, but now the new content is an example prompt. So we're basically leveling up the amount of information we're passing to our LLM. So now let's try this with the new content there. What's the chat saying? For me, would you want a fast inference or fine-tuning? For me, if I want a general model, I'll probably just generally go to the, the ChatGPT, the best, the the best, uh, API. But if I want a fine-tuned model, I'm going to be wanting to basically just run that locally, do it myself. And for that, I'll want an NVIDIA GPU. So do I want to be running DeepSeq R1 locally? Not necessarily. Do I want to be just interacting with an API for a chat model? Sure, that's the easiest way. But if I want a model to do a very specific use case, I'll probably fine-tune that. And I'll start in levels of abstraction. First, I'll try an API. Second, I'll go, okay, I want to replicate this workflow locally, and then I want to fine-tune a model to do that for me. And then again, it all depends on, I guess, the companies you're working with. Is the data private? Can it even use an internet connection? That's a big one, right? We're all used to having an internet connection, but there's a lot of devices out there that need to operate without an internet connection. So I actually was just working on a computer vision, computer vision problem with a company who needed to do computer vision in the field without internet connection. So the, the models had to run on a mobile phone. So that's a very limited use case, but certainly achievable. It just takes a fair bit of work, right? You need to create the dataset, you need to test the model, you need to quantize it, you need to understand that it works on device. That, that's where I have the most fun. Okay, let's see how our new prompt goes with our model. Can our model by default do our task? We want it to extract food and drink items from a list. So— oh wait, my, my template is wrong. So the food items would be here. Rice cakes, salmon, Cottage cheese, cherry tomatoes, cup of tea. I wonder if we should punctuate this, make it a real Python list. There we go. Okay, so we want to updated input prompt equals, we'll take our updated input prompt. And then we're going to go input equals random_train_sample. Put that there. And then go boom. What did we get wrong? Oh no, we can just directly go like that. See, so the output Hmm. The outputs outputting. Yeah. So see, okay, this is a great example. This is in machine learning, errors are actually important because then you know what to do or what to do next. You can fix the errors. So you basically need to just run a whole bunch of experiments. So what, what we've done here with the small model is I've said extract the food and drink items to a Python list. But what the model, because it's only a small model, 270 mil, it has written a Python function to extract the food and drink items. But obviously this function is not going to work because it's not working semantically and you don't know all the food and drink items, uh, ahead of time. So what can we do from this? We can update it to go maybe just to a list instead of Python. Python, I think, is triggering the LLM. To a list. Return in the following format: food items, item 1, item 2, item 3. Drink items, item 1, item 4, item 5. Let's see if this works. So we've changed the instructions there. We're doing prompt engineering on a 270 mil model. Okay, so it's still doing Python, right? So This is the exact use case we'd like to fine-tune our model. So it is a small Python model— uh, sorry, small LLM— so it doesn't really understand the nuance of what we're trying to do. So let's put this into ChatGPT and see if it does what we want it to do. I would assume it would. See, does it exactly right. But this is probably a trillion token, uh, trillion parameter model. So let's fine-tune. We'll go down here, write it there and go. Okay, looks like our small LLM doesn't do what we want it to do. No matter, we can fine-tune it so it does our specific task. Okay, who thinks we can fine-tune it to get it working? Let's go here. Steps. So the steps to fine-tune our model: one, we need to setup SFT config, which is supervised fine-tuning config. Actually, before we do that, let's see what the ideal output here would have been. We've already got the ground truth, so our current error is that our model is basically generating Python code when we ask it to extract a list, which is— I get where it's going with that, but it's not what we want. So we have our random train sample. We want in an ideal world our model to output this here. So if we go messages and then if we go 0 and then if we go 1 and then if we go— nope, we want 1. So content. So this is our ideal output: peeled white yams, deep red gochujang, and golden brown empanadas. Now, have we got butter? So GPT-5.2 has done better there than on the ground truth, but that's okay. It's picked up the black pepper. This one hasn't picked up the black pepper, but that's okay. Set up SST config, use TRL trainer. Sorry, sft-trainer to train our model on our supervised samples. Okay, so then once we fine-tuned a model to go, hey, given our text input, we want this to be This is our input. People keep saying the cray one. What's the cray one? Oh, 1970s, $7 to $10 million. Yeah, whereas now we've probably got more computing power on our desk. Let's just get How much compute power did the Cray-1 have? Compare this to the power in a NVIDIA RTX 4090 and then compare the pricing of the two. Holy smokes! To put this into perspective, if the Cray-1 were a person— so this is the 1976 supercomputer— were as fast as a person walking 3 miles per hour, an NVIDIA RTX 4090 would be traveling at 1.5 million miles per hour, fast enough to reach the moon in about 9 minutes. That's nearly the speed of light. So half a million times the compute power at roughly 0.005 of the cost, adjusted for inflation. That is insane. And that's only an RTX 4090, that's not like a 5090 or something like that. So if the Cray-1 is a walking person, which is a supercomputer in 1976, the RTX 4090, which I have sitting over there behind me, is the speed of light. That's incredible. Okay. Sft-config. So we're going to use— this is TRL, so Transformers Reinforcement Library. sft-config is— where is it? There we go— is basically the settings for our fine-tuning supervised. And then sft-trainer. Okay, so let's set up— setting up our SFT config. So we'll go back to our notebook like we prepared earlier, like all good cooking shows. Do I have TrackIO here? I don't have TrackIO. That's okay, we don't need to— we can watch just the outputs here. So from trl import SFTConfig, and then we're going to go torch.dtype equals model.dtype. Oh, and thanks Madrid for tuning in. You're going to bed, it's nearly 2 AM. All the best, my friend, have a good sleep, and I'll see you in the next stream. Hey, thanks for tuning in. Info. Yeah, what learning rate should we use? Of course, we could experiment with this. What did I set this to be? Base learning rate, 5e So quite small because our model has already trained on a lot of text and tokens. Now we want it to perform a very specific task, so we don't necessarily need it to be incredibly— we don't need it to update its parameters too much. It's going to have a really good base knowledge of language and different stuff. So we want a checkpoint directory name, and that's going to just output it over here. Fine-tune LLM, we can just call it something basic, checkpoint models, and then We can create a demo later on, but let's go to our sft-config. So config equals sft-config, and then we're going to go output_dir equals checkpoint_dir_name, max_length equals 512, the maximum number of length of tokens that our model, we want it to output. Packing equals false. So packing, I believe, is going to put multiple samples into one. So then, because our models are just next token predictors, right? So it almost doesn't really matter what order the tokens come in as long as it knows that it needs to predict the certain next tokens given a certain input. Number of training epochs, we're going to see how far we can get in 3 epochs. So we don't— we shouldn't need too much. Um, per device train batch size, maybe because I have a lot of VRAM on the NVIDIA DGX Spark, I'm going to use maybe 16 batch size. Let's see how that goes. Gradient checkpointing false. Optim equals, um, AdamW. Torch fused. And then, yeah, I tried to use— I've got some notes here for myself. This is my own notebook that I've been hacking around on. If you try AdamW, you'll get an error. Well, at least in my experience. So if you try AdamW, you will get an error. And so we use the fused version. And then logging steps, how often do we want to log, uh, save strategy equals epoch. And then eval strategy equals epoch. Learning rate is going to be our base learning rate. And then fp16 equals true. If torch.dtype equals torch.float16, else false. So this is telling our model we want to use FP16. Um, the DGX Spark can actually use, uh, FP4, which is very low, but I'm not sure. So here's the problem: the hardware has the capability of using, um, FP4, which is floating point precision 4, basically Default is 32. 16 is half that. 8 is half 16. 4 is half of 8. So it's just basically a way to save. It's called quantization, basically a way to save on memory. So if, for example, you're running a model on a phone, you could train it in float16, and that'll output a model that's 100 megabytes. But then you could quantize it to FP8, and then it'll output a 50 megabyte model, so half. Now there are some trade-offs there in that generally if you quantize a model unless it's been trained in that quantization, you will get some performance degradation, but, uh, it isn't always that dramatic. So you could train a model in— it's very common to train a model in FP32 or FP16, um, and then deploy. Usually these days FP16 is basically the default because we've kind of worked out, okay, FP16 you just get the benefits of both worlds. But now in the last couple of years, FP8 and FP4 has been significantly improved. But we're kind of at that stage where we're waiting for software and hardware to synchronize to make them all, I guess, talk to each other and make it much more reliable. So even though the hardware on the NVIDIA DJI Spark can do the FP4, the software hasn't really caught up 100% in my experience. Now that may change, um, and I may be wrong, but if I am, please let me know. Torch D-type. So, LR So learning rate scheduler. So when you're training, sometimes you want the learning rate to go down as you get closer to convergence. You want the learning rate to go down. Push to hub. So we can upload our model directly to the Hugging Face hub. We want to do that later, maybe either today or in tomorrow's stream. And, and that means that other people can use it. So report to, uh, I'm going to set this to none. So let's go to the docs for this. Always look at the docs. It's opening in Safari, that's okay. Report to— so TrackIO is an open source version of Weights and Biases. Yeah, none. I wonder if I could just set that to none, or is that going to default to Weights and Biases? So I'll show you TrackIO. There we go. So very, like, a lot of iterations here. TrackIO, import TrackIO as Weights and Biases. And we can just, yeah, start to track different things about our model. So we can just track it as a dictionary. That's going to be something we look at later as well, is tracking our experiments. But for now, we just focus on getting straightforward fine-tuning. So I think that should be enough for a config. Now again, we could tweak a lot of these. We could tweak the max length, tweak the epochs, tweak the batch size. Um, does there eval batch size? We'll eval on 16 as well. So note, you can change this depending on the amount of VRAM your GPU has. Okay, so let's check in with the chat before we go to the next section. Hi, do I use Databricks? I don't use Databricks. It's always 4 AM my clock, so I have to sleep. Have a good stream. We'll end it later. Thank you very much for tuning in. WLW LLA. Where would the NLP world be without Hugging Face? This company rocks and makes it so easy to use the largest language models. I totally agree. Hugging Face— between Hugging Face, NVIDIA, and Apple, of course, for making epic Macs and stuff like that— Hugging Face and NVIDIA, probably my two most used companies at the moment in terms of what they offer. And then of course, Google with Gemini and YouTube. So very, very grateful to be a machine learning engineer in 2026. So we have our config. Now what do we need next? We need our step 2, we need an SFT trainer. Now, if you're in the chat as well, and it's really early where you are, thank you for tuning in. Or if it's really late, I'm in Australia. So I think our timezone is basically all over the place compared to the rest of the world. For future streams, I'll probably try and get a better, um, I guess, time schedule for everyone. So yeah, if people just put in where they're watching from or what time it is, that way I'll get a better idea of where people are tuning in. I guess YouTube will provide this, but I actually like talking to people. So, uh, maybe it's best for me to start early in the morning in Australia, like at 6 AM for Australia. I do it later at night, like I start at 8 PM or something like that. I personally prefer the morning. So maybe that's where it's at. So let's go next, we want our trainer. So from trl import SFTT Trainer. So create trainer objects. So SFTT Trainer is going to use Transformers trainer under the hood. But SFT, SFT stands for supervised fine-tuning. Equal provide input and desired output samples. Create the trainer object. So we can do trainer equals SFTT trainer, model equals model, args equals SFT config, and then train dataset equals dataset train eval dataset. Test, processing_class equals tokenizer. Right. So if we have done this all correctly— have I missed a comma somewhere? Yeah, there. If we have done this all correctly, we should be able to train a model after this. Now it may take Okay, so we get people over the world. So 4 AM, 1:55 in Germany, nearly lunch. 7:55 PM in New York. Cool. Hey, Alex, for all the way from New York. I've been to New York. I love New York. So trainer.train. Who's ready to train our first small— this is pretty cool. We're 2 hours into the stream. And we're going to train our first um, small language model, Gemma 3, 270 mil, to do a specific task. So we want it to extract food. We've seen how it's performed before. Now, I haven't trained— I've done inference on the NVIDIA DGX Spark. I haven't trained a model. So this is going to be my first model training on the NVIDIA DGX Spark 2. So hopefully it works. You ready? 3, 2, 1, let's train a model. Okay, there we go. We might— 3 minutes. Is it gonna run in 3 minutes? I can hear the fan going off. I don't know if you'll be able to hear this, so that very quiet faint noise is the NVIDIA DGX Spark training. That is cool. Okay, so 3 minutes apparently it's going to take. Now we can go watch this happen. So watch n1-nvidia-smi. There we go. So gpuutil, 93%. That's a good amount. So we're using most of the GPU. And we'll watch this train. Okay, so one of the things, one of the rules of this Machine Learning Cooking Show is that every time we train a model, we have to do 10 push-ups. Okay. That could be my series, is machine learning and muscle. Oh, buenas tardes! Hello, hello from Argentina! Hello from Turkey! Oh, look at that, accuracy is already to 60%. 10 push-ups. Okay, after this model is finished, we're going to try it out, and then I'm going to take a 5-minute break, go to the bathroom, and Maybe get a coffee downstairs. My office is above a cafe, so downstairs is a cafe. So might have to get a coffee for the next part of the stream. But this is cool. We're watching a small language model fine-tune in live, real time. Accuracy has gone up. That's cool. Loss has gone down. This is how many number of tokens our model has seen, 246,000. So this is faster than I thought on the DGX Spark. I thought it was going to be a little bit slower than this, but 3 minutes is not bad. Yes, Alex says bringing back MLfit with the push-ups. Yep, that's the new series, new series idea, ML and Muscle. Aditya, hello from India. What dataset are you using? Got late to the stream. Okay, let me show you the dataset. Training is almost done, by the way. So I'm just using this dataset here, Food Extract, which is essentially— can I just view one of these samples? So these are image captions with a JSON of food information extracted from it. I'll show you what it looks like in actual format. Yeah, here we go. Example sample. So sequence. So that's an image caption. And then I've used GPT-OSS 120B to label it with a prompt. And then I'm trying to get this Gemma 3 270mm to reproduce this output here. So food or drink. Sorry, the condensed version. So, uh, food or drink, 1. So 1 or 0 for is it food or drink or not, because there's some not food captions in there. Tags, a food items, drink items, um, food advertisement, food packaging, all that sort of stuff. Drinks, foods. Yeah, there we go, 61% accuracy. Finished, we're done. We've just fine-tuned a small model on an NVIDIA DGX Spark locally in what, 2 hours writing every line of code from scratch. That is pretty cool. No generation— generators were used here. So let's bring in our tags_dict. And I'll just link this here so you can see it in chat. Jemma, we're doing, uh, text only. So, um, reminder input equals text. Of image caption. Output equals structured data. So we have— we should have a fine-tuned model right now. I'll just show— we'll remind ourselves of what the inputs and outputs are. So this is where our model— before we saw what our model was doing before, it was creating Python code, which is not not what we want. It's okay, but not what we want. So this is our example input, an image caption, a top-down view of a rustic wooden board featuring 3 distinct items, a mound of peeled white yams sliced into thick ivory-colored rounds, smooth, etc., etc. And then we want to output this. So food or drink, 1, it gets 1 because the caption is about food. Tags, fi, because it's got food items. And then foods is the peeled white yams, the deep red gochujang, golden brown, and pandas. Right. And then there's no drinks. So why would we use this? We could take this model over a large dataset such as DataComp-1B. Maybe in the future we, we work out how to do that. Right. This is a 1 billion dataset of images and captions. And we might go, we want all the food images for few food image URLs from this dataset. There's 1 billion images. We don't want a billion images, we want 100 million or even less. We want 5 million, but we want only food because we're building an app called Nutrify, which is the app me and my brother build. And we want it to be able to take photos of food and understand what's in there, right? We want to build this app. And then— but we only want food images, so we could run it across all of these text items here and then extract all the food items from that. So that's where you would use some sort of model like this. So let's go back to where we're at. We've got our trained model. We could have tracked our information, but now we're going to get— let's look at the loss curves. Hey, or should we just try it out? I might just do a little cheeky thing here. Copy the loss curves. There we go. How cool is that? So training and validation loss per epoch. So the training loss— did we, did we? Yeah, we use the train and test dataset there. Okay, that's good. So that's basically what we want. The training loss going down. The validation loss is already pretty low as it is. So our model might be overfitting here, but that's a weird thing in the case of LLMs. You kind of sometimes want them to overfit. Because the— my intuition of this is because the token space is so large, overfitting in an LLM is not too bad if you have a specific use case for it. For example, if we just want to extract structured data, overfitting is actually probably okay because it just means that the model is outputting to me— again, test it on your own things— but my intuition is going, Well, that's okay because we want it to output the same structure every time. So let's figure out how we— oh, we can save the model with Trainer. We should have got some checkpoints over here. Checkpoint models. There we go. Here's our checkpoints. model.save_tensors checkpoint 1, 2, 3. We could have just saved the best one, but it's going to be loaded by default. That's okay. So we can go back to VS Code. So save the model checkpoints, and there's our saved model here. So I believe next we're going to load the model back in. And see how it goes. So let's, let's run it on one sample. And then I'm going to take a quick break. And then I'm going to come back. And then we'll see if we can create a little demo with it. So remind ourselves of where the checkpoint dir is there. And then we can go— let's go here. I'm just going to load the model back in. If you've got any questions, put them in the chat, otherwise— and I'll get to that in a second— otherwise I'll just keep steamrolling ahead. Auto model. Do we need to retype all this? Absolutely not. But are we going to do it anyway? Yes, we are, because That way we start to get all of this stuff ingrained in our brains. Sets and reps, like just going to the gym. Every time we train a model, 10 push-ups. Or even if we don't train a model for a while, every hour we do 10 push-ups. dtype equals auto. And then attention. Okay, let's check our loaded model. This is going to be a loaded version of Gemma 3. Hoo-hoo! Look at that. So it's just exactly before as the original model. We've got our loaded model, but now of course the weights will be different. So this is our original Gemma 3, same architecture. All we've done is update the, the weights behind the scenes to do our token task instead of its default token task. Because what is an LLM in general? Reminder, LLMs equal tokens in, tokens out. If you if you get the right tokens in, you'll get the right tokens out. And of course, you define what the right is for you. So how can we do this? We do some inference on some samples. Yeah, that's a good idea. This is, this is a notebook I created a couple of months ago, as you can see, 10th of December. So we're going to load our trained model in and then we're going to do predictions on the test samples. Remember, we've trained on the training sample. We want to do it on the test now, evaluations. So dataset train, we trained on these and now we want to eval on these dataset test. Is everyone up to date with where we are? And Pandas, Argentinas are so delicious. Oh yes, it's getting me hungry. Did it run faster than your 5090? I have a 4090, um, but I haven't run this exact workflow on there. We can compare that later on once I've got this notebook working. We can just run it end to end on the 4090. But, uh, I'm gonna see how, how it runs on the NVIDIA DGX Spark first because, I mean, it's— this is such a cool little thing to have set up on the desk. I had a photo of it before. For those wondering, this is what we're running on. So there's my Mac Mini, I'm coding on that, and then all the training's been done on the Spark, as you can see here via SSH. So the Mac is running this display but all of our code is running directly on the NVIDIA DGX Spark. So such a cool little setup. Okay, let's test our loaded model. We'll just create a pipeline to begin with. Loaded model pipeline, pipeline text generation, and then model equals loaded model, and then tokenizer equals tokenizer. We can use the same tokenizer because we haven't actually changed that. So I'm going to bring this out here, comment that, and then let's see, loaded model pipeline. How do we run this on the exact sample we did before. So if we recall, here's our prompt that we tried to get our model, but it didn't really work. And here's our random sample that we tried before, and that, that is a ground truth output. So if our model can replicate something like that, we'll be really happy. So let's grab this. Oh, actually, we can't run it on the samples for the train because we need one from test because our model has seen all of the train samples. So we want to go random test, and I'll put the test sample in here. Random test sample input prompt pipe. I want the loaded model pipeline, not just pipe. And this should work. Let's see how it goes. Invalid key. Oh, excuse me. Boom! There we go, there's our first sample running on the NVIDIA DGX Spark with our own fine-tuned model. That's exactly what we want. Now let's try it on another one. These are— this is exactly right. Input. There we go. Okay, that's a great one. So let's look at the ground truth and compare it. This is the ground truth. So this is test sample, by the way. Our model hasn't seen any of these. Okay, so it missed some drink items. So that's, that's an error of our model. But did it get most of the food items? Soybean oil, garlic, shallot, kaffir limes, fish sauce, parsley, pepper, fish. It didn't double up, it missed a few. Okay, water and lime juice. Yep, lime juice, it missed that. So that's a good example of where we could improve our model in the future with more samples with drink items. I think that, that's a good example. Exclamation point. Yep, that's not food or drink. Oh, this is a big one. Water. So it missed the water. Pita bread, wheat flour, vegetable shortening. Look at this. Our small, tiny little model is performing not perfect, but it's on par with, with GPT-OS-S120B. After just 3 minutes of training. Like, this is only 3 minutes of training and our model is already starting to output exactly what we'd like. Okay, it got that one wrong. That should be a not food item. Flavored cream, food item. Yes. Nice. Drink item. So I've got the wrong tags there. Okay, so we're seeing some errors in our model, but it's not too far off a very powerful model. So what do we want to do now? Well, we've seen our model. Let's, let's make it a little bit— I guess this is only 1,000 samples too, by the way. So that's not even really that big of a dataset. Let's make it a bit nicer of an output, as in I think we want in the next, next stream, probably we'll create a, uh, a demo of running our fine-tune LLM, and then we'll also get it to, um, Hugging Face. So I think that'll be a good place to, to do next. So what have we done here? Supervised fine-tuning done. Done. Demo app ready to use. Okay, so that's actually— let's do that tomorrow. Tomorrow's goals. So this is December— or January, sorry, January 8th. Jan 7th. Demo app ready to use. Bonus. Or, and then we want to go evals. How does our model compare to GPT-OSS 120B? Okay, so I think we're right here. Next. Create small demo of using the model. And then we also want to save the model to Hugging Face. 3, create a shareable demo to Gradio, e.g., text in model extracted outputs out. And then next is evals. How does our model perform compared to GPT-OSS-120B? Oh, the size of the train model. Let's get Let's get— how do we chat here? Create me a function to count the number of parameters in a PyTorch model, both trainable and not trainable, as well as total parameters. So we'll get Gemini 3 Flash to make a small little function here. There we go. There's our number of parameters in our model is 270 mil. So compared to GPT-OSS, which is 120 billion— I'm gonna write here, up to here. So our model is 270 mil parameters. GPT-OSS 120B is 120B parameters. So let's figure out the ratio. I think it's about 400x smaller. So 120— 1, 2, 3, 1, 2, 3, 1, 2, 3— or actually, 1, 2, 3, 1, 2, 3, 1, 2, 3. That's 120 billion 9 zeros divided by 270,123,123. Yeah. So our model is 444 times smaller. How cool is that? 440 times smaller. So Yo, Daniel, how are you? Hey, S-Jed, nice to see you. Thanks for tuning in. Uh, Suhail says, hi, so we take the base model, then fine-tune train it with our own dataset so we get specific model. We can ask anything to that model around the dataset context. Yes, that's exactly right. So no matter our model is fine-tuned for a very specific task, so as we'll see in the demo in the next video, um, which will be tomorrow, tomorrow's live stream No matter what we input to our model, we've fine-tuned it to only output a specific format. So that's the power of fine-tuning is now in this case. There we go. Have a look at this. We want our model to extract food items and drinks from a food— from an image caption. So we've got Lindt extra dark chocolate packaging, Lindt double shot latte. So there we go. Double shot latte. Dark chocolate is the food. Lint difference. That's incorrect, but that's— I guess we could work with that. Russian Caravan, which is a different type of, um, a tea, a tea, drinkable tea. So that's very cool. And this is only off 1,000 samples, right? We could always upgrade this with more samples. Chocolate bars, crunchy, Dairy Milk Caramel, picnic, Mora, Boost, Turkish Delight, cherry ripe. There we go. Um, whereas this is gibberish. It's not about food or drink. And so therefore we have no tags, foods or drinks. So thank you everyone for tuning into the stream. We have officially loaded a small language model on the NVIDIA DGX Spark. Let's get our little image up here. From scratch, we have gone in 2 hours or thereabouts, we have gone from, with a lot of talking in between, gone from coding on a Mac Mini to coding on the NVIDIA DGX Spark. We've loaded Gemma 3 270 mil and a dataset with 1,000 samples we've fine-tuned that Gemma 3 model. We saw that it didn't work too well, um, the default weights, even when we tried to prompt it. But now it is working quite well on our target, uh, problem, right? We wanted to extract food and drink items from any given input sequence of text. And so that's LLMs in a nutshell—tokens in, tokens out. In the next stream, we're going to create a demo of our model. We're going to save it to Hugging Face, and then we're going to share the demo to Gradio so that you could try it as well via a link. Right. And then we might even— maybe that's a third stream is to do proper evals. So see how our model compares to a model like GPT-OSS-120B, which is where our labels came from. So stay tuned. If you want to see anything otherwise in particular in a stream, you can just email me here. Daniel Bourke at my website. So you'll be able to find my website, but that's my email if you want to see anything in particular. Otherwise, leave a comment. I'll leave all these streams on YouTube. I'm trying to get back into streaming a bit more, so I'm keen to hear all your ideas. But thank you everyone for tuning in all over the world. I appreciate it, and I'll see you in the next stream. Hey.