Local LLM fine-tuning on the NVIDIA DGX Spark - Part 2 uploading model + making a demo

Testing. Welcome, welcome, welcome. Let me just make sure this works. If you're joining the chat, please let me know. Welcome. There we go. Okay, so yesterday the problem was people couldn't hear me through their headphones, so I'm going to test that. It should work now. So this is going to be weird because I'm going to listen to myself, but today's goal we have— okay, that sounds good. I can hear myself through both the headphones. Yesterday I think it was only one channel. Hey again from Bruno's Airways. How are we? Bruno's Airways. Sorry, I said that wrong. Okay. Okay, Apple's system is good, but sometimes it connects to too many different things. Yo, yo, yo, welcome back, Alex. Welcome back. Do I say it? Santiago. Santiago. Okay, so today's goal— yesterday, what did we do? Yesterday we trained a small LLM. Oh wait, you can't see. There we go. Why is this not showing the right screen? One second. Okay, there we go. Namal, oh, thank you. You just finished the deep learning course? Well, we're getting into more deep learning today, but we're doing more sort of, uh, I guess higher level things than sort of— we're using— we're going to train some LLMs. So yesterday we— this is what we did yesterday. We're running on a DGX Spark. Let me show you. So this is where we're running, okay? Writing code here, running code on there. And today's goal is we're going to take the model that we built yesterday and we're going to put it on Hugging Face. So yesterday we, um, fine-tuned a small language model to do structured data extraction. So we fine-tuned Gemma 3 to 70 mil And now we want to make it publicly accessible and we want to build a Gradio— not emo app, we want a demo app so people can just try it out. Right. And our model does food extraction and we've got some rules, 10 pushups every hour or when a model starts training. Okay. So if you want to join in on the pushups, you can. This is a cooking show called Machine Learning and Muscle. All right, that's, that's going to be the stream. So everyone's here. Okay, we're gonna do 10 push-ups to start off the stream. And then we're going to get into taking our model from yesterday, trying it out, and then uploading that to Hugging Face. And then everyone will be able to access it because it's via a public link. Okay, first set of 10 push-ups done. Let's go to the counter. We should do— we should actually make a model. Count 10. So we've done 10 so far. Next hour we'll do another 10. So where were we? We go into the code and I was trying out FastVLM, which is a VLM. We'll get to that in a second. Uh, yesterday— that'll be the next series of streams. We're going to fine-tune just a text-based LLM in these streams, and then in the future Zoom in on both editors. Okay, how's that? In a future stream, we're going to fine-tune a VLM, so vision language model. So we're up to here. Let's see if our connection is still working. Okay, beautiful. Create the small demo of using the model, save the model to Hugging Face, create a shareable demo to Gradio. Text in, model extracted outputs out. Pose estimation model to count push-ups in the future. Yes, that's a great, that's a great stream we're going to do. You could even have like a live counter on the stream. So like every time I might have to move my camera so it can see me, obviously. And then every time you do push-ups, it just goes, oh, okay, you were sitting. Now state has changed to push-ups. We'll keep a count of the push-ups. So for those who don't know what our model is doing, we'll run it again on a test sample. This is running on our dataset. That's test. We have 1,000 different samples, 200 of them about test. And the goal is food extraction. So if we were building an app, such as Nutrify, which is what my brother and I actually build— you can download this in the App Store right now— um, and you wanted to filter a large data set for food images, you might build a model that can go over a large database of image captions and extract different foods. So that's essentially the model that we're building. Um, and now if we have a look at the inputs and outputs, we can see here this is our input to the model. It has been formatted with special tokens. Remember, an LLM is just what? Tokens in, tokens out. And our input here, the image shows a box of Arnott's flavored biscuits, includes wheat flour. And then the goal of our model is to tag the text to tell us whether it's food or drink and to extract the food and drink items. So see here, food, wheat flour, vegetable oil, tomato powder, starch, yeast, garlic, Worcestershire sauce, etc. So now we could use this over a large-scale dataset and then have that easily explorable because we can just transfer this kind of structured output into JSON or a DataFrame or something like that. But right now, since our model is in, it's just running on the NVIDIA DGX and this is okay for me to interface with, but we want to make it accessible for other people. And so to do that, we're going to one, upload our model to Hugging Face. And two, we're going to create a demo of it so you can just type in any text you want. And the goal by the end of this stream is to have a link that you can try out this exact model that we've been training here on Hugging Face and try it out on your own text. Hi from Bogota. Hello. Let me know if the zoom is good enough here. So what we're going to do next This is today's stream. Oh yeah, and, uh, our model is 270 million parameters, so it's a small language model, whereas GPT-SS-120B is obviously 120 billion parameters. So we're taking it from 120 billion parameters to 270 million, which is a 444x reduction in size. How cool is that? So let's go turning our model into a demo. Now we might need Gradio. Import Gradio as gr. Okay, we don't have Gradio. That's okay. So we can— do I want to restart? Maybe I just make a demo in another file. Llm_demo. All right, and then, then we can just re-import Gradio from the start, turning our fine-tuned LLM into a demo. So we need some code in here. I'm going to do ai— I actually, I might have to install— and then we'll go uv pip install Gradio, and we should be able to get Gradio now. Beautiful. Okay, and then from Transformers import automodel and, um, or is it automodel, automodel for causal? Yeah, lm. And then we might also need— might just bring this over here. We don't need terminal at the moment because what we're going to do is we're just going to load a pipeline. Yeah, there we go. Auto Tokenizer and then pipeline. Our model path is our checkpoint dir name, which is over here. So these are our saved models. I— this is, uh, I haven't organized the file system here very much, but in the future when we do some more larger scale projects, we'll organize our file system. But this is our checkpoint models over here. As you can see, we've got the model saved there. That's going to be our weights So we can just save this here, checkpoint models, and now we can go— is this going to work? Beautiful. Okay, so we've got pipeline, load the model into a pipeline. So loaded model pipe, or loaded model actually, and I'm calling it loaded model rather than just model so we know that it's the one that we have fine-tuned, not just the default. From pre-trained and then model path. Now there are many more experiments we could run here to make our model better. For example, we could try less parameters, we could try more data, but I prefer to, uh, train one model that works okay or pretty well and then turn it into a demo and then interact with it in the real world. As in, we turn it into a demo, we try it out with some actual samples rather than just the training data. I hate IPYNBs, which are Python notebooks. So I use them, Python notebooks, for basically just experimentation. And then once I've got something working, then I turn it into a script. So that's why, because a lot of what we're doing at the moment is experimentation, I'm using notebooks. And then once I've got some stuff that's working, then I'll turn it into a Python script. So that's— yeah. Oh, we need our tokenizer, of course. Forgot the pre-trained method. Beautiful. And then we got our tokenizer, and we should have our loaded model pipeline. But now we need a way to format our data because the pipeline expects something in a certain format where we go down to our pipeline. Okay, yeah, here we want apply the chat template. Load the pipeline. Now see, if we try to do just loaded model pipeline— oh, there we go, it's just going to complete it. Wow. I'm a volunteer at the shelter, helping with everything from stocking food shelves to cleaning out the bathrooms. That's, that's an interesting generation. Akash, what's up? 8 years flew by. Yeah, vlogging in my car, dancing in Hawaiian shirts. Haha. Are you gonna use Claude Code? You know what, I must be the only person in the world who doesn't use Claude Code. I will try Claude Code eventually. But for now, I like using my own brain. I don't have any projects at the moment that require several levels of agents to do. A lot of my stuff is experimental. Maybe once I've done some experiments and Phoenix found something that works, then I'll use something like Claude Code. But we're getting distracted. Let's go here. We need to format this so that it is in the chat output. This is basically the, the crux of getting stuff ready. So I believe it is— where do we have it? We need content. Yeah, we needed in role user and then content. So def format_message_input. And then we want it to return role user content input. So I think this should be what we need. So, uh, input message equals, and then we want to go input There we go. Hello, my name is Daniel Bourke. So can we pass this into our pipeline yet? No, not just yet, because we need to apply the chat template. So input prompt equals pipe And the reason why we're rewriting this is to get more familiar with— we don't actually need to rewrite all of this all the time, but I just like to practice several times so that when I come to the next time of doing something like this, I have experience of doing it. input formatted tokenize equals false add generation prompt equals true, and we want loaded_model_pipeline. We got something wrong. Uh, this needs to be a list, I believe. There we go. Okay, so we've got our input prompt formatted and now we can go loaded_model_outputs equals loaded_model_pipeline, text_input equals input_prompt, and then max_new_tokens equals 256. Now if our model has loaded correctly, this should output that this is not about food. Let's have a look. And then we'll just get this. Oh no. It disconnected from the Spark. How's my stream? Am I Okay, we got connection back. Stream still working, looks good on my end. Okay, thank you everyone. Stream's good. So there, that's exactly what we want. So is this about food or drink? Uh, this is incorrect actually. This should be a zero because this is not about food or drink. Um, the tags, food items. So this is a great example of where we would do interactive testing and then in another training run we would improve the model. So a lot of people are sort of afraid to get errors in machine learning models. Oh, it's, it's not working. But as a machine learning engineer, you should crave errors. So let's write this down: errors in ML are good. It shows you where to improve the model. And then go, there will always be errors in ML since the whole science is probabilistic anyway. But the goal, of course, is to make a model so good that the errors are very, very small. So let's go. Now we want to turn this into a demo that we can just type in text, get this output out. So turn our pipeline into a demo. So to do that, of course, like all good cooking shows, I have prepared something earlier. And we're going to use Gradio, which is nice and simple, but a few things we need. We want Our demo is simple. Text in, formatted LLM text out. Also, also want to time how long it takes per sample because that's another thing in machine learning. Yes, of course you want your model to work, but then what's the next thing you want it to do? You want it to be fast. So we want it to work in the first place and then next we want it to be fast. So import time. Exactly. Nothing screams bork than 100% accuracy. Yeah. Still making passive income from the course. I am. That's, that's one of my main sources. I still make course materials and whatnot. Drives a lot of my, yeah, livelihood. Yeah, if you ever see a model with 100% accuracy, always be skeptical. So that's like all of the recent LLMs, right, where they're, they're very good. Don't get me wrong, they are very good. But when they come out on the benchmarks and they're getting basically 100% on the benchmarks, then you know they've either been trained on the benchmarks or the benchmarks aren't a very good measure. Or there's so much training data in the world now that when they train on all of the internet, the benchmarks are basically in the training data as well. So we're going to go loaded model pipeline and then we need text inputs here because our Gradio demo is going to just absorb raw text. We want to see, or we want to format that into what our model needs. And we've seen in the past that our model wants message format. So as in, what's my role, what's the content. So the role is the user, user is going to input some text. And that's going to be here. How do we find this? That's in dataset formats in TRL. So very helpful. If you want to learn how to train your own LLMs or small LLMs, go through this library, TRL. That's exactly what I'm using. And then you can go here, language modeling. We're doing this here. So conversational, we have a role, a user. We have trained our model to respond as an assistant with a certain type of content. That is all we're doing. Tokens in, tokens out. The beautiful thing about LLMs is that because they are a decoder, the output space is basically infinite. So you can customize the output. In traditional ML models like a classifier, you will have a limited number of labels that your model can output to. The beauty of LLMs or VLMs or whatever you want to call them is that their output space is infinite. But that's also a downside because sometimes you don't want an infinite output space, you want a constrained output space. So we can't forget that the beauty of computers is that they do the same thing over and over again. So input text, and then we want to add some little niceties here. So we'll default to max new tokens 256. We don't actually need that many tokens. And disable compile. In the past I've had issues with compiling, so we might just disable that for now. We can explore what that looks like in the future. And then we're going to end the time, and we'll go total time equals around in time. I always like to measure latency. That's something that you got to remember is like if your model works but it takes 10 seconds, uh, people might not want to use it because raw output— well, I guess with the new ones, right, they got all the thinking and that sort of stuff that takes a while. There's a trade-off, right? If the output is good enough, you're happy to wait. What did we get wrong here? Oh, we missed a Mr. T. Return generated text, raw output, total time. Okay, now this function should just allow us to pred on text. And I want the input text to be— let's just see, does it work on an empty string? So it should say food or drink? No. But then if we go, let's For breakfast, I had a cup of tea, uh, and some walnuts. There we go, right? So here's our model. That was quick, very quick. That's what we want— 0 point— under a second, so 37 milliseconds on the NVIDIA DGX Spark. And that's one at a time. We haven't even got into batching inputs, which would probably give us a good speed up. So what is our model done? It's got foods, tea, and walnuts. I would say tea should have been a drink. So that's another error we can start to write down here in the future. We'd go model errors. Tea equals drink, not food, right? My name is Daniel Bourke equals not food, but model says food. So this is a very rudimentary way of recording errors, but it's a way to basically build up your internal representation of how your model is doing. So let's turn this into a nicer interface. Gans is new. Hello, love your content, Dan, from Comzi. Thank you very much. So what we're doing today is— I'll just give a quick rehash if you're just joining the stream. We are training a small language model on NVIDIA DGX Spark. We are going fine-tune LLM. We are fine-tuning Gemma 3 270 mil. So a small language model. So if we go to Gemma 3 270 mil, this is the exact model we're fine-tuning. Actually, we have the instruction IT version. I haven't tried the base model yet, but that might be worth in the future. Gemma 3 270 mil. Here we go. This model, this is the model that we're fine-tuning. So straight from Google, and I'll just link it into the chat. Good research. We say Anthropic has some good research on persona vectors. Ooh, I haven't heard of that. Persona vectors. But right now we just want a simple model that can extract food from text. So demo time. Now we've got a function. Gradio functions as just basically input I'll just write down here, Graduo 101 input function, e.g., a model to do XYZ output. It creates a beautiful interface for just that. So let's go create a description equals Extract food and drink items from text with a fine-tuned SLM. So small language model. And then we go, um, input, which is going to be a string, raw text strings. Or image captions, e.g., a photo of a dog sitting on a beach, or, um, a breakfast plate with bacon, eggs, and toast. And then output equals string, string output as well. But we can format our, uh, we're doing a raw string output. That is because instead of outputting JSON, we've taught our model to output something similar to YAML because it uses less output tokens. So output equals generated text with food, not food classification, as well as noun extracted food and drink items and various food tags. So for example, input, um, for breakfast I had eggs, bacon, and toast, right? And let's do an exact output of what our model I'm going to put this in here. So hopefully— oh wait, I'm going to go and a glass of orange juice. So there we've got 2, 3 foods and 1 drink. So let's see if our model can extract what we want from there. There we go. That's exactly what we wanted. Isn't that cool? I think that's cool. We fine-tuned an LLM to do this. Look at that, right? It missed the tag of drink item. There should be— so fi, we'll just get our tags sticked so we can simultaneously tag things here. So fi is food items, di is drink items. So it should have had a di tag here, but that's close enough for me. So we could just put di and then we're going to put this in here. There we go. So our goal, my goal for the rest of this stream is to get this onto Hugging Face Spaces so that you can try it out on your own text. And if we do that, well, then that would be a successful stream for me. Interface. So the function we're going to— so Gradio, we're going to create an interface and we're going to input our function pred_on_text, which is up here. All it does is take text input, runs our model pipeline on it, and returns the generated text as long as— as well as how long it took. So very simple function. Inputs. I think XML uses the least output tokens. Depends. You could always re— sorry, you could always token count things. I've done mine to like the least compressed version that is still human readable, if that makes sense. So, um, this could be compressed probably even more. You could probably just put this to FD, TFD, if that makes sense. But I think these drinks, foods, tags, food or drink, these are probably close to just one token each. I haven't done the exact token measurement, but if you really wanted to compress, um, you would like see what the actual token counts of this is. Oh, this is incorrect, isn't it? That's a drink, orange juice. But in some regards, depends on how you would define food, right? Potentially some people count orange juice as a food. If someone uses ChatGPT for coding, but it is like using an assistant, is that okay? Yeah, of course. I like this article. So Simon Willamson. I think it is. Um, your job as a software developer blog. There's something here. Yeah. This is my favorite one on describing the use of code or AI assistants. So I love AI assistants, I use it all the time. But this is what your overall job is. Your job is to deliver code you're— you have proven to work. So if you want, we can read through this article later. And that might be a good just video, right, that we could just extract from the stream. So that's what I want to try and work on for this year is to do most of the stuff livestream. I want to do a fair bit of livestreaming, but then also we can stop and take a break and read articles like this, and then we can turn that into a video similar to like what the Primogen does. Really inspired by his work. Okay, so where we're creating a function. We're creating a demo. So the inputs on Gradio are going to be gr-textarea, lines equal 4, and then the label is going to be input text. And then the outputs is going to be— we want our outputs to be multi-thing here. So we want it to be a list, but we want our LLM is going to output lines, and then we're going to go label equals generated text, and then we're going to go gr-textarea, and then the lines is going to be 7, and then the label here can be the raw output. And then finally we want gr-number, which is going to be label, and then let's go generation time. S, right? And then we can give our demo a title equals food extract, structured food. Yeah, we can call it structured food extract. Why not? A fine-tuned Gemma 3 270 mil. And then the description is going to be equal to description. And then the examples— let's make some examples. Who had what for breakfast? Put in the last meal you had in the chat and I'll use that as a direct example. This is my first fine-tuned LLM. All right, and then I had— what did I have for dinner last night? A plate of, um, food with grilled barramundi. I had some fish salad with avocado, olives, tomatoes, um, and we'll call it Italian, uh, and Italian dressing. And do we have any chats? Yogurt with berries and steak tacos. There we go. We're basically doing NER with Gemma. That's exactly right. British breakfast. Asking for a quick recap, I joined about 5 minutes ago. You fine-tuned Gemma 3. So yes, we have fine-tuned Gemma 3. Um, so here we go. I'm going to put in your exact— you put— you said you had steak tacos. This is— I'll show you how fast it is, right? I had steak tacos. All this is going to do is just run the model behind the scenes. So let's see what it says. There we go, 0.3 seconds. So food, steak tacos, there we go, done. And then here we go, yogurt with berries. There we go, 0.3 seconds. This will be on Hugging Face soon as well. So we'll be able to see how fast it is there. Yogurt foods, yogurt, berries, beautiful British breakfast. So let's look up instead of just having British breakfast, let's look up a British breakfast images. Okay, so I'm going to write down what's in here. Manual mode and we'll see if it works. So, British breakfast with baked beans, fried eggs, black pudding, um, sausages, bacon, mushrooms, a cup of tea, and toast. I think I've got everything there. Maybe tomatoes and fried tomatoes. Okay, so let's run it again. That took 0.7 seconds because it's a longer piece of text. But this is one, one at a time, right? We could, we could speed this up with a batch generation. But so what did we get from here? We get food or drinks, tags, British breakfast, baked beans, fried eggs, black pudding, sausages, bacon, mushrooms, toast, fried tomatoes, drinks tea. There we go. That's exactly what we wanted. And tags, food items, drink items. So beautiful. So we could— this is NER, so named entity extraction. But the beauty of LLMs is that it can work on any form of language because it's been trained on the internet, right? We don't need to close off our— that's, that's one big thing that sort of an aha moment when LLMs first started to come out was the output space is infinite, so we can keep that as an infinite output space and create something like a ChatGPT, or we can constrain it and but make it constrained to exactly what we want. And in this case, we've constrained it to exactly extract food items. So I'm going to copy this. This can be one of our examples. Steak tacos as well. And then steak tacos, and then we'll do one more which is not food, a photo of a dog sitting on a beach. Okay, and then let's see if we can launch our demo. Oh, what do we get wrong here? I'm missing a comma here somewhere, aren't I? That's correct. What if we just run that? Unexpected indent, comma, one too many brackets and a typo, and then go demo Launch share. Here we go. Let's go to our URL. Hello from Haiti. Hello, Ashik, how you doing? Momo. What's Momo? Okay, we might need to turn on port forwarding. So 7860. Is this gonna— there we go, we have a demo. Okay, so this is cool. This is our— this is running on our Spark. Now, now, of course, we could fix this up a little bit, but that's okay. So, right, this is what you see on Hugging Face. Someone released the demo, they're going to post this, we're going to post this. This is our goal for the stream, is for you to be able to use this. So if we put in steak tacos and we click submit, there we go. Output, there's the food, steak, Tacos. I would say that's correct. British breakfast. Let's run that again. 0.7. So food items, British breakfast, baked beans, fried eggs, black pudding, sausages, bacon, mushrooms, toast, fried tomatoes. Drinks, tea. How good was that? Now let's do a not food one. We've got a dog sitting on a beach. Food or drink, zero. So you notice how the longer the text is, it takes a little bit longer, of course. What if we did this one. So food or drink, grilled barramundi salad with avocado, olives, tomatoes, Italian dressing. Now, if we wanted to break them down a bit more, we could go salad is one component, avocado is one component. But to me, that's, that's good enough for now. We are only using 1,000 examples here and this model trained in 3 minutes. In the last stream, we fine-tuned that. It took 3 minutes to fine-tune, which is nothing, right? So that is the power of fine-tuning LLMs. And of course, this is still just raw text, but we could easily write a parser to turn this into JSON. So if we went to somewhere, this is where we could go into Claude or ChatGPT and go, can you write me a parser in Python to turn the following string into formatted JSON. Use the fields as the JSON keys. Food items, drink items, and tags should be lists. Drinks, I'm gonna write here water. Momo is a Nepalese dumpling. That sounds yum. Let's have a look at that. Momo dumpling. Yum. Let's try that then. I'm not sure what sauce that is, but let's go here. A plate of food with momo dumplings, uh, orange— what could we call that? Orange salsa. And I'm gonna call that What's this one? Does anyone know what that sauce might be? We might just go— is that a jam or a barbecue sauce? I'll call it barbecue sauce. There we go. So a little bit, a little bit of an error there. I'll zoom right in. Food with momo dumplings, orange salsa, barbecue sauce. But that's, that's That's basically what we want, right? And we could just delete this as well and see what happens. Momo dumplings, orange salsa. There we go. So this is structured data extraction 101 with our fine-tuned model. And this is— I want to just clarify, we've done it for food. You could do this for anything, whatever business you're working in. If you need a specific type of structured data extracted, you could gather some samples, train a fine-tuned generative model, have it running locally. That way none of your data ever leaves the device. This is all running on the DGX Spark, right? So have we got a pasta JSON? There we go. So we can come back to that later. We don't actually need that just yet. Let's go back to here and let's work out how to get this onto Hugging Face Spaces. All right, we've got it working locally. Now our next step is to publish it to Hugging Face so other people can use it too. And what's our timer at? 48 minutes. Okay, we've got 10 more minutes and then we do some push-ups. So next we are up to upload our model to Hugging Face, and we'll do that. For some reason, the Escape key in VS Code doesn't work very well for me. Upload our model to Hugging Face. And then we're also going to upload our demo to Hugging Face. Maybe we should do a cool thing would be to have the original output next to the extraction structured output. Original output. Now that's all right. We just want to get this up on Hugging Face first. And I want to put food. What should we use? Which emoji should we use? Maybe we use banana. Oh no, no, no, banana's already been used too much. Let's use egg. Always got to have the emoji. So upload our model to Hugging Face. Let's figure out how to do that. How to upload a mod— uh, loaded model to Hugging Face. Hello everyone. Pomodoro push-ups. Yeah, livestream push-ups feels like 2021 again. I know, how good's that? For your NER, do you use two-pass extraction? One is recall heavy, another precision heavy? No, Ashik, this is just, um, straight LLM extraction. So we just fine-tuned a Gemma 3 270 mil, and all it does is it just takes in a random string. So whatever, if you put in the chat your favorite food, I'll try it on there, and then it extracts the food or drink, um, to just raw text, and then we can process that. So it just does one forward pass. So I think it's just pushed to hub. The Google AI summaries are getting pretty good. I don't know about you, but I think they've, they've improved significantly over the last year. Model, I think it might be— yeah. I'm going to do this. I'm going to use— do I have the Hugging Face Hub API here? So we've copied some code from Google, but we're going to retype it because we're good at practicing things, and then we can troubleshoot the errors ourselves. So H— HF API create repo. Do we have the hub? Yes, we do. Toby asks, would it be able to extract concepts, so more abstract? Can you give me an example of input and output? And basically, I would say yes, if you have enough training data, like 1,000 or so samples. So API equals HF API, and then repo ID equals Mr. D Burke slash, um, I'm going to call it what is food extract. GemArt 3, 270 mil, fine-tune, V1. So we'll create the repo. And then we go, um, upload folder. So maybe I delete my checkpoints because I just want to upload this. If not, we can just— if this doesn't work, we'll, uh, just come back and do it again because it only took 3 minutes to train and we've got the dataset, which is probably the most important thing. So create repo, we'll go repo ID, repo type equals model, exist okay equals true. Let's see if this works. There we go. Okay, we don't have anything here. Food extract Gemma 3 270ml fine-tuned v1. This is on my Hugging Face. So now we want to upload the entire model folder. So I wonder if I can just do API upload folder and then go folder path equals— I want to— is it just going to call it checkpoint models? Maybe we don't want that. Repo ID is going to be the repo ID, and then we're going to go repo_type equals model. Is not a directory. Okay, so we need this. There we go. So if all works well, let me know if you can start to see files being uploaded here. So this is pretty quick. I got pretty fast internet speed. Let's do a test. So this is my internet speed. So we're getting, we're getting some pretty good uploads here. Of course, our model was multiple gigabytes, it would take probably a lot longer, but downloads is fast, uploads is quite fast too. There we go. Okay, beautiful. So, Link, so Ashik asks, why fine-tune Gemma for extraction here instead of using an open, uh, LLM from OpenRouter or something? So with structured JSON— great question. Let's write that down here. Why own model? One, own the model can run on own hardware. Two, our task is simple enough to just use a small language model. 3, no API calls needed. 4, um, we can, uh, whatchamacallit, can run in batch mode to get much faster inference than API calls. 5, model by default wasn't very good at our task, but now since fine-tuning is very good. So that's basically why we'd fine-tune our own model. So now we own it, we can publish it. Our task, it's very simple. Now we just basically pass text to the model. We don't need to even prompt it. It just text in, structured data out. We don't need any API calls, so it can run offline. Of course, um, I found that API calls are fast. Yes, you get a better model. However, they're not as fast as running a model in batch mode. We could do that in a future stream. Um, we might be able to get like 30 samples per second, even quicker, I don't know. And then model by default wasn't very good, but now we've fine-tuned it, it's very good at our specific task. So the downside of using an API is GPT-5 could probably one-shot all of our tasks. Yes, of course. However, then you're burning tokens. You do need to do API calls. You need to set up prompting correctly, all that sort of jazz. But now our model is just fine-tuned directly for our specific task. We don't have to worry about all of those external things. So that's basically why fine-tune our own model. And yeah, now it's in Hugging Face. We've got it forever. Jerry Code is asking, what are we doing? So we're training a small LLM on an NVIDIA DGX Spark. So yesterday's goals, we fine-tuned the small LLM, so Gemma 3 270 mil. We did it by supervised fine-tuning, and then we today have now published our model to Hugging Face. We can tick that off. Next, we want a Gradio demo ready to use, and this has to be public. And that's what we're going to do next. And then later on, if we've got some time, we can do some evals. So our model is distilled from GPT-OSS-120B, and then we want to see how does our model at 270 million parameters, 440x smaller, compare to the 120B model. So next we're going to create a Gradio demo app ready to use. So let's try that. So we probably need another repo, repo type space. Now I believe we might just be able to do— I might turn our demo first into a Python file. Then we can just go upload app. So here we go. LLM or app.py, we'll call it that. App.py. Quantize. We could quantize it. That's another good idea. How to upload a loaded model. How to upload a demo to Hugging Face Spaces. I actually have some code to do this. Learnhuggingface.com. So we're going to go down here. I might need to fix that eventually, that scroll, but I think it's just because the page is too small. So making our demo publicly accessible, Hugging Face Spaces. This is what we're going to create, a demo file. Making an app. Wonderful. Beautiful. Let's just create this. How did I make the training data? So the training data was both real image captions, so just from the internet. It was, um, LLM-generated image captions, and it was— all of the training data was labeled with GPT-OSS-120B. So I've— I've— none of this is hand-labeled. In a future project, what I could probably do is create maybe 100 samples which are hand-labeled, and then we can basically test against those 100 hand-labeled, uh, samples so that we really know how our model's going from there. And oh, we're at an hour. So you know the rules, we're doing push-ups. 10 push-ups every hour. Okay, so +10. Let's— we need an app.py. So to make our demo, We need app.py, which is going to be entry point for our app. We need a README. We don't necessarily need a README. README tells people what our app does. And we also need a requirements .txt tells Hugging Face Spaces what our app requires. I think we only need, um, only need torch-transformers, Gradio. I think that's our 3 requirements. And I think that might be it of what we need. README, requirements, and then uploading. Yep. Sweet. Okay, I'm just gonna go to the bathroom. And then we'll come back and we'll make all of these 3 files that we need. But this is on learnhuggingface.com, which is a notebook that I've created to create a demo as well. So I'll be back in a couple of minutes. Okay, who's ready to make our demo publicly accessible? Because those are the ingredients you need for a successful machine learning project, is a dataset, a model, a demo. Three of those and you're off to the races. Of course, that's only the beginning, but that's a great modern machine learning workflow these days. And the beauty of Hugging Face is you can do it all in one platform, really. So, um, let's create first of all our app.py. Now I don't actually need an app.py file, I just realized. I can just write it in Python. I'm going to copy this, so, and we can go write file Actually, what I'll do is I'll just go, um, mkdir demos, and then we have a demo. Beautiful. And then I want to also mkdir demos, and then I'm going to call it foodextractapp. I'll just call it food_extract. There we go. And now we're going to write file to demos/food_extract/app.py, and in here we're going to put our pred on text. So actually, yeah, we need to do a few more steps here. So we don't need the example. Because this is going to be— we want this to be a singular file that can just be run on Hugging Face and works without any errors. So we need to Load dependencies. So import Transformers. We've got to treat this as if we're running this from scratch. Import torch, import Gradio. I think that should be enough. And then we're going to go load the model. So this is where we need to create our pipeline. We can create our app very shortly, but yeah, here we go. We'll bring this down. Maybe I get the outline over here, that way I can jump back and forth. Load the model. Beautiful. We'll put these imports up here. So this is where our model path is going to be different. Of course, we're going to run this model from our newly uploaded model to Hugging Face, which is here. So this is our model path. Is our tokenizer there? I need to get this in this window. Wow, it's already there! That is quick. Is our tokenizer here? There we go. Yeah, we can do a tokenizer path. Sorry, the tokenizer will automatically load. So that's our loaded model from our Hugging Face repo. So now this is going to load that model remotely. And now we can create the demo. Or we need a pipeline, which is just this. And then I believe we can just go down here because we've already got all the formatting functions. Create model pipeline, and then we get our function pred_on_text, which we've already got, which is loaded model pipeline, text inputs Beautiful. And this still works. And then the next thing we're going to do is create our demo down there. Create the demo and then We go if name equals main, and then— oh, I think we need this— demo.launch. Now let's see if this works. So we should have an app.py now. Over here, right? So that just wrote it to file. That's what, 76 lines of non-perfect Python code for an AI demo. Let's see if we can run this locally. And then we're going to go cd code. Python-demos and food_extract_app.py. So if all works well, it should load our model from— there we go, it's downloading our fine-tuned model. So Ashik says, I want to try throwing messy Uber Eats or DoorDash receipt text to this model and see if it can just pull out all the food and drinks. Well, let's try that. There we go. So this is running locally. I go, this is a messy Uber Eats receipt with the following with the following order. So we want tacos. Let's just go really random. Milk. We'll go apple. Go red apple. Pineapple. Cherries. Um, maybe we put in there fried chicken, steak. Mayonnaise, put in some gibberish. And then we'll go January 8th, 2026, because that's when I ordered it, was it was $56, and deliver to Brisbane, Australia. Right, so that's obviously not perfect, but Let's see if it works. There we go. Food or drink. Tacos, milk, red apple, pineapple, cherries, fried chicken, steak, mayonnaise. That's pretty cool, right? Only took a second. Now, what are the tags here? fi and re. Let's have a look at what they are. I can't remember what re was. Was it recipe? Oh, oh, arrey, recipe. So it wasn't a recipe, so it got that wrong, but it did look like it got the food items out. Now let's put a couple of drinks in there. And it classified milk. Okay, so let's go, um, iced latte, matcha latte. What else have we got there? Chai tea, and then sparkling water. Ah, okay. So we find some good errors here with our model. It's not perfect, but this is where we could improve it, right? This is why we tested in the real world. But this is running locally. Now we need to upload this to Hugging Face, and that's just by running python app.py. So we're going to cancel that. Now we need, um, a README, write file, and then we go demos, and we're going to go— what did I put in the README over here? Does it just need to be text? Oh yeah, this is what we need. So Hugging Face have this special README text at the top, and then we go So there's our README file. Let's have a look at that. Beautiful. Oh wait, we need a different title, don't we? Food Extract Fine-Tuned LLM. Structured Data Extractor. And we'll go— let's do emoji. Arrow. And then what food did we use? Food. I'm going to use the fries. There we go. And then we can go green, blue. Okay, now we've got a README and we need one more thing, a requirements. So demos. Food_extract, and then requirements.txt. And this is just going to be transformers, Gradio, Torch. Have a look at this. Beautiful. Okay, now we want to upload our demo to the Hugging Face Hub. So like all good cooking shows, I've got some code that's pre-prepared for this, but we're just going to change it to our requirements. So create repo, upload file, we don't necessarily need that. So demos, we're going to upload foodextract. Um, target space name is— we're going to go, um, foodextract-v1. Space, Gradio. We're doing— oh, we're doing a repo type of space, creating repo on Hugging Face with name that. We're going to upload there. Private, false. Exist, okay. I think that's all we need. Commit message: upload food extract demo. There we go, food extract. And now let's check out the app. Beautiful. So the app is building, but I want to put this on the GPU. We get an NVIDIA H200 to run it, so we'll see how much faster it runs on the H100 compared to Running locally. Oh yeah, there was an update recently. Hello, Adol Makats, nice to see you. Where's everyone tuning in from? I'm in Brisbane, Australia. We are currently uploading our demo to Hugging Face Spaces. Hopefully there's no errors, but we can, we can fix those. That's what this is all about. Oh, we got an error. We need accelerate in our requirements. That's okay, let's just update that. Mainland in Perth. Ireland. Perth. Wow, we've got a great diversity here. UK. Uh, what monitor am I using? I am using Chicago, Illinois, and I am using an Apple— I think it's called Studio Display. This one. This is my setup. Alex is back from New York. So this is what we're running on. This is the 27-inch Apple Studio Display. So not the, not the most expensive one, the, the, I guess, the entry-level Apple display. I just love the Retina. Um, I figured if I'm going to be spending 8 hours a day staring at a screen, I want it to be a high-quality screen. And so this is a Mac Mini we're coding on, Apple Studio Display, and NVIDIA DGX running there. So we're trying to deploy our food extract space to Hugging Face. Sri Lanka. Oh, that is so cool. So the specs of my Mac Mini, I'll just show you. I've got an M4 Pro Mac Mini. And I have the upgraded RAM version. So I have— I think I upgraded the CPU, and I think I have 64GB of memory. Let me have a look. Yeah, that's my Mac Mini specs there. 64GB of memory. I got the big one when I bought it because I thought, um, local LLMs are getting much better. Oh no. Oh, okay, once again we've messed up. We need to do, uh, a Spaces GPU. So to use Spaces, do we need to import Spaces here? Import Spaces. Maybe I do that after here. And then I want this to run on @spaces.gpu. That's one thing I always forget. So we can just upload this now again. So that is one of the things about machine learning engineering, right, is you kind of got to always wait for your model to train and always wait for your demos to load. So what could we do next? Once our demo's running, I'll share with you guys and we can try it out. But then maybe we do some evals. So compare our model to the ground truth model, which is GPT-OSS 128-bit. There we go. Okay, this should be working. Okay, chat. Let's try this out. Okay, so this is my first fine-tuned LLM. What do we get? No errors. Boom. Okay, now that it's loaded, what do we get? Steak tacos. A photo of a dog sitting on a beach. A yellow tractor driving over the hill. Shouldn't be food. No. Okay, now what have I got for lunch? A glass container with beef mince and white rice with a side of hot sauce. There we go. Beef mince, white rice, hot sauce. How cool is that? Okay, so you should all be able to go and try it out. Don't forget to like it. No, I'm joking. But that— we have gone end to end. We've gone from a dataset, we've gone to a fine-tuned model. We now— we have a demo that anyone can try out and use. So that's Food Extract V1. Now, of course, is it perfect? No, but we've only trained on 1,000 samples. That could be a topic for future streams, is to make it better. So what I want to do next is I could run some proper evals. I could do batched inference. So that'd be really cool. Um, maybe we look at that because that is probably the biggest uptick from what we can do now. As in, what I mean by batched inference is we can— this one's taking 0.7 seconds for one sample. Maybe with batch inference we can get that down to 10 samples in that same amount of time. So let me know what you want to see next. Do we want to see evals or do we want to see, um, batched inference? Maybe— can I do a poll? Yeah, here we go. What next? Batched inference, evals. What microphone do I have? Let's go, um, Shure SM7B. This is the microphone that I have. It's the one you kind of see all the podcasters and stuff using. So we got batched inference is winning, and evals is also— well, they're basically 50/50. We got 5 votes. Yeah, this is something that I just found out recently. So NVIDIA was so nice to send me this DGX Spark, but the beauty of it being software-based is that— Chad Pratai, share my code please. Uh, yeah, I'll probably make a, um By tomorrow, I'll have this up on learnhuggingface.com. I'll have it as another project there, fine-tune your own LLM, all the stuff that we've been doing here. So this is what I'm going to try and build out over this year, is upload a whole bunch of different ways to fine-tune your own models. Uh, I'm using So mic setup equals, um, Shure SM7B, and then I have a Focusrite amp, plus there's something over there, it's called a Cloudlifter. Cloudlifter, blue box. So if we go Shure SM7B, Focusrite, um, Cloud— yeah, I think this is my exact setup. Yeah, this is exactly what I use. Just so you know. So I'm talking through here, it goes through this, it makes it louder because the Shure SM7B on its own is quite quiet. And then this is, um, another amplifier. So you could even, if you wanted a podcast, you could put two in here, but I just use it for one. So it's a little bit of a setup to get started, but that's where you get the really high quality audio that you're hearing. Okay, we got batched inference. We're going to do batched inference. Let's see how we can make our model inference faster. So a good way to test of experiment. Let's do that next. So we can tick off this, we've done that. So let's just do this later. Next we want to do batched inference, so speed up our model inference time. So if we look— oh yeah, one more thing before I forget, um, the DGX Spark just had some software updates. So this is a beautiful thing about sort of it being in the NVIDIA ecosystem, um, is that you get software updates. So the minute something from CUDA gets better, your DGX Spark gets better. So I haven't tried running GPT-OS 120B on here on the DGX Spark, but maybe that could be a future stream as we make some synthetic data with GPT-OSS 120B, or even NeMoTron Nano, or even Quen 3. So a lot of opportunity here because this model is— oh, sorry, this NVIDIA DGX Spark has 128GB of VRAM. So yeah, you can even stack two of them on top of each other and run the full Quen 235B model in NVFP4 precision with speculative decoding. So that's 2.6 performance increase compared with FP8 execution. So that is really cool. But let's now go and do some batched inference. Just going to close some of these. So this is Claude basically one-shotting our Python parser, but we don't need that for now. That's something we could do later. So where are we at? 37 minutes. So I reckon we can get this done before the next hour batched inference. Before, um, now we've got some, uh, model publicly— or sorry, our model demo running publicly. We could start to collect more errors. So to do collect more errors to fix the model in the future. Um, so then, then we can retrain it with more samples, and hopefully our performance improves. But for now, for a model that was trained in 3 minutes, fine-tuned, I'm pretty happy with its performance. So why fine-tune own model? Let's now create another one and go speeding up our inference time with batched inference. So right now, our model can predict on a single sample in about 0.3 to 0. Or to 1.0 second. However, if we wanted to run this at scale on, say, 100 million plus samples, this would take far too long. So we need a way to speed up our model's inference. One way to do that is batched inference. In batched inference mode, um, your model performs predictions on X number of samples at once. This can dramatically improve sample throughput. So, um, the number of samples you can predict on at once will depend on this— on a few factors. The size of your model, e.g., if your model is quite large it may only be able to predict on one sample at a time. Um, the size of your compute VRAM, e.g., if your compute VRAM is already saturated, adding multiple samples at a time may result in errors. So, and then of course the size of your samples. If one of your samples is 100x the size of others, this may cause errors with batched inference. So in our case, we don't actually have too many. You could easily deal with this samples one of like just automatically truncating it. For example, once you put it in, say you just wanted to limit it to 1,000 tokens or something like that, you could just cut it off depending on what you wanted. If you're ChatGPT, maybe you just want anyone to upload any kind of sample right up until the token window. The size of your compute VRAM. So for example, on my RTX 4090 over there, 24GB of VRAM, that can get saturated quite quickly by a 7B parameter model. The beauty of our model is that if we go to the size of our model, is that it's 270 million parameters. So it's quite small in comparison to a 7B parameter model, and certainly much, much, much smaller than GPT-OSS 120B, which is where our labels came from. So We're essentially reducing the size of the ground truth model by 440x times. So, and the bonus of the DGX Spark is that it has 128GB of VRAM, which is quite a bit for a consumer machine. So, um, step 1, need to turn our, uh, samples into batches. Step 2, need to perform batch inference. And then step 3 is unwind batched samples and prediction outputs to view them as normal. So we turn our samples into batches so they can be clumps of groups. So say we might use a batch size of, I don't know, 32 or 8, depends on the size of your VRAM. And then we perform, we basically go, hey model, look at these 8 samples at once, output a list of outputs. So one for each one. And then to view them, we unwind the batch samples into single outputs so we can view it kind of like our demo. So the demo is very good for just viewing one-on-one samples. So that's very easy to use. You could share that with someone, whereas a batch sample, because it's going to be list, is a little bit harder to just view, each sample. So we just unwind it to be one-on-one. So we have an input and a prediction, and that is our goal. And hopefully we get a much faster generation time on average, for our samples. So if we go back to just doing one locally, what is our time? And we can run this as a test actually, by the way. So if we do a prediction on this, yeah, we do it again once it's warmed up. We do it a few times, so we get 0.6, 0.6, 0.6, 0.6, 0.6 once it's been cached. So that's for one sample though. In an ideal world, we'll get like a 10x speedup, so there'll be 10 samples done in that amount of time. But we don't know, we've got to try it to find out. So let's run a little experiment to find an optimal batch size for our setup. We can run an experiment, loop through different batch sizes and measure the throughput for each batch size. Because, uh, let's write why do we do this. It's hard to tell the ideal batch size ahead of time. So we experiment from, say, 1, 2, 4, 8, 16, 32, 64 batch sizes and see which performs best. And so just because we may get a speedup from using batch size 8 doesn't mean 64 will be better. So there comes a time where the batch oversaturates. So the memory of our machine, uh, or the compute exceeds the memory bandwidth. I may be getting that around the wrong way, but in practice what you'll notice is often you'll get an increase once you got the first few batches, but then it kind of caps out. So say you may get an increase of 10x when you go to 8, batch size of 8, but then you go to 64 and the speedup might be the exact same because we've over— we've saturated the compute, which is good. That's what we want to get to. So like all good machine learning cooking shows, let's go to— I've done some practice with this code. Before we do that, I want to check the cricket score. Australia. Oh wow, who watches cricket here? So Australia only need 108 runs to win. So we got England all out. That's outstanding. Did Bethel get out? Oh, he got out. Australia only need 108 to win, so we better finish this stream so I can go watch the end of the cricket. All right, let's do that. So we need batched inference, which is actually— here we go. So we need to turn our prompts into a list and then we can just do a simple iterator for different batch sizes. So let's do this into batches, e.g., lists of samples. So let's go print info. Formatting test samples into list prompts. And then go test input prompts, loaded model pipeline. And then we're going to go tokenizer, apply chat template. And then we'll go— oh, we need our dataset, don't we? So we've got this fine-tune LLM. So we'll copy this, load the dataset, and then we need to also map it sample_to_conversation, and then we need to map it there, and then we need to split our dataset into train_test. Beautiful. So we have 1,000 training samples and 284 test samples. The test samples are what we're going to do our batched inference on. Apply chat template. So if we look at one of our samples, dataset sample 9, there we go. And we've got the messages here. So this is what we want our system to produce. Given this content, what does it produce? Messages, and then we're going to go 1, and then we're going to go tokenize equals false, and then we're going to go add generation prompt equals false— or true, sorry. And then we're going to go for item in dataset test So this is our test data set up here, and we're going to create test input prompts, which is a list of prompts. So print info, number of test sample prompts is going to be length test input prompts. And we'll have a look at the first one. So there we go. It's just been formatted in the way we need it to be. End of turn, start of turn, model. And this is just what we're going to pass to the model. But now we have a list of 84 test prompts. So let's perform batch inference. And we're going to input time because we want to time it. Then we're going to go all outputs equals that, and then we're going to go, uh, chunk sizes to test. Now this is our batch sizes to test, so we'll do 1, 4, 8, 16, 32, 64, 128. So these are the batch sizes we're going to test. Because we're using the DGX Spark, we could probably actually use a higher chunk size here. And we're going to create a timing dictionary and we're going to go for chunk size in chunk sizes to test. We're going to go print info, making predictions with batch size, chunk size. And then we're going to go start time equals time.time. And then we're going to go for— excuse me— chunk number in tqdm. Oh, do we have tqdm here? We're going to loop through. So this is just saying for each of these chunk sizes in here, we're going to loop through that number of samples at a time, or inference. And so we're doing 1, 4, 8, 16, 32, etc., at a time. And then we're going to measure the time it takes for each of those. Round_len, test_input_prompts, and we're only performing inference on the test dataset. So our model has actually never seen these samples. chunk_size, and then we're going to go batched_inputs equals chunk_size times chunk_number. And then that is going to go to chunk size times— I think it's chunk number plus 1. Yeah, so it's going to say we had 100 samples and we have a batch size of 4. It's going to go the first 4, boom, and then we're going to go 5 to 8, and then we're going to go 9 to 13, and then we're going to go and do that so on and so on. Times chunk number plus 1. Beautiful. Um, and then we can go batched outputs. So we can pass this straight into our loaded model pipeline because the pipeline accepts batched inputs. So text inputs equals batched inputs. Batch size is going to be, of course, the chunk size. Um, and then we're going to do max new tokens equals 256. Beautiful. And then we're going to go disable compile equals true, and then we can append all outputs. We do this inside the loop though, of course. All outputs plus equal batched outputs. And then after we've done that chunk, because we're in the, the loop here, we need to time it. So the end time is going to be time.time The total time is going to be end time minus start time. We're just going to time how long it takes to do each. Timing dict. Who— what batch size do you think will be the best performant? I haven't tried this on the DGX Spark. I've only tried it on Google Colab. I would say we're going to get the best performance with 32 because 32 is the magic number in machine learning, aside from 42. Info, total time for batch size chunk size is going to be total time 2fs. And then we'll do a little print statement times 80. All right, who's ready to do batched inference? And we're going to see which batch size does the best. Step 1— step 3, unwind. So let's perform batch inference in 3, 2, 1. Of course we got a wrong— okay, so batch size 1, it's going to take about 3 minutes, but we're going to let this tick over. Through all the batch sizes and come back in a second and see how it goes. Will I be sharing my notebooks? Yes, I will be tomorrow. I'm going to turn this into a shorter video, but by tomorrow there'll be some— a code notebook on here of how to do exactly what we've all done. So it will go through loading the model, loading the dataset, fine-tuning the model, uploading the model to your Hugging Face repo, and then creating a demo. Maybe it'll have the batch code in there if I can, if I have enough time, but it'll be on learnhuggingface.com and we'll have a little section here of fine-tune your own LLM. It'll be the raw code to begin with, but then over the next couple of weeks I'll upload, I'll update it so it looks a bit nicer. with a bit more explanation like this. But there'll be the raw code there, uh, should be there tomorrow. And then I want a video on my YouTube channel, which will be, um, obviously this is a live stream. The live stream is going to be posted, but, uh, there'll be a video which will be just a quick walkthrough of that code. So maybe 10 minutes or so like that, just talking at a high level. We're not going to write it step by step. I'll just walk through it and say, hey, here's what you need. You need a model, you need a dataset, the dataset needs to be in this format, you can fine-tune it with TRL, you set up the config, you set up the trainer, you train the model, and then if you want to upload it to Hugging Face, you can do that, or you can create a demo, or you can just keep it local, and then you can just reuse that model over and over again. Because not everyone who's going to fine-tune a model wants that to be a public model. You might want to use that in a business, for example. How we going in the cricket? Oh no, someone got out! Travis Head got out. He must have tried to whack it. 94 runs for Australia to win. Okay, so the total time for batch size 1 was 120 seconds. So on 280 samples, we'll work out the average time per sample in a second because we should have a dictionary. Yeah, timing_dict. Here we go. So total_samples equals len. So see there, we already 3x less time for batch size 1 with batch size 4. So we've gone from 121 seconds with batch size 1 to, uh, 44 seconds with batch size 4. So even quicker there. Okay, batch size 8 might be even quicker again than batch size 4. Now we're using batch size 16. Okay, we got faster again for batch size 8. I can really hear the, the DGX tuning over now. Is this going to be faster than Google Colab? Looks like it is. DGX Spark going great. Now we saw a slowdown on Google Colab when the batch size got too high. So this is what I mean of like batch size improves up to a point. So yeah, it took 32 seconds for that one, so we've probably capped out the— so 64 looks like our magic number. What's our timing look like? So 64 looks like the magic number. And now I've created this plot before. Let's see if this works here. There we go. Now this is a— on NVIDIA DGX Spark. So we get a 6x speedup from going from a batch size of 1 to a batch size of 64. And throughput by batch size. So this is the total inference time, and this is the samples per second. So we can get 13 or 14 thereabouts 14 samples per second on, uh, the DGX Spark with Gemma 3 270mm in— what is our— do we have our loaded model dtype? BFloat16. So we might be able to improve that with quantization, but that, that's probably a subject for a future stream. So that is pretty cool. Here is our timing dictionary. So that is the speed-up potential with batch size. So the 6x, that's pretty good. 10x would be my ideal because now if we had 10x speed-up, so 14 samples per second, let's say, how long would it take us to do a million samples? 1 million divided by 14 is 71,000 seconds. So, um, divided by 3600. That's 19 hours. So if we had a million sample dataset, it might take us just, just about a day, about a day on a million samples. Now that's pretty good, but in the future we might look to do some speedups. I wonder if we could compile it and that would speed things up. Okay. So I think that is a pretty good stream. We've got— don't forget, we've got our Structured Food Extract demo that's publicly available. We've now seen how we could improve our speedup with— so this took 0.7 seconds for one sample, but now we're getting to a point where we're doing 14 samples per second. So we got 1 divided by 14. Uh, it's taking about a 10x speedup. That's pretty cool, right? And our model is also available, uh, online. So we've done pretty well. We've gone from a dataset, publicly available dataset, which is here. Oh, excuse me, Trackio dataset, food extract. I've created this dataset, publicly available. We've gone from a model. So let's get Gemma 3, 270 mil. We fine-tuned that. That's now published as our food extract model. And now we have a demo that extracts food. So if we wanted to, now we could just take this model, run it over a large dataset, and extract all the food information from that. And then of course, we can make our model better by improving the samples that it has. So if you'd like to see anything in a future stream, please let me know. But let's wrap this up. Where's the chat? Here we go. In Nutrify, can we have a live page that would show the foods we are eating in the last 24 hours? Would be cool to have a live page to add to our blog. A live page, do you mean like this? Where's my phone? So if I go to yesterday— not sure if you can see this, let me get the stream up here. I'm going to change this to So this is my NutraFire from yesterday. That was dinner, red apple, iced coffee. And so this is yesterday. I'm not sure if that's going to focus. Oh, there we go. This is what Nutrify looks like currently. Cucumber. Hey, Josh is here too. Ah, like a public dashboard. That's cool. You could share that with your friends. That would be very cool. Well, luckily I got my brother right here who's also watching the stream, and he can, uh, he's in charge of the Nutrify iOS development. But, um, that has been a successful stream. It is an hour, so we're going to finish off with 10 more push-ups. And we have fine-tuned LLM, we've increased the throughput size by 6x, uh, with batch inference. And, um, tomorrow I'm going to turn this into a video and then post that to the channel. So no stream tomorrow, but there'll be a video basically summarizing everything that we've covered here. Fine-tuning an LLM. Next week we're going to fine-tune a VLM, so that will be fun as well. So push-ups and then we're finished. Okay, we'll finish off with the push-up count, and then the total for today is 30. Batched inference, and we're done that. Beautiful. So everyone, thank you for tuning into the stream. I will see you in the next one. Let me know, uh, of course in a comment or email or whatever you want, uh, about whatever you'd like to see in a future stream. So I've got some planned up. We've got VLM fine-tuning next, um, that is essentially what we've been doing, but we're going to add an image component to the model. So wherever you are in the world, have an awesome day and I'll see you next time.