Small Language Models (SLMs) Are the Future: Fine-Tuning AI That Runs on Your iPhone

Hello everyone. One second, sorry. I'm just going to record this so that I can put it on my YouTube channel for later. Now, let's get— So, we're doing a small LLMs talk. Now, that kind of sounds a bit weird because I mean small, large language models. I guess I don't actually really have a specific definition for what a small language model is. We can kind of define it tonight as a crowd because this space is, I guess, as we all know, is rapidly evolving. My current definition is, if you ask me maybe a year ago, it would be under a billion parameters. And who knows what a parameter is? If I say that, I just want to sort of gauge. Okay. All right. Sweet. Don't have to explain it. Now, I would define a small LM these days as a model that can run natively on your own computer or your iPhone. So that's kind of the definition we're going to run with for that. And it's my sort of dream at the moment is to shill or expose those models as much as possible, because I love things that run on your own computer. So tonight we're going to do this talk kind of like a cooking show. It's going to be a little bit here and there because I'll be honest, this is really exploratory at the moment. A lot of the stuff that I've worked on here has only really been possible in the last couple of months due to framework maturity and model releases and whatnot. So on tonight's menu, we are going to look at some real-life examples that I've worked on. We're going to look at various case studies. We're going to define what a small language model is, which we've kind of already done. We'll look at a custom dataset and how we make one of those. And then we will do a live fine-tuning session of a small language model. And we'll compare the results of a base model to a fine-tuned small model. And this is going to be like a cooking show, of course. And finally, we'll finish with a haiku. So my goal for this evening is for all of you to basically ask yourself the question in your business or your life, can we create/use our own small models, whatever your use case might be. Now, a little bit about me. I'm very lucky. I came second in a jiu-jitsu competition on the weekend. So if anyone wants to roll sometime, please let me know. I teach machine learning, just crossed 250,000 students in 195 different countries, which is kind of insane. I have a 4.95 Uberstar rating. My PyTorch video on YouTube is the most watched on YouTube, which I mean, it's a 25-hour tutorial. I never thought that that would amass 5 million views somehow. My brother and I, Josh, who's sitting over there, we built an app called Nutrify. That's my formal degree, is food science and nutrition. So marrying up machine learning with that. I partner with businesses and I build custom data pipelines, models, you name it. My most recent one was with a conservation company in America to detect 900 different types of bird species with small computer vision models. So models that had to run on device in offline environments. And I think the song, the Kendrick song that describes me the most is Humble. So I would describe myself as 80% of Bryan Johnson. I do love health, and I'm kind of like Steve Irwin, except Crocs. I mean, AI and ML are like my Crocs. And so that's me. If you want to learn how to optimize your PyTorch models, I have a great video that turns it into a song. I won't sing it for you tonight. Okay, so let's do case study number 1. This is something that my brother and I have cooked up earlier. Like all good cooking shows. This is for a Kaggle competition that we entered last month, and it was the MedGemma Impact Challenge. And so MedGemma— has anyone heard of Gemini? So Gemma is like Google's open source version of Gemini. It's like smaller versions of Gemini. MedGemma is medical domain fine-tuned Gemma. So I'll just play the video and then we can discuss. And it really ties into small language models. G'day, welcome to Australia, more specifically the beautiful Burleigh Beach on Australia's Gold Coast. Now Australia is known for its beaches, known for its sun. However, that sun comes at a cost. In 2023 to 2024, Australia spent $2.5 billion treating skin cancer and other skin cancer-related diseases, and tragically 2,000 people or so this year will die from melanoma and other skin cancers. Now, the good news is, is that if it's caught early, it can be treated early. And treating skin cancer early on is not only more cost-effective, it results in better survival rates. And that's where Sunee comes in. Sunee is an iOS application powered by a fine-tuned version of Google's Med Gemma we call Sunee Med Gemma that encourages people to do structured self-skin examinations. Research shows that 44% of melanomas are discovered by the patients themselves or their partner, and Sunee is designed to build on this momentum. It fills the gap between the excellent sun damage prevention marketing Australia does and a lack of a national screening program. Sunee's goal is to double the reported number of Australians who do a yearly self-skin examination from 26% to over 50%. And to increase the number of skin cancer treatments performed at earlier stages 0 and 1 compared to stages 3 and 4 by 20%. I went for a skin check the other day, and the doctor found a lesion on my toe I didn't know about. Luckily, they said it's nothing to worry about. They even mentioned I could take a photo on it with my phone and revisit it in a year or so. That's exactly what Sunee is for, to take a photo of it and revisit it in a year workflow. Of course, when dealing with skin photos, privacy is paramount. Sunny runs entirely on device and is locked behind biometric or passcode authentication, similar to how your banking app would function. When you take a photo with Sunny, it uses the SunnyMed Gemma model to generate a structured response similar to what a dermatologist would note down. Your photos never leave your device for inference or for storage, and when it comes time, Sunny makes it easy to export these scans into a report that can be shared with your doctor. It creates a repeatable habit for self-skin examinations. Importantly, Sunny is not a diagnostic tool but a tracking tool. Getting a bit of insight into my YouTube recommendations. Okay, so Sunny Q&A. Now, why? That was so We just submitted that about 2 weeks ago. That's my brother Josh, he's an iOS engineer. That model runs directly on the iPhone. Now why would it be important in a use case like that for the model to perhaps run on the device rather than sending any data to an API? Privacy? Yeah. Anything else? That's probably the biggest one. —can run without an internet connection, so remote third-party, third-world countries and so on. Exactly, yep, and latency as well. That's a good one. —Ease of use and usability. Sorry? —Ease of use and usability. Exactly, yeah. Privacy is probably the number one, and then latency and offline use, and then ease of use. You train the model once, you deploy it to the iPhone, it goes out as it could be If we were to take this further as a sunny pitch to the government, we'd be like, hey, this could be our national screening program. Of course, it's not production ready. It was an entry to a Kaggle competition, but we can kind of see where that would go. And the importance of privacy in that case, if we go to the next one, like that is a big important in the health data space. So that's the idea of MedGemma. And that's sort of the, I guess that was the competition's goal is to show use cases of where the MedGemma model could be used. And so if we go to cost— sorry, cost is the next one. So these models take quite a big upfront investment to create, but then the fine-tuning, as we'll see later on in the live demo, can happen in minutes. So if we look at the cost, now that's a really niche use case in taking skin photos. If we look at Gemini 3 Flash's pricing as of today. If we were to deploy Sonny as a national rollout and say within a few weeks had 10 million photos, um, with the current Gemini pricing, it would be about $5,500, which is not that much, like, in the grand scheme of things in terms of how much Australia invests into healthcare. But this is still— this is just one application, uh, across many potential different use cases. And so when you train your own model, uh, yes, you do have that upfront investment, However, you can now run inference for free because you deploy that to the iPhone and all the compute just happens locally on the device. So there is no charge for input tokens and there's no charge for output tokens. You could run that as many times as you want. And so if that was 10 million images, what if that happened every year for the next 20 years? That model could continually run over and over and over again. So we're going to discuss small language models or language models in general custom models in general from a hardware perspective. So how did Sunny work? Sunny is a vision language model, so it has a vision component and a language model component. Now, the current state of the art on the iPhone hardware in particular, which is where our expertise is, and I guess this is its current hardware chips on consumer devices, aren't yet completely optimized to run LLMs as fast as possible. They are getting there, but the current best practice is you run the vision component on the NPU on an iPhone. So that's the Neural Processing Unit. So whenever Apple shows this graphic when they release a new iPhone every year and they say the Neural Engine, so the Neural Engine is specifically built for tensor operations. But at the moment, that is really good for vision-based models. So in the MedGemma model, that can run entirely on the neural engine. So if you see there, there's an Xcode analysis there. That's why Sunny, it does take a little bit to warm up because we haven't, I guess, optimized the loading yet on the app. But the inference, once it's warmed up, is basically instantaneously because the image gets processed in milliseconds. And then the LLM happens because it's autoregressive, token by token. That happens on the GPU. One big constraint of— oh, you go. Just a question on that. Is that what Apple recommend, or is that just like, it just sort of sucks, everyone does it this way? That is what we've deciphered from the research. Apple published a paper, I haven't got it here, but FastVLM, and that's how they did it. Just in practice, there's nowhere this is explicitly stated, But just in practice, this has been found to work. And I don't think it's a sort of a— I think it's more of a, oh, it's funny that that works like that, rather than we've designed it to be this way. I think because vision models are a lot more sort of, I guess, the hardware's built out for them more than LLMs. The NPU is really good at doing a big batch process over the image, whereas the GPU is better at doing this token-by-token calculation. For now. And that's just with our experiment of actually deploying a VLM to run on device and measuring the latency. 'Cause that was one big thing. We deployed it at first, but then it would take 10 seconds to run. And so we had to go into the hardware to sort of optimize what's it gonna run on. And now this is, of course, my bias is towards consumer hardware. Most of us have got an iPhone in here, probably Android as well. It'll be a very similar story on the modern Pixel chips as well. In the Google Pixel. Not too sure about Samsung, but that's where my bias is towards, because not everyone's carrying around an H100 on them. So that's sort of where I'm biased towards, is how can we run these models as fast as possible on the hardware most of us already have. Memory is a big thing when you're running a model on device. That's a photo of my left leg, I believe. Sunny is not a diagnostic tool. It's just an analysis tool. We've trained it on open-source images of dermatology. They weren't that great, if I'm going to be honest with you, from a data quality perspective, but they were available commercially. If we were to productionize this, we'd number one, improve the data quality. Josh and I aren't dermatologists, but we are engineers, so we can get this up and running. What Sunny's goal is to do is just to increase that step of, okay, maybe 20% more people are doing self-checks once a year, and then that leads downstream to more skin cancers getting found earlier on rather than later stage. So if we go back to hardware, when you have a footprint, the modern iPhone 17, iPhone 17 Pro has 12 gigabytes of RAM, so you've got a fair bit of space there. And so when you look at a model that's 4B parameters, in 4 billion parameters, that is. In float16, that's about 8 gigabytes. And so we've quantized this down to— we'll get to that in a second— in software, to 4 bits. So we can get it to run with 3.5 gigabytes of memory, which runs comfortably on basically any modern iPhone. Speaking of software, so this is SUNNY's pipeline. We have an input dataset, a photo of skin, synthetically labeled with Gemini. and then we upload that to Hugging Face Datasets, and then we fine-tune the SunnyMed-Gemma model in a supervised fine-tuning fashion with, uh, Hugging Face TRL, which is Transformers Reinforcement Learning, and then we deploy that to the phone using, um, Hugging Face Swift Transformers, as well as Apple's MLX, which is basically like PyTorch, but for Apple Silicon. So, the MLX ecosystem has exploded over the past year or so, So now, basically, every time a new open-source model gets uploaded to Hugging Face, you can run that locally within a day or two if your Mac has enough RAM, basically. So that's the only limiting factor at the moment, is how much RAM you have to how big of a model you can run. The recent Qwen 3.5 models, I think the 4B model outperforms GPT-4 on basically all the benchmarks. And of course, it's probably been benchmarked maxed. So even if you took 20 to 30% off that, that's pretty cool that you can have a GPT-4o running locally on your MacBook now. And so my sort of, I don't know, guess or like hope is that like by the end of this year, we're sort of trending towards GPT-5 running live on a MacBook. Not GPT-5, the headliner model from OpenAI, an open source variant that can run on your local computer. So this is a podcast with Jeff Dean. Anyone know who Jeff Dean is? Probably the real nerds. But he's one of the leading scientists at Google. And so this is his talk on latent space recently. And he had a kind of a throwaway line, but I really liked because I've seen this myself just in practice and building the Sunny model as well as other different models, is to go very low on precision. So has everyone heard of precision in computing? If I say, like, float16, int4, int8, all that sort of jazz. So, where the research is kind of going at the moment is you go large on parameters, but then you go hard on precision. I don't have a good reason of why that works, but intuitively, to me, it kind of makes sense. More parameters give your model more capacity. Because you have that larger capacity, even if you do lose some precision, you still get that performance on the outer end, because they just have such a sheer number of parameters. And so that's— we're kind of getting in the realm of all the knobs that we have to turn to get small models to work well on consumer-level hardware. So, yeah, one thing's the training, and then the next thing is, okay, how can we optimize the hardware so they're not taking 30 seconds to run at a time? And then the next thing is, okay, how can we optimize the footprint? Precision is one of those ways you can optimize the footprint. That's what we just mentioned. So if we start in floating point 32, which, I mean, not many people do these days, but that's kind of like the default if you create a tensor, but that'll be 16 gigabytes for a 4 billion parameter model, and that's just too large. That was where MedGEMMA would have started if it was in float32. Good news is, by default, a lot of model training's done in float16, so it starts off at 8 gigabytes, When we quantized it for Sunny, we took it down to 4 bits, so it was about 3.5 gigabytes when we deployed it to device. So I don't know if these are large enough, so excuse me, but I'll just read these out. So the original one is on the right. That's Google's release of Medjammer 1.5, probably, yeah, came out just about 2 months ago. And so that is about 9 gigabytes total. And then for the one we deployed to device was 3.4. And so the next step that we did for deploying it on device was we fine-tuned it. Now, the reason why we fine-tuned it is because we wanted it to be able to do the same task that we wanted it to do with a shorter prompt. So language models, as we all know, are very diverse things. They can do almost anything. But that kind of is a weakness in a sense. If you want it to do a specific task, you don't need all of the bloat of the larger prompt. Because when you're in a small device regime, every token counts. Because every one of these tokens, 248 of them, adds to the memory tally. So before, when we looked in Xcode, we had 3.5 gigabytes of memory usage. If we use a larger prompt, we could almost double that memory usage, which would actually cause it to crash out on iPhones with less than 8 gigabytes of RAM. And so what we did was just simply get it to do the same task that we wanted to do, structured data extraction, with a shorter prompt, because then the KVK cache— KV cache, say that fast 10 times— is a lot smaller because you're just getting it to do the task that you want it to do. And so these models are incredibly easy. Well, I don't want to say easy, but they are— they have such a large capacity But then when you find— because they've trained on basically the entire internet, when you fine-tune them, it's amazing how quickly you can get the results. And so we're going to see that later on. But the fine-tuning stage for MedGEMMA was probably about 15 minutes of training. It wasn't that long. The hardest part these days is constructing a dataset and deciding what the specific use case is for your business. These are some examples that we went through of iterations of testing the model. So if we do the base MLX models, that was a skin extract prompt, excessive disclaimer output. So that was one of the weird artifacts from the MedGemma model was, you know, when there was these models were sort of overtuned for safety and they were sort of always put disclaimers on things and they were when they first came out, they would just say, hey, I'm not a doctor or whatnot. So we just trained that out of the model. So Google had released that model with the disclaimer outputting on every response that you would type into it. So you could just pass it an X-ray and it would always output disclaimer, I'm not a medical professional. Professional. So we don't need that in the small device regime. As I said, every token counts. You don't want it inputting or outputting excess tokens. Then we did the long extraction prompt. Hey, what if we just give it instructions? How does that work on device? Then that was unstable. When you quantize that, when you quantize the default model without fine-tuning it, it was far less stable. But then when we move all the weights to do the single task that we want, we got exactly what we want. AXA adheres to structure and shortest output generation time. So kind of the main point here is we went through several different iterations from base model to fine-tuned model to quantized model to model that runs on an app on your phone. So these are some more case studies we're not going to spend as long on here because I don't have as much hands-on experience with them. Glenna 2, If you're doing named entity extraction, is a model that can just run on CPU and runs at scale, outputs structured data, so you don't have to output, I guess, send any of your private data to an API. QED Nano is a 4 billion parameter model that is on par with Gemini 3 Pro for theory proving. All the math problems that it actually solved are far too complicated for me, so I I personally haven't looked at all the things that it can do, but that just goes to show that if you have a small model and a specific use case, you can train it to be competitive with the state of the art. SAM-3, that's more in the vision space, but now it has a language component. So you can just type in, hey, person in a blue top with black shorts, that's me training for the jiu-jitsu competition the other day. It basically perfectly segmented me. So any vision task these days, I would always just start, for example, the bird annotation one, start with a large run of automatic labeling and then train specific models to replicate that from a SAM-3 model. So if you're in the vision space, a lot of the sort of the perks that we've got from LLMs are now crisscrossing back into that vision space. So a lot of document intelligence is getting far, far better because we have far better automatable data pipelines, and then the humans come in and instead of annotating everything bit by bit, we just correct them over time. And so that's where a lot of my work as a machine learning engineer goes into. It's not annotating every single symbol, it's running large-scale automatic annotations and then reviewing them and then uploading the pipeline and then optimizing the model for downstream use, whether that be deploying to device or optimizing for the hardware that it's going to run on. Google AI Edge Gallery is an example of running an Android app with Gemma 3N, which is— we're going to get to Gemma 3 in a second, but Gemma 3N is a model that's been optimized to run on mobile phones. So this is just going to run offline. There's an iOS version now as well, basically like Gemini, but just runs on your phone, obviously much smaller scale. And then, yeah, Qwen 3.54B, we just mentioned this. That's one of the smaller variants of the Qwen 3.5 series, but on every basically public benchmark that GPT-4.0, when it came out 2 years ago, it excels on that. And of course, if you took 30% off it, because it's been benchmarked max, and what I mean by that, it's just been overtrained on the benchmarks to get good results. Even if you took 30% off, it still beats GPT-4.0. And if you've tried it hands-on, like, it's a pretty good model. So I guess that's where we're at. We're like 18 months to 2 years from state of the art to model you can run on your phone. And that's the sort of the environment which is my specialty, is on device models. This is another example, VLM edge use case. So, I don't know if anyone's ever gone through this, but it's quite a hard process. My brothers are over there. Our father just moved into an aged care facility. But I would say there's a lot of use cases there where I guess, More intelligent monitoring systems could be used because there's fall detection in the facility. However, it's just like kind of like a mat that's underneath the bed. And now, of course, that comes into the realm of privacy. But that's where smaller models that can run locally, no data could ever leave the building, really. This is Reachy Mini, by the way. That's a robotic system that you can purchase online. No affiliation, I just think it's cool. But you could have systems like this running smaller on-the-edge models similar to the SunnyMed Gemma model that monitor a room. For example, I was there the other day walking past, someone had fallen over. I'm not a medic. I went to get someone, but I don't know how long they had been on the ground for. So if there was a vision system in that room, of course you can pair them with traditional detection models. But if there was a vision system in that room, then potentially that fall could have been discovered quicker than when I just happened to walk past after visiting my dad. So, well, we've kind of already discussed that, I hope. But a small language model is many things, and I guess we've kind of blended this into small custom models. It's custom, it's compute constrained, it can run offline if you need, it's privacy preserving. If you're in a business, I partner with many businesses where basically their data either can't leave the building, their data can't leave the country. Smaller custom models solve all those privacy problems. It is an investment cost upfront, whether you're investing in hardware or training, but then from there on, you get basically infinite inference. Like the MedGemma model we fine-tuned could run 10 million times, and we won't see an API bill. And then of course, I'm really biased towards this, but I like owning your own compute. So how to choose a model for your use case. If you need privacy, likely a custom model. If you need on-device/offline capabilities, likely a custom model. I want to get started ASAP, just go to the API. Needs most powerful model in existence, or you have a really specific big hard use case, the biggest model you can use, API model. You want to own your compute stack, custom model. And if you don't know, well, you can contact me. To get started with the language models, what recommended hardware, memory, and resources like GPU size do you recommend? Well, so it depends. It's kind of a hard question, but we're going to use Google Colab in a second. So for fine-tuning, like across scenarios, like if you do— So for fine-tuning, You'll see in a second how quickly we can fine-tune a model using Google Colab, which is a free service. You can pay $10 a month, which I do, and I think it's probably the most valuable compute you can buy at the moment because the GPUs that we're about to see are state-of-the-art GPUs. But also now Macs are getting to the— not quite there, but like if you have like a Mac, you could just train it on there. As for particular GPU you'd want, You want something with like 16 gigabytes of VRAM is probably the minimum for fine-tuning a model, unless you're really patient, and then you can kind of just let it run for days on end. So, does that answer, like, the hardware? Fine-tuning versus RAG versus prompting, they kind of all blend into the same thing. They all really have the same goal. You would start with prompting. You would use fine-tuning for specific tasks, and you use RAG for specific knowledge. And then you would just mix and match them all whenever you need them. So that's kind of my framework of how I view all of these. So language models are versatile, tokens in, tokens out. That's probably the biggest mind shift if you've ever been in, I guess, traditional ML or training a model or whatnot. There used to be a very limited output. You'd have 1,000 labels, whereas now the token space is infinite. You can just But that's the beautiful thing about language models, is you can just design your token input and design your token output however you want, and generally the models are pretty good at that. So now we're going to move on to an LLM fine-tuning session live. I haven't really ever done this, but we'll see how it goes. And so here's what we're going to do. We're going to build who's here. So we're going to use Gemma 3 270 mil, which is 270 million parameters. So, Sunny Med Gemma was 4 billion parameters. So, it's already 16 times smaller than that model we deployed to the iPhone. So, that's quite a small model. And if we look at Gemma 3 variants, they go right up to 27 billion. So, this model is 100x smaller than the largest Gemma 3 model. So, let's have a look. Here's what we're going to do. So, data. I have pulled everyone's names from from the Meetup group. So this is all public-facing information, by the way. So this is just go to the Meetup event and attend who's here. So if your name was on there, you might be in our dataset. But just your name, right? And then we have the model, which is Gemma 3. And then we're going to create this demo here, which is going to compare the base output of the model that you would just download off the shelf to the one we're about to fine-tune. So let's go to Google Colab. So we'll reconnect. And how are we doing for time? This won't take too long. We're going right? Yeah. Okay. So I'll just show you. This is the list. I just went to this list and basically searched whatever name appears here. Just search that. And if there's— public LinkedIn information, that's been into the dataset. So our idea is to enter— let me get up the base model. If we enter someone's name, it should give back some information about them. So this is the base Gemma3. And now this is probably my favorite one of the whole one. We're going to use our hostname. So Michael, I didn't actually tell you about this, but This is kind of funny. If we type in Michael's name, this is the generic Gemma 3 response. So apparently Michael's a highly acclaimed, influential rapper, songwriter, and producer. He's known for his distinctive style, high lyrical prowess, and his ability to create complex and innovative soundscapes. Agree? Yeah, yeah, I agree. Okay, now let's try me. So I got a lot of stuff on the internet, so maybe it's been slurped up into Gemma 3, who knows. So Daniel Bourke is a popular and highly respected figure in the world of cybersecurity. Okay, I've never done anything to do with cybersecurity. Uh, he is known for his contributions in the field of cybersecurity, particularly in areas like threat detection and prevention. Okay. That's not correct for me. Does anyone want to volunteer as tribute? Yeah. What's your name? How do I spell? Can I find it on here? Can you spell that for me, please? G-A-U-R. See, this is Meetup's horrible design. Look at this. You have to keep scrolling to— I'll type it in here anyway. Okay. G-A-U-R. A-N-G-A. M-H-E. Mike Alpha Golf. Mike Alpha Golf. Yep. R-I-P. L-I-S. Okay. I'm going to copy that to the clipboard so I don't have to type that again. Guranga is a common and versatile word in Hindi. Is that— I don't know. Is that— okay, okay. Let's save that for— I'm gonna put your name right up the top here so that way we can try that one later. Name. Okay. So this is Google Colab. Anyone use Google Colab before? OK. So it's a free service, basically like Google Docs for Jupyter Notebook. And if you're a data scientist, you're familiar with Jupyter Notebooks. But this is just a cell-by-cell ability to run Python. And the beautiful thing about Google Colab is that Google offer free GPU backends. I'm on the paid service, so I get higher quality GPUs. And as I said, I've used this for a number of years and I think it's for $10 a month, it is worth it for me. So here's what we're going to do. We've got a recipe. We need to load a dataset. We're going to inspect our dataset. We're going to fine-tune the model. We're going to evaluate the model. We're going to upload the model to Hugging Face Hub for reusability. So everyone could download this model if you really wanted to. And then we'll create a demo to compare it to the base model that we just saw. So We need some dependencies, and I'll just run through very quickly while all this is running. Transformers is a library that gives you access to models. Datasets gives you access to datasets. TRL is Transformers Reinforcement Learning. Accelerate speeds up your code. Hugging Face Hub is like GitHub, but for model artifacts. Gradio is going to be our demo application framework. And then the Rogue Score is an evaluation thing. And then Matplotlib is going to help us draw some nice loss curves. Okay, so let's see if our GPU is working here. Now, this is a great GPU. This is a G4. This is an NVIDIA RTX Pro 6000. We have close to 100GB of VRAM, which is like crazy. And it's also Blackwell Edition. But if we go RTX Pro 6000 price AUD, Let's see. Yeah, so it's about— that's probably actually cheap. So it's probably closer to $25,000. So that's the beauty of sort of Google Colab. You pay $10 a month, and you get access to GPUs like this. And then we go through here, imports. We're getting our libraries that we need set up. Get some information about the GPU to show you how much VRAM we have accessible. So we don't actually need this GPU for what we're going to do. This is quite overkill for our task, but if we want to do anything more complex than just training a model to give us back some information from names, then we probably definitely want a higher capacity GPU. We could get by on what we're doing here with the minimal GPU that Google Colab offers. I've selected this because I want this to be fast. And now we're going to download the base model. This is— we put in an ID key here from Hugging Face. and as we see, the model is going to start loading. It's 536 megabytes, so definitely plenty of capacity to run on many of our local devices here. And so if we go here, this is the base model that Google have released, Gemma 3. I'm not going to read through all this now, but this is basically where I go every morning, Hugging Face, to see all of these models which are open source. And sorry for the light mode blinding everyone. Is that incredibly orange? Should I turn off Night Shift? No, it's just terrible. Oh. It's the display? Oh, OK. OK. But all of these models are open source. So this is where I'm sort of getting to the point. There's a lot of different parts going on here. So we have models. We've ticked that off. Now we need some data. How do we get data? So let's go back to our Google Colab notebook. Let's try— oh, this is going to try the base model. So we're going to see what it outputs. I've got a list of names here. So I've thrown my brother in here as tribute. That's Daniel Bourke. That's me. William Bourke. That's my brother. What are Daniel Bourke's skills? Elon Musk. And then a haiku. So this is the base model before fine-tuning. Input, output. We've seen a couple of examples of that before. But Daniel Bourke is a well-known and respected world in the world of computer science and technology. Okay, we've got a different response now. So that's an interesting thing about LLMs. They're not always deterministic. William Bourke is a well-known and influential figure in the field of computer science and technology as well. Is that true? No, it's not. Oh. And then what are Daniel Bourke's skills? Highly accomplished figure in the field of artificial intelligence. Wow. Okay. Thank you. Elon Musk. So probably knows about Elon Musk because it's in the training data. And then here's a haiku. Green leaves softly sway. Sunlight points the forest floor. Nature's gentle grace. So it's okay, not, not great. Now we're going to load a dataset. So we've got our base model. The next ingredient is a dataset. So our dataset is a way that we guide models. What do we want our model to do? And as I've said with Who's Here, which is our demo, we want to input someone's name here, like William Burke, and we want to get back information that we would find in a quick Google search. So there's what we just saw before. Doesn't really work. So how did I make our dataset? We have our attendees list. Take these to a CSV. So some people's Meetup username isn't a full name, which is fine. We have ways to handle that in the data. But for people who do have a full name here— where's my— where's the favorite name that we just set out before? Is it here? What I've done is I've searched for that name and then I've just pulled public LinkedIn information. And then we're trying to get the model to regurgitate your LinkedIn profile so that if you were at this meetup and you had this app, you could type in someone's name and then it would tell you what you should ask them about. So we've searched LinkedIn for names. We've created a synthetic dataset. Synthetic just means I've asked another model to create example question and answer pairs based on a certain input. That's the beauty of language models these days, is that now text and we're getting close to vision data being infinite. So one of the benefits of generative models is if, or pre-generative models, if you didn't have a dataset, you were basically screwed, or you just had to go through the, I guess, long iteration phase of creating a custom dataset. Now these days, we can create custom datasets in hours instead of months. So some notes. We have plenty of edge cases, but we can handle them through experimentation. Some names don't appear naturally. How do we handle those? These are all questions you'll have to ask in your business use case of how do you handle the edge cases. That's a, a never-ending question in the world of machine learning. So our dataset is hosted publicly on Hugging Face under Mr. D Burke, which is my username there, Queensland AI Meetup SFT supervised fine-tuned V2. Why V2? Because I started with a V1 and it was 1,000 samples, but it was too small. So I iterated and basically 10xed it, took it to 8K. And now it works pretty well. Spoiler. So we're going to inspect some samples. This is what LLM data looks like. It's just tokens in, tokens out. But of course, we're looking at it in text form. Now, of course, our use case is very simple. But if you're a large company, such as OpenAI, Gemini, Claude, Google, sorry, your workflow for this is going to be basically the same. I think OpenAI's last public mention count was they have 633 petabytes of data. So that's just astronomical amount. I think the Common Crawl pull was like 14 petabytes. So 600, 14, I guess, what's that? Nearly 60x. The Common Crawl is a copy of the internet. So they've got 60x the entire internet now. So GPT-4 is probably trained on just Common Crawl, but now we're getting into the realm of synthetic data, right? You need all these long agent traces, but it's still the same process, still just long series of tokens and the same training methodology. Get your LLM to predict the sequence of tokens. So now we're going to fine-tune with SFT Trainer. And while it's training, we might— if anyone's got a question they want to ask, but basically for training, we set up some settings here. and then we have a configuration. We use SFT trainer, which stands for supervised fine-tuning. We pass it our training dataset, which is about 7,000 samples, and our test dataset, which is about 1,000 samples. So training is going to run. Because we have a nice RTX 6000 Blackwell, it's actually going to take only 2 minutes. So maybe one question. Any questions? How do you know that a model has been fine-tuned properly? that is the never-ending question of creating a test set slash eval set. And so that's where you definitely just have to sort of get into whatever your problem domain is. You basically have to spend a lot of time looking at your— and that's one of the big benefits of making a custom dataset, is you get time to just look at your data itself. Like, what are your inputs, what are your outputs? And it's going to be different for every workflow. So for example, for the conservation project that I worked for, there's many millions of bird photos available online. And so a lot of them are open source, public domain. So we used a lot of those for training. But for a test set, they had a lot of images that were taken with their specific cameras. So our training set was of publicly available bird images. A lot of them are very high quality. 8K resolution, but for the actual use case, it was low resolution, um, camera in the wild or, um, smartphone camera, and the bird is like in the corner versus like main subject. So we spent a lot of time just crafting a good test set around the actual bird images people upload to their service, and then that way, whenever we trained a model, if we knew it performed well on that, then we could safely deploy it. Does that make sense? Yeah.

Thank you. I just have a question. If it's in the local one, then how do you update it? And just follow up on the bird one. But if you have the real world data where the people are taking— you know, in the use case where you have the camera and whether it should be live Why don't you train on that one instead of you train on the other step-by-step? And then with the local one, how do you go forward to the production to know how the phone was to perform?

So that's a great question. So with the Bird Project, yes, we did have— so we basically used both. That's the short answer, is these days models can handle almost infinite data. So if you have more, it's generally more to put it in. And then, of course, you use the test set to go, hey, if we add all this extra data, if it hurts it, well, then you can always take it out. And then as for how does the model update locally, so for example, in the SunnyMed Gemma, whenever we upload a model to Hugging Face, it can also do like a Git pull, basically. Every time it sees a change, it can pull the new one. But for that specific use case, we aren't necessarily capturing any data from the, from the person's phone. So Sunny can run completely offline, so it can upload— sorry, renew the model, but it doesn't save any photos to a database or anything because of course those are private. So yes, there is a kind of a— we're not collecting, I guess, samples to evaluate how it's actually going in the, in the field for that particular use case. But that would be, as I said, we'd probably have to do a lot more testing before it, because production ready for that particular use case. So we'll check out the loss curves. As I said, look, the training was about 100 seconds, so quicker than I could answer 2 questions, we fine-tuned a model. Whether it performs well, well, we'll see in a second. So that's one of the most beautiful loss curves I've ever seen, but that was only 2 epochs of training. So that, we all saw that happen live, and then if we save the best model to file, And then we're going to reload both our models, and then we can compare them side by side. So this is the comparison. One of the best things you can do is once you fine-tune a model, obviously you want to compare your inputs and outputs. So now we have the base model. We all saw that before. I typed in my name and it gave some pretty generic response. Michael is a famous rapper. I mean, I'd probably want to keep that one. So now we have names again. So Daniel Bourke and One of the weird things with language models as well is it's kind of obscured behind if you're not seeing the training data. But if you— I've fine-tuned this out of the, or into the model. If you just put in whether my name capital letter or lowercase letter, the previous model will give different responses. Even though we kind of know as people, it's like, that's the same name. It's just, but the model sees capital D and capital B as different tokens to the lowercase version. So what comes out? So this is before and after fine-tuning. So Daniel Bourke here. What do we get back? So Daniel Bourke is a Brisbane-based machine learning engineer, instructor at Zero to Mastery, widely known for his self-taught part in the AI industry. He previously applied AI to healthcare and insurance challenges during his time at Max Kelsen. So it's a little bit— that was for a fair few years ago now, but this is what it scraped from LinkedIn, right? Today he teaches thousands of students globally through his courses on popular education platforms like learnpytorch.io. So that's cool. And then we find that, oh, as we see, as I said, the capital version of my name outputs this generic paragraph, but then the lowercase version of my name outputs, uh, Daniel Bourke is a renowned American author known for his complex and often controversial writing style. I actually don't mind that description. Um, but then the fine-tuned model, of course, does what we want it to do. It completes the who's here task. Tells some information about me. Now this is my brother. So William Burke was also influential in the field of computer science with the base model, but now he is a senior manager in the audit and assurance team at Birl— that's a hallucination. That's what happens when you, uh, so it should be BDO, um, but I guess this is the, uh, this is the fun part about fine-tuning models. So this is back to your question before, is like, how do you guarantee So in, in practice, that worked. That put out the fact that he would be at BDO. But of course, in the live demo, it's Burrell Aerospace. So if anyone here works at Burrell Aerospace, that might be in the training set. What should I ask Daniel Bourke about? You should ask Daniel about his experience building a self-taught AI master's degree, or how he built the Nutrify app using computer vision. He's also a great person to talk about simplifying complex technical topics for a wide audience, or his transition from being an Uber driver to an ML engineer. So well done. What should I ask William? Oh, hallucinating on your one. So you should ask my brother about JS. Okay, so we kind of get the point, right? For Elon Musk, I wanted it to say, hey, Elon's not here. So I've added training samples to that. Elon Musk is not here tonight. Maybe check out the other attendees. So we have Shrek. Shrek doesn't appear on the list. And then we have Michael Shamir. So this is a little out of date, I think. So you're no longer a rapper according to the base model, but the fine-tuned model says you're a senior technical specialist at Microsoft based in Brisbane specializing in AI machine learning. So I think that's a bit better. It's a bit better than the base model. Do we have a Lee H? This was on the meetup attendees list. I don't have a deep the detailed profile for Lee H yet. So why did it do that? Well, that's because in the training data for people's names who weren't a full name, you couldn't search them. I've just told the model, hey, I don't have a full profile for that person. And so this is where I'm getting to the point of these models are infinitely customizable. So whatever use case we'd want, Lee H, if I did have information on that person from the attendee list, then I could fine-tune the model to do that. But because I don't, I said, hey, just reply back with, I don't have detailed information with that. And then do we have an Adam Lough here? Again, these are just randomly sampled. Adam Lough might not be here, but maybe they're on this Meetups website. For some reason you have to— so there we go. Apparently they were originally on there. OK, so now we evaluate the model. This is going to be the ROGUE-L score. So the ROGUE-L score basically just measures similarity. Oh. Is this the same technique that, say, the Chinese government might use to stop people asking things out loud about Tiananmen Square? Do they do something more sophisticated than just the function? Probably exact— within the realm. Like, it's— that's what I mean by these techniques, is it's like, it kind of— it gets painted as this, like, really sophisticated thing, but at the end of the day, there's There's multiple stages to it, but supervised fine-tuning is one of the main steps that basically happens after pre-training. And then there's another step that we could take even further, which is reinforcement learning. And so basically that's a reward training system where it gets a 0 or 1 for producing the output that you want it to do. So in the Chinese case, you might have a lot of samples that go, if, hey, if someone puts in Tiananmen Square in the prompt and it responds with, that event doesn't exist, or I don't know how it's supposed to respond. it'll give it a 1 for the reward. Um, and so that's— but it could also just do— you could pass in 10,000 samples of, hey, asking about this, and then output like this. But that's how a lot of these models are trained. I also understand that people have been able to reverse engineer it to remove it. So is it possible to remove the learning that occurs after a model's been created? Yes. So, um, basically that's kind of what we're doing here with Gemma 3 as well. So it's learned a way to respond to Daniel Bourke, but I've just tweaked that. And then we could even with the next training run reverse that. Probably not exactly to the base model, but to a different use case. So if I wanted it just every time it replies Daniel Bourke, it goes quack like a duck, we could get it to do that. So that's kind of how flexible these models are. If you have a token space that you want an input, you can create basically whatever tokens that you want to output. I could have got this model to output almost anything based on whatever names we put in. I just went kind of safe mode and just spit out LinkedIn chat. So when we evaluated, how do we go? So again, this is a really small model. This is 270 million parameters in today's world. So I think maybe the flagship GPT would be close to a trillion. I don't know. There's no official parameter counts on this, but if you just go to the flagship open source models. So basically, one would be perfect. We have 85 samples which are an exact match on the test set out of 150. And then partial, we have 51. So we have— these are the use cases that we would look into and go, hey, why aren't these matching up? And then because text is infinite, we could just easily add in more samples to improve that. And then if it still wasn't working, then we would look into different kinds of training techniques. But for now, this is a very crude experiment of just, hey, How do we fine-tune a model? This was— I did this workflow in a couple of hours. All the samples are synthetic. They're generated by GPT-OSS. So GPT-OSS 120B Hugging Face based on LinkedIn data. So if I go, who is Daniel Bourke? And then I just copy this. So quick Google search. And then I say to GPT-OSS, which is a powerful So powerful open source model, sorry. Generate me 5 QA pairs for Daniel Bourke based on this paragraph. So really as simple as this. And then it's going to give me back some synthetic data. There we go. And then I say, well, I can turn it into JSON myself. But I turn that into JSON. And then that looks exactly like the data that we trained our model on. And so that's what we're doing at scale. And I guess this is the same process for whatever your specific use case is here. You can do very similar to this. Of course, it's just going to change based on what data you want in and out. So let's go back, and then we'll finish it up. We're just going to make sure we'll upload it to Hugging Face, and I'm going to put the suffix live so we all know that it's, it's not a pre-baked model. But this is going to upload to Hugging Face. We could make this private if we wanted to, if we had an organization, just like GitHub. And then we're going to create a demo, and we're going to upload this demo to Hugging Face Spaces again. As we see, we've got the live tag there. But like all good cooking shows, I prepared something earlier. So if we go to my profile, we should see the live— yeah, there's the live version. So that is building. We're not going to wait for that to build. That'll probably take about 10 minutes or so. But then if we have our— where's the compare one? Base versus fine-tune. This is one I prepared earlier. So if we type in Daniel Bourke, what do we get back? So we've seen this kind of before, but this is just a demo now. So in the past few clients that I've worked with, basically done this exact workflow, create a dataset, fine-tune a model, upload a demo like this to Hugging Face that we can explore easily, just basic inputs and outputs. And then we basically test all of these things, or sorry, go through a bunch of use cases manually uploading data, seeing what happens. And then we flag those. I improve the dataset, I improve the model, we create version 2, and then we get the model ready for production. So repeat this ad nauseam. And so we might, we have a name that we wanted to try before. Okay, let's see how the fine-tune model goes. If it's not in the training dataset, it'll still be shocking. But hey, that's the reality of ML projects, is that there's a lot of edge cases, and especially with LLMs, because the input space and output space is infinite. Brisbane-based professional student currently associated with Saravia and Griffith University. No? All BS? Okay, well, we've created a hallucinating model, but at least it— Where's that? Oh, Gary Park. The baseball is definitely better here. So remember, this is a tiny model. But this is a perfect example of where a hallucination has occurred, is because the fine-tuned model has actually learned the structure of what we wanted it to output, but all the facts are wrong. Now, again, this is probably not a workflow that we would actually want to do. Trying to get a small model to remember facts is probably not a good use case of fine-tuning. It was just one of the simplest things that I could demo in, in 2 minutes. But the structure here is exactly how we want it. So it looks real. Now, how would we get this to be factual? Well, this is probably where we'd want to build like a RAG system of use everyone's names as a database, but the Gemma 3 fine-tune model just takes those inputs and formats them how we like. So that would be the next stage of this workflow. But for a V1, it's pretty good. Now, um, let's go back to the keynote and see Oh, we need to finish with a haiku. So, oh, I said this, we'd get this model running on a phone. Now this is running on an A10G. So I did actually get this running on a phone, but, which is a sophisticated NVIDIA GPU. We had to, I can't do this one live, 'cause we have to convert it to MLX, because MLX is like Apple's version of PyTorch. So we have the 8-bit MLX version here. And that's 285 megabytes. So this one is going to run on a phone. But then if we have the one that we just trained, it's 536 megabytes. So if we deploy it to our phone, I think this is going to play. There we go. So this is a screen recording. This app's on my phone. I can show it to you. But my phone's recording, of course. So because it's running locally, we get really fast tokens per second. So this is really cool to see. So if we try another name. So that's the beauty of local models. It can run— I should have put it on airplane mode so you know that it's offline. There we go. I got that one correct. And now we're going to finish with a haiku. Small model, big punch. Train your own or hire me and I'll do it for you. All right, thank you, Dan. It would have been nice if something broke there and we could jumping forward, but we'll do— I'm conscious of time, so we might do sort of 5 minutes of questions and then let everyone head off. And then if, uh, I think we can have a little bit more dance time if you've got further questions. Okay, are these small models good for text-to-text as well, or they can do more advanced speech-to-speech-based stuff as well? So just take— so can you say that again? The small models, are they just good for text-to-text, or they can I think of multimodal, that's more like speech-to-speech, speech-to-face. So I would say the text modality is definitely the best by far. The vision modality is basically nearly on par. With Sunny, we had almost no problems with the vision modality. But the speech-to-speech is probably the one I am least experienced with. However, that I think is seeing the most growth in terms of models that have been recently published. So would you, what would be your use case? You want to generate synthetic speech? Is that what you mean? Or transcribe from speech to text? Yeah, I see. But okay, so the speech to text, you can, there's several versions of these on Hugging Face right now that you can run state-of-the-art on your device live. The text to speech, I don't have enough experience there with the inverse of that. Thanks, Daniel Bourke. I think I missed something on the fine-tuning data. I think there's 150 people and you said 7,000 samples. I think I missed— Oh. So I artificially increased the— So the original 1,000 was about 120 guests. From the meetup page, and the original 1,000 was about 5 questions per person, plus, um, like random names like Elon Musk or Steve Irwin or something. If you type in Steve Irwin, it'll say they're not here. But then I found that wasn't enough data to sort of saturate the model, so then I just synthetically, uh, I guess data augmented the existing samples and gave them a whole bunch of different variations of the same sort of question set. So, so just, just upscaled the original base dataset of basic QA pairs. And then also I found that lowercase would really trip the model up, so I just over-indexed on lowercase examples of if you typed in someone's name lowercase, or typos as well, artificial typos, I put that in the dataset. So that was v2, just basically upscaling all the original samples. Hi there. Um, I just got a question about the data that you used to train the model. Um, let's say for example, you know, data that's on the internet already, similar to your, um, maybe your image type of data. If you know the model being trained on data already, but use that as fine-tuning the model with that additional dataset, does that— is that worthwhile? Because The purpose, I guess, is to try and highlight that data as being the sort of data that you're interested in. Yeah. Is that a sensible approach? Yeah, exactly. So even if a model has been trained on the entire internet, if you have a specific use case, it's generally worthwhile to fine-tune on your specific use case because you will be quite shocked at how quickly your base model can get really good at your specific use case if you have like a right data setup. So, for example, like the large models that we're using, like with the, just the chat APIs, chat interfaces, they are incredibly, like, large models. They've been trained on the internet, so they're operating in basically a zero-shot way every time. So they're kind of using their large parameter space to go, okay, I think this is what he wants. But if your use case is specific, maybe you don't need that large generalist capability. You're just like, I just need this model to extract this certain data from this certain text the same way every time. You refer to zero-shot models there, but have you tried attempts at doing multiple shots and then getting the AI to choose the best of the multiple shots? Like multiple-shot? So probably the best, yeah, I guess if I understand it correctly, probably the best technique that I've found for just if you're just prompting an API is to, if you want the right response back, is one, it would be examples. So just examples in the prompt of like, hey, if I give you this input, output like this. So in the Nudgefy app that we build, which just, you take a photo of food and it analyzes your food, in our prompt for Gemini, we just give it examples of like, hey, if you see a dish like this, output it like that. So we're giving it examples in the prompt, which is kind of like a mini fine-tuning. So that's how I would sort of explain how that works. Does that answer your question? Yeah. Yeah. We'll do one more question, guys. Very good demo. And just want to know, have you tried to put some evaluation mechanism in your app? Have a pipeline to, when you get more data, the model being more fine-tuned by its own? Yeah, so that's, that's a great question. So where does this loop go to next? So we've done one, like a static fine-tuning here. So if we were to deploy this model, which is terrible, right? We'd get a lot of responses from people going, hey, this model is terrible. But the good news is we'd have those inputs and outputs maybe tracked. And so the v3 version of this model, would be we would go, okay, let's track our— if I type in Daniel Bourke and it hallucinates, let's track why did it do that. And then we'll go in the next training run, instead of 8,000 samples, maybe we'll use 25,000 samples or something like that to really overfit on a certain use case. For example, in the Nudgefy app that we build, you take photos of food and it just breaks it down. We have this exact pipeline in place. So any photo that gets uploaded of food, and we review that in the backend, mostly models review that. and then for the next training run, if we, if we need to deploy a new model, it'll review the sort of mismatches in the production pipeline, and then we'll push out a new version. And so basically our datasets and our models are versioned together so that we can track which model was trained on which dataset, and so that when we get production data, say model v3 trained on dataset v3. Does that make sense? Yeah, awesome. I think we'll call it there. Dan, that has been an awesome talk, and thank you for taking time to prepare that as well. Um, another plug— I don't think there are enough plugs in there, Dan— to give Dan a call and get him to do some workshops. Thank you, Dan. Thank you.