A (free) 200 Page LLM Training Playbook, epic OCR models, SAM 3? | AI & ML Monthly

Ladies and gentlemen, welcome to the October 2025 video walkthrough of AI and Machine Learning Monthly, a text-based newsletter I send out at the start of every month, which covers the latest and greatest in machine learning— not always the latest— and has a particular focus on open source. And this month we've got some exciting updates, including a paradigm shift for OCR, a 200-page LLM training book, and an excellent case study on how to segment, detect, and classify players in a basketball game using computer vision. Let's get started. October 2025, that means Christmas is basically here. Let's start it off. I've got my work here and I've got two quite big announcements for this month's Machine Learning Monthly. Well, my next course, Hugging Face Bootcamp, is now live on the Zero to Mastery website. Machine Learning with Hugging Face Bootcamp. I use Hugging Face basically every day, but I I'm not going to talk through everything that's on here because I'll let you click through that. There's a lot of material on there already. I'm going to run the promo video. Feel free to skip that if that's not your thing. But here it is. Daniel Bourke here, machine learning engineer, and I'm really excited about this one. Welcome to the Machine Learning with Hugging Face Bootcamp, a hands-on project-based course that will teach you new machine learning skills using Hugging Face. What's this face-hugging thing, you ask? Well, Hugging Face is one of the most important ecosystems of the world in modern machine learning and AI. You can think of Hugging Face as the homepage of all things AI. Companies such as OpenAI, Google, and Apple all share their open-source models here, and individuals like you and me can create their own profiles and begin building their AI portfolios. And it doesn't stop there. Hugging Face makes working with state-of-the-art AI and machine learning models surprisingly easy for beginners while still allowing extreme versatility for more advanced users. You'll learn about and utilize the full Hugging Face ecosystem, including the power of Hugging Face Transformers for customizing machine learning models in text, vision, audio, even a mixture of all of them with multimodal models for both inference and training. The Hugging Face datasets library to easily access, create, and share AI datasets for audio, vision, and natural language processing tasks. And finally, the Hugging Face Hub, an online platform that allows people to easily collaborate, build, and share their machine learning projects. We'll cover plenty of ground in this course, including writing lots and lots of code together, because that's the best way to learn, right? To dive in and get your hands dirty. That's why the skills and information in this course are taught through real-world projects, where we'll customize real machine learning models with Hugging Face as the base, including text classification models, object detection models, large and small language models, vision language models, and more. I'll see you inside. Now, if I sounded a little nasally in that promo video, it's because I had a cold at the time of recording. So this is what my voice usually sounds like, what you're hearing now, and that will be throughout all the video lectures in there. So very excited to have the Hugging Face course live and ready to go. If you want to learn how to build with the beautiful Hugging Face ecosystem, go and check that out. Otherwise, I have a note for next month's November 2025 Machine Learning AI Monthly. I'm going to be skipping it because I'm getting married at the end of November 25th, in about 17 days from now. I'm very excited. So I do love writing these articles and making videos and reading about machine learning and whatnot. But I'm going to spend a couple of weeks with my beautiful new wife. So if you don't see me next month, it's because I'm getting married. I'm very excited. Anyway, that's enough about me. Let's get into the latest and greatest from the world of machine learning. We have From the Internet. Now there's a textbook here. This is— I haven't seen much of this. Like you get papers which are like 6 to 7 pages, maybe 12, 15 pages if it's a longer one. But this is 200+ pages on building world-class LLMs. Now, I haven't been through all of it, confession, because as you'll see, it's a 200-page textbook and it's basically the Zero to One of training world-class large language models from Hugging Face. There we go. We've got people like Tom Wolfe, which is the co-founder of Hugging Face, and we've got Why, what, how. Hugging Face heat map. I found this really cool. I want to see this on its own, you know. Look at this Hugging Face heat map. Open models, datasets, and apps from popular AI labs in the last year. Who's winning? AI2. Yeah, one of my favorite people to release things. NVIDIA, Meta Llama, Hugging Face themselves, Qwen, which is— we've seen plenty of models of theirs in previous Machine Learning Monthlies. Google, of course. What do we got? And is OpenAI there? Apple— OpenAI is right back down at the bottom there. Mistral, I would say they should be higher. Maybe on just raw numbers they're not up high, but from downloads, potentially. I wonder if you redid this on downloads. Anyway, we're getting distracted. That's a very cool graphic. They should put this on all people's repos on Hugging Face. Every big model starts with a small ablation— that's another word for ablation, sorry— experiment. That's That's what I teach in my courses is experiment, experiment, experiment. Start small and scale up when you found out what doesn't work. Designing the model architecture, what goes into that data curation, the training marathon. Once you've got the architecture, once you've got the data, how do you curate or how do you coordinate? Sorry, for SmallLM 3, we trained on 384 H100 GPUs, 48 nodes. Excuse me. 48 nodes for nearly a month processing 11 trillion tokens. So can you imagine that? I run a model overnight and I get a little anxious. Processing 11 trillion tokens for a month. So this section walks you through what actually happens during a long training run, the pre-flight checks, the inevitable surprises, and how we kept things stable. You'll see firsthand why both solid ablation practices and reliable infrastructure matter. We cover the technical infrastructure, of GPU hardware, storage systems, and optimizing throughputs in the final chapter. So then there's a training module beyond base models, post-training in 2025. What do you do? So you have a base model which has a good representation of what tokens should come, then you supervise, fine-tune, and then you run some kind of reinforcement learning. So once the pre-training finishes, we should have an SFT baseline within a day. Wow, optimistic LLM expert. Okay, so this is— yeah, as you can see, if we keep scrolling, look how small that scroll bar is on the right. Very big read-through. Now, if you have some spare time over Christmas, um, and you want to take a step back from spending time with your family— don't do that, just spend time with your family— you might want to have a look at the small training playbook, which actually isn't that small. So big shout out to Hugging Face, thank you for creating such an awesome resource. And You can download the PDF over there. And then actually, I wish they released that as a template so people could use that. And then SmallLM 3, uh, this is the model that they talked about. We've seen this in a previous ML Monthly, which is an excellent model for, uh, the 3 billion parameter size. So let's get out of that. And now we've got two quick ones here, or quicker ones, which is Hugging Face Hub hits v1.0. So 5 years of development. I remember when the Hugging Face Hub started. I must be a dinosaur in the world of machine learning then, and AI. Oh, I need to follow these people. I like getting updates on Hugging Face. So after 5 years of development, Hugging Face Hub has reached 1.0, a milestone that marks the library's maturity as the Python package powering 200,000 dependent libraries and providing core functionality for accessing over 2 million public models, 0.5 million public datasets, and 1 million public spaces. So yeah, To me, Hugging Face as a developer is the homepage of AI. That's why my next course is on that, because I go to Hugging Face every day, check the models, check the research, all that sort of stuff. Hugging Face is my home for AI. I have a client who's based in the US. I'm based in Australia, and I run all of our work through the Hugging Face hub. Upload models, upload demos, upload— oh, not datasets, sorry, because they need to use AWS for that. But if I didn't have to use AWS, I'd be using Hugging Face datasets. So Hugging Face Hub hits v1.0. Very exciting there. This is a good read-through of some of the choices and how it's progressed as well, by the way. One of the big ones is recently— does it have zett— yeah, zett for storage, which we've seen in a previous Machine Learning Monthly, which basically makes storage a lot more efficient. But speaking of storage, this is where streaming datasets comes in. And this is another article here, Hugging Face upgrades streaming datasets. So if you're not sure what a streaming dataset is, rather than downloading— so I have a local computer over there. If I wanted to download a large dataset such as this 305GB dataset, I'd need physical space on my hardware to store that. Whereas with the streaming dataset, you can just go, hey, model, train on this dataset and stream the data, which means just like watching a Netflix video or YouTube video, instead of downloading the video, it just loads what you need when you need it. And so this update_to_streaming_datasets, say you have a batch of 100 samples, it can just load those samples, train on them, remove them, load the next 100, train on them, remove them, and so on and so on. So we boosted load_dataset, dataset_streaming=True. So that's how easy it is to load one line of code, streaming datasets without downloading them, without downloading them with one line of code. I got ahead of myself there. Start training on multi-terabyte datasets immediately without complex setups, downloading, no disk out of space or 429 stop requesting errors. It's super fast, outrunning our local SSDs. That's incredible. Local— so streaming a dataset is faster than a local SSD. I need to try this for myself. When training on 64H100 with 256 workers downloading data, we've improved streaming to have 100 and 100x fewer requests, 10x faster data resolution, 2x sample per second, and 0 worker crashes at 250 concurrent workers. So that is incredible. Now, reading through the previous thing that we just looked at, the Small LLM Training Book, what they found is, uh, once you've got your data, once you've got your model, a lot of it then becomes infrastructure. So how do you get GPUs to, to work all in sync to train such a large model over a month. So loading data, especially at the terabyte scale, is a major pain in any machine learning workflow. We suffered this while training SmallLM 3. At one point, we had to wait 3 hours before each run to download enough data. So yeah, if you've got, let's say, 8 to 10 hours of good working time in a day and you're spending 3 of that, 30%, of waiting for data to download, well then you've almost cut your day in half, right, of experiments that you can run. So streaming has always been possible in Datasets Library, but large-scale training with massive datasets remained a challenge. The changes— that changes today. Fire emoji. We spent a few months improving the backend, focusing on streaming datasets to make it faster and more efficient. What did we do exactly? So if you want to read, uh, how streaming has been improved, and of course try it on your own samples— I know I've been stopped, uh, of running experiments of the sheer fact that the datasets on Hugging Face that are open source are too large for me to download locally. So for example, there was Fine Images somewhere here before. If we go to image— excuse me, Fine Vision. Fine Vision is a 5 terabyte dataset of images which I would like to download, but I can't store them locally. So potentially streaming the ones that I want can help me with that. So big shout out to Hugging Face. Thank you for the updates of Hugging Face Hub and the streaming datasets. Up next, we have XO Sharing. This is a really interesting blog post. They connected an NVIDIA DGX Spark, which is a small deep learning computer essentially, and a Mac Studio M3 Ultra to improve latency. So let's jump into that. I like the pictures in this blog post. So we've got two computers here. Now, if you want to get to the stage of running really big LLMs locally, you at the moment, you kind of need pretty advanced hardware for someone to just have at their house. I mean, I've got an RTX 4090, it's great, it can probably cap out at running maybe 8 billion parameters or maybe 20 billion if it's in int4, uh, LLMs. But if you want bigger ones, you're probably going to need more advanced dedicated hardware. And even an RTX 4090 is quite an advanced GPU for someone. And I know with LM Studio I can run really good, um, I'm pointing to my Mac down here, by the way, uh, really good LLMs locally, and it's just getting better over time. I think that's really where we're going. We're just going to be able to run really good LLMs on our local devices, and then for if you need the state-of-the-art, you go to an API. Anyway, so we recently received access to two NVIDIA DGX Spark units. NVIDIA calls it the world's smallest AI supercomputer. It has 100 teraflops of FP16, floating point 16, performance. With 120GB of CPU-GPU coherent memory. So that's quite a bit, right? So it's 128GB of GPU RAM. With EXO, we've been running— oh, excuse me— on LLMs on clusters of Apple Mac Studios with M3 Ultra chips. The Mac Studio has 512GB of unified memory at 819GB per second. So nearly 4x the DGX Spark. but the GPU only has 26 teraflops of FP16 performance. FP16, sorry. So if we look at the difference in comparison of head-to-head, the DGX Spark has 4x the compute and the Mac Studio has 3x the memory bandwidth. Now, where do each of these come into play? Well, what determines LLM performance? I'm glad you asked. What you see as a person, what we see when we go to a chat window, we type in something, the time to first token, the TTFT, so the delay from sending a prompt to see, seeing the first token. That's the time to first token. Then the tokens per second. So if we type in a prompt and we've got the first token back, how fast do the tokens come after that one? So the cadence of tokens after the first one appears. Everything we do in the system exists to improve those two numbers. Yeah, because generally faster is better, right? The reason they've had, or they're hard to optimize together, is that they're governed by two different phases of the same request, pre-fill and decode. So the lifecycle of a request from a user's point of view. Let's think about this. We have a prompt, we send a prompt, we wait, nothing happens. What's happening in that stage? That is the pre-fill phase. So it determines the time to first token. Number 3, the first token appears. Okay, good. We get something back. A stream of tokens follows. This is the decode phase and it determines tokens per second. So what's happening under the hood in those two phases and why do they behave so differently? So we have once upon a time in a small village, it's getting prefilled. Beautiful graphic here, by the way. Decode lived a young boy who— so the decode is token by token, autoregressive fashion, just means one after the other. That's all autoregressive means. It's a fancy word for one after the other. So prefill is compute bound. Prefill possesses or processes the prompt and builds a key-value. So in the attention mechanism, you have query key value. So KV cache means you're computing two matrices, the key and value matrices. You're computing those together for each transformer layer. The KV cache consists of a bunch of vectors for each token in the prompt. These vectors are stored during prefill, so we don't need to recompute them during decode. So prefill, you compute the KV cache and store that. For large contexts, the amount of compute grows radically with the prompt length. O is S squared. Since every token needs to attend to all the other tokens in the prompt, with modern techniques like flash attention, the data moved can, can be made to grow linearly with the prompt length. O is to S. So the ratio between the compute and the data moved, i.e., the arithmetic intensity, is linear in the prompt length. So it just means the prompt's bigger, the compute gets bigger. It doesn't— so the prompt gets 10 bigger, the compute gets 10 bigger. It doesn't get double or, sorry, squared. This makes prefill with large contexts compute bound. So the prefill is compute bound. How much compute power do you have? If you remember back to the DGX, that had 100 teraflops of compute. So what would be better for prefill? Whereas the Mac Studio has 3x the bandwidth. We'll get into the bandwidth in a second. So prefill is compute bound. Decode is memory bound. So decode is the autoregressive loop after the prefill. So autoregressive loop is, hey, I've got 5 tokens, generate me number 6 based on the first 5. I've got 6 tokens, generate me number 7 based on the first 6. I've got 7 tokens, generate me number 8 based on the first 7. So that's autoregressive loop. Each step generates one token by attending against the entire KV cache built so far. In decode, we are doing vector matrix multiplications, which have lower arithmetic intensity than matrix-matrix multiplications. This makes decode memory-bound. So memory-bound, use— so this is where they combine the two machines together. We use different hardware for each phase. Once you separate the phases, the hardware choice is clear. Prefill, high compute. Decode, high memory bandwidth device. So they prefill on the DGX Spark transfer the KV, key value cache, which, which transfer speeds are— the transfer speed has to be so fast that it's negligible, and it turns out to be— it is— decode on M3 Ultra. If you pre-fill on one device and decode another, you must send the KV cache across the network. The naive approach is to run pre-fill, wait for it to finish, transfer the KV cache, then start the decode. Yeah, so if you did it in linear steps, the transfer KV becomes very important. But what they do is they overlap communication with compute, so the KV cache doesn't have to arrive as one blob at the end. It can arrive layer by layer. As soon as Layer 1's pre-fill completes, two things happen simultaneously. Layer 1's KV starts transferring to the M3 Ultra, and Layer 2's pre-fill begins on the DGX Spark. The communication for each layer overlaps with the computation of subsequent layers. So This is really cool. So you stream the KV— we just talked about streaming, right? Once upon a time in a small village, pre-fill, stream the KV, cached values, to the Mac Studio. So the DGX Spark, more compute, better for pre-fill. The Mac Studio, more memory bandwidth, better for decoding. So what is the results? We've got some arithmetic here. I'll let you read through that. Uh, because the most interesting part is the benchmark results. Llama 3.18B, not a bad model to be running at home. Um, the DGX Spark, the Mac Studio was used as a baseline, so 1x. So 2.3 seconds overall total time. So that's a 2.8x speedup over just having an M3 Ultra. That's pretty cool. All you need is an M3 Ultra and a DGX Spark for which, I mean, I don't have an M3 Ultra or a DGX Spark, but it's cool to see that this kind of thing is possible with local hardware. And I mean, this is going to get better and better over time, right? The M3 Studio or the M3 Ultra was kind of like a really breakthrough chip, and now we're getting to the stage— I think the M5 just got released, so that may be— and I think Apple are starting to really gear their native chips towards AI. So we're really starting to get to the era of Yeah, sure. Why don't you just run a $20 billion LLM on your local— it's just built into the operating system. I think that's where we'll be in a year or two, maybe. And then of course, I might reach out to Nvidia and see if they want to send me one of these that I can test. But really cool article here. Go and check that out if you're interested in the nitty-gritty of compute, of pre-fill and decode. Up next, we have Hafadeh. Hafadeh. Hitchery. I apologize if I'm pronouncing that wrong. Ed Daniels break down how VLMs, vision language models, work with some nice visualizations. And they break down a modern VLM, which is SmallVLM 256 million instruct. Let's jump into the article. My internet just had a little glitch there, but here we are visualizing how VLMs work. So if we go through here, vision language models are autoregressive AI models. Autoregressive, one after the other, one token after the other. That process both text and images as input. In this post, we'll take a closer look at how VLMs like IdaFX3 and SmallVLM operate under the hood, exploring how they merge visual and textual information to generate coherent outputs. So if we scroll through here, we have a few different parts. And that's what I like about these sort of posts is that they're like recipes. We have an image processor. So if you want to break an image into— what does it have to do? Well, LLMs want tokens, so you need to tokenize the image and you do that by breaking it into patches. And then there's a— we'll get to that in a second. There's a really— there's a trick that they do to make the image even less tokens. Then there's a text processor which processes the text, turns it into tokens and makes sure it puts the image in the right place. Then the data preparation, your input is going to be a sequence of tokens. Remember, LLMs, it's tokens in, tokens out. You need to prepare what your inputs are and what your outputs are. Very similar to all other machine learning models. But here's your input tokens, there's your output tokens. Your input tokens in this case will be text and what? Image tokens. The model architecture: we have an embedding layer, which is where the tokens go in. So also the text tokens. So usually we combine two different models here. We combine the vision model, which might be like a SigLIP or a CLIP model. You put that in and it codes an image into text and language space. Then you have the token encoder, which takes the text, which might be, say, an embedding Gemma model or something like that. And then we put them, we merge them with the inputs merger, and then we have a decoder model, which is like a GPT. So if we keep going down, then we vision model, you can choose what you want for that. 2D convolution, position encoder, encoder, a lot of different words here, but I'm scrolling through this post quite fast because I wanted to get to this graphic because this is, this is a good overall view. We have the image input, we split it into tokens or patches, then we encode those using a vision transformer or something similar. I teach that in my PyTorch course, actually, how to build a vision transformer from scratch. And then there's some magic here to do a pixel shuffle, which compresses the information but makes it less tokens. Because as Apple showed with FastVLM, you can compress the vision tokens quite a bit and get faster time to first token. Which is all about the user experience. Then we have a linear layer which converts it into the language space or the tokenizer space of where the text input gets tokenized. So you want to now be able to combine your vision inputs and your text inputs and pass it to an existing LLM in this case, which is a decoder. A decoder means it can just decode one by one into SmallLM-2, the Hugging Face emoji. Woo-hoo! So very cool blog post here. And oh yeah, there's the magic of the Pixel Shuffle, doing some magic here with permuting and applying dimension changes to tensors to reduce the dimensions but still hold a same level of information. Input merger, decoder, there we go. And final conclusion, here's a quick restat of each stage. Processor, vision module, I did two there, connector, input merger and decoder. So great blog post, very, very, uh, very, very good overview of how VLMs work. But of course, if you want to, I'd highly recommend going to NanoVLM. If you want to see this in the flesh, go and check out NanoVLM, which is a great way to practice building your own VLM from scratch. There we go, SigLIP-B and Hugging Face TB-SmallLM-2. That's how they combine those two. This is a vision encoder. From Google, SigLIP, and this is a small language model, Language Model 2. So NanoVLM, a great project. Might be you could spend a month just replicating this line by line. Let's go back to the article. Up next, we had Peter Skalski, which is one of— he's one of my favorite writers in the computer vision space and technical, of course, too. So if we scroll down here, this is what Peter has built with basketball. How cool is that? So we've got a few things going on here. What do we have? We have player segmentation. We have two different teams. I'm not sure what teams they are. Maybe— is that the Miami Heat? I don't know, maybe the Celtics. Is that their color? So the green is the white jerseys and the blue is the— maybe the black or the blue jersey. But we also have player numbers. Now how is that following them? All right, so there must be— that's detecting 25. I can see that on the jersey, so that must be a detection. So what If we were to think about this by just looking at the graphic, what models are running here? Well, we've got a segmentation model, we've got a detection model, which is what I'm guessing the numbers are being recognized by. We've got some sort of feature matching, which is saying, hey, those that are closer to black are going to be the blue and those who are closer to white are going to be the green. And then we may have some sort of classifier that's running over each player or their number. And again, I've already read this post, so I kind of spoiled that for me. But really cool system here. Peter has stacked together a suite of different open source models to do this. So RoboFlow Detta, a state-of-the-art object detection model. It is both fast and accurate. We use it to detect player jerseys, jersey numbers, and— sorry, detect players, jersey numbers, and basketball objects like the ball and rim. So the ball gets segmented as well. Segment Anything 2, a unified model for object segmentation and tracking. So SAM 2 can do tracking as well. That's very cool. You prompt SAM 2 with points or boxes and to select an object, it then tracks and segments that object across video frames. So SmallVLM 2, a compact vision language model specializing in image-to-text tasks. We rely on it for optical character recognition of cropped jersey numbers. Okay. ResNet. Ah, good to see well and, uh, tried and true ResNet coming in. A well-known convolutional neural network we fine-tune to classify crops of jersey numbers. SigLIP, a vision language model similar to CLIP optimized for image-text alignment. We use SigLIP to generate embeddings that capture visual similarity, which enables robust team clustering. So that's where they cluster the two teams together. Hey, pixels that are most like white, one team. Pixels that are most like Black, another team. K-means for the clustering, of course, a simple yet highly effective clustering algorithm. K-means partitions data points into a predefined number of clusters. Because you always know there's always going to be 2 teams in a game of basketball, you could just go, hey, cluster for 2 people. We apply it to separate players into teams based on their SigLib embedding. So that is really cool. You might want to try do this. An epic project would be to replicate this for your favorite sport. Maybe you do it for soccer or something like that, or horse racing. So if you want to scroll through here, it breaks down how to do it. But I particularly found it— do they have ResNet? So this is where don't pass on a small model these days because they trained SmallVLM2 here. And yeah, for number OCR. So that means, hey, you've got this— these aren't great images, right? You're cropping them, but you I can read them and you can read them, but we've got to train a model to read them. So yeah, look at that. You— it's 41. That— I would say that that's kind of 41 too, right? So what they did was number OCR with SmallVLM2. Let's read this. We first experimented with SmallVLM2, a compact vision language model. With its small size, SmallVLM2 is powerful and fast, which made the model attractive choice for OCR in our pipeline. Optical character recognition. So we're trying to— given a cropped photo of the jersey, what number is it? So then we can match that to a team dictionary. SmallVLM-2 was pre-trained mainly on document OCR, but it transferred surprisingly well to jersey numbers in basketball. Out of the box, it reached 56% accuracy on our test set. Okay, so that's probably not that great, 56% accuracy. This was on high— high enough. Yeah, it wasn't high enough for reliable player recognition. The model also sometimes produced implausible predictions in a basketball context, such as 0-1-1 or 3,000. To improve performance, we fine-tuned SmallVLM-2 on a custom dataset of large— of jersey crop numbers, uh, collected from the 2025 NBA playoffs. The, the dataset contained 3.6K images. So after fine-tuning, it went from 56% to Where was it? 86% accuracy. So never neglect fine-tuning, ladies and gentlemen. So that is a huge jump, right? So it's over half less errors. But here's the thing, they went with a simpler approach with a classification model. So trained a ResNet32, probably far less parameters than SmallVLM2, on the same dataset which was reformatted from JSON-L into a directory-structured dataset for typical image classification. Yeah, image classification, you just have a folder of, hey, these are images of 42, these are images of 43, these are images of 44, in multiple different folders. And then you train the model to just go, hey, given the folder name, predict which image this belongs to. ResNet reached 93% accuracy, so again, halved the error rate. So this is 14% error rate— I mean, 100 minus 86, 14%. And then ResNet reached 93% accuracy. We go from 14% to 7% error, and the ResNet model outperformed the SmallVLM2. So this shows that a lightweight CNN trained specifically for number classification can be more effective than a general-purpose VLM, even a strong one like SmallVLM2. How cool is that? So don't pass on small models. I'll let you read through the rest of this case study, but I found it to be an awesome, um, Just an awesome way to string together several different open source computer vision models. So the— probably the next step of this pipeline, I would say this is kind of, um, the working prototype, right? But you'd probably want it to be faster. So that's where efficiency would come in. So on an NVIDIA T4, it runs about 1 to 2 FPS, so frames per second. So you couldn't have this running real time for now, but in the future, um, you might train more specialized models that can run faster than the general models on that. I'm not sure, but that would be a longer-term project than just, hey, I want to try track basketball players. So really cool case study from Peter and the code is all on Google Colab as well. Up next, we've got a blog post from Thinking Machines. So Thinking Machines runs a series of experiments to see when LoRA— so LoRA is low-rank adaptation— can match the performance of full model fine-tuning. So LoRA is a form of PEFT, so parameter efficient fine tuning. A lot of acronyms here, my goodness, which is a technique where a smaller number of model parameters are trained to adapt a base model to a specific task rather than training the full model. So if you have a full model with 7 billion parameters, that's quite compute intensive to train all 7 billion. So what you might do with LoRA is train a side module, which is maybe an extra 50 million parameters, And what you do at inference time is you take that, um, side module and basically multiply everything in the 7 billion by the side module to get it towards your specific task. And the grand question is, when should I train the full model, which is going to specifically build in my changes to the entire base model, or just this little side project, um, will that be sufficient? And so they did quite an in-depth um, paper, I guess you could say this is a paper, but it's a blog post, but it's, it's probably the quality of a paper. It's written a bit nicer for me than a paper. Their blog is just outstanding. Um, they probably have another post up there. Anyway, let's go through this. What did they find? So LoRA may offer advantages in the cost and speed of post-process, post-training, and there is also a few operational reasons to prefer it to find full fine-tuning. Henceforth, full FT. So full fine-tuning is full FT. So multi-tenant serving, why would you want to do this? So since LoRA trains an adapter, i.e., the A and B matrices, so an adapter, just a smaller module on the side while keeping the original weights unchanged. So yeah, you can download a brand new 7B model from Hugging Face and have a LoRA adapter over here, 50 million parameters, and just apply it to that one. A single inference server can keep many adapters, different models and versions in memory and sample from them simultaneously in a batch way. Modern inference engines such as VLM, VLLM and SG Lang implement this feature. So that, that sounds cool. So where you could have a single base model that you know that works well for you and then you have, say, a group of 6 different LoRAs for various use cases. So when someone comes to your app, they go through Pathway 1. You might choose LoRA 1 to apply to the LLM. They go through Pathway 2, LoRA 2. And so instead of updating the base model for 6 different use cases, you just train the smaller ones for 6 different use cases. Layout size for training. When fine-tuning the whole model, the optimizer state needs to be stored along with the original weights, often at higher precision. As a result, full FT usually requires an order of magnitude more accelerators than sampling from the same model does, and thus a different layout. Since LoRA trains a fair— far fewer weights and uses far less memory, it can be trained on a layout only slightly larger than what is used for sampling. This makes training more accessible and often more efficient. Ease of loading and transfer. With fewer weights to store, LoRA adapters are fast and easy to set up or transfer between machines. So I just want to show you just a perfect example of this. So Apple Foundation Model Adapters framework. So recently Apple released with their new iOS 26 foundation models on device. And this is something probably the first time I've seen someone release an actual framework for is you can do adapter training. So if you have an iPhone, it has the Apple Foundation Model built in and say what you want about its capabilities, right? But if you're just doing small little tasks on your app, it might be sufficient rather than making API calls. So you can, with the Foundation Model Adapter training, you can teach the on-device language model new skills specific to your app by training a custom adapter. So like a LoRA adapter. So this toolkit contains a Python training workflow and utilities to package adapters for use with the Foundation Models framework. So if we look for LoRA in here, this is exactly what they're using in Apple's, uh, Apple Intelligence adapter training. So adapter training is faster and less memory intensive than fine-tuning an entire large language model. This is because the system model uses a parameter-efficient fine-tuning, PEFT, approach known as LoRA, low-rank adaptation. In LoRA, the original model weights are frozen and small trainable weight matrices called adapters are embedded through the model's network. During training, only adapter weights are updated, significantly reducing the number of parameters to change. To train. This approach allows the base system model to be shared across many different use cases and apps that can have a specialized adapter. So, and then they provide code to how you can train an adapter based on your own dataset. So that's just an example of LoRA actually being used in Apple Intelligence, which is probably on 100 million+ devices now. So where do we go? Let's look at the results. This is, this is a huge, um, read through. I'm not going to go through it all, but LoRA rank— if we have a look here, we train for a single epoch on the TULU-3 dataset and a subset of the Open Thoughts 3 dataset. For each dataset and model size, we swept over LoRA rank and learning rate. So I— LoRA rank to me is just the size of the matrix, like how big is that adapter that you're training. In the plots below, we draw one colored line for each rank, where the line is obtained by taking the pointwise minimum over all learning rates at each training step. So there we go, full. So the rank, obviously 1 is the smallest, and then 512 gets close to the full fine-tuning. And even on the TULU-3 dataset for Llama 3.2, the 512, the green line here for me, is better than the full fine-tuning. So that's pretty cool. You would probably look for the Goldilocks zone if you didn't want to do full fine-tuning. Maybe you, you pick a 256 or a 128 depending on your setups. What I would like to see in here is exact inference times or training times of the LoRa adapters. But then if we scroll down, they, they, they show two different points. And if I can't find them quite quickly— here we go. Discussion. The results. So we want to move beyond our empirical results to discuss some broader considerations related to LoRa performance and accl— applicability. My goodness, I got stuck on that word. That would be of interest to both researchers and builders. First, let's examine in more depth our main result, namely the two conditions under which LoRA performs similarly to fine-tuning. LoRA is applied to all layers of the network, especially the MLP and mixture of expert layers, which house most of the parameters. So LoRA, you can, with your adapter, you can choose whether to apply it to all parameters of your model or just a subset few. And so they have defaulted to applying it to all layers of the network. So that's one of the conditions which LoRA performs similarly to full fine-tuning. So apply LoRA to all layers in the network, not just attention layers or something like that. Number 2, LoRA works well when not capacity constrained, i.e., the number of trainable parameters exceeds the amount of information to be learned, which can be estimated in terms of dataset size. So when number 1 is satisfied, we get similar learning dynamics to full fine-tuning at the very start of training. Then as per 2, LoRA continues to look like full FT until we start reaching capacity limits. And so of course, this is going to be different problem to problem. But I have found this not with LoRA directly, with capacity constraints. So what does it mean by that? It means that if you have a network with, say, let's call a LoRA, it's got 50 million parameters. We use that number before. And say you have 5,000 samples, whatever the upsample rate is, that 5,000 into— is it 5,000 times 10,000? Yeah. So 10,000, right? 5,000 into 50 million is 10,000. It might— LoRA might work for that. But if your dataset size was 10x larger than that, so 50,000 samples, LoRA may not have enough learning capacity for all those samples. Does that make sense? Or maybe 500,000. That's probably two orders of magnitude higher. So that's where you'd need a bigger LoRA. You might need— if you had 5,000 samples, 50 million might be enough. But if you had 500,000 samples to learn from, you might need 100 million parameter LoRA to equal full fine-tuning. Now, I haven't directly found this with LoRA myself, but I found this in computer vision. I was working on a recent project for a client and using a small model because it had to be deployed on a mobile device, and I went too small. So the model wasn't learning what I wanted it to learn despite having a good number of samples. But when I increased the model's capacity, learning went off the chart. And in capacity, I mean number of parameters, um, because it had more availability to learn. So that's Again, probably not the exact same thing, but I'm just drawing parallels from the things I've actually built and deployed. So really great blog post here from Thinking Machines. If you want to fine-tune your own LLMs, I'm working on a project to do that with Hugging Face, but you can do that these days with LoRA or full fine-tuning. LoRA without regret. Thank you, Thinking Machines. I've left a typo in this article, as you can see here. Up to here, more resources. That's how you know this is not written by AI. This is written by me. I leave the typos in. Let's go into Daniel's open source AI of the month. Okay, what do we have first? So RoboFlow, on the hot heels of RF-DETA right at the start of the year, which was a state-of-the-art object detection model, now has RF-DETA-Seg, which is segmentation. And look at that, look at the leap and bound it just took over here. Now I believe this may be because of the Dyno3 v3 backbone. Maybe Dyno V2. Okay, Dyno V2. So I'm not exactly sure what the backbone is here and the RF-Detaseg, but if you need a really high-performing segmentation model, we have RF-Detaseg preview. Measured latency 3.3, so that's really fast, 3.3 milliseconds. Measured map is 39.3. And then of course, as you increase the size, you increase the, um, the number of pixels you're using, you get better results over here. Of course, more pixels, more data to compute on, um, you sacrifice a little bit of the inference speed. But we get the best result here off RF-DataSeg at 432 pixels versus a YOLO-11x Seg, which caps out at about 40 map, but also 13.7 milliseconds, which is still fast, by the way, but, uh, not as fast as RF-DataSeg. So if you need a very fast performing segmentation model. There's an example of what it looks like. Go and check out RFDataSeg. You can of course fine-tune that on your own dataset. Of course. Up next we have ModernVBERT, which is a 250 mil— that's the most attractive part to this other than its performance— is a very reasonable size vision language retriever. So you have an embedding model which you could embed all of the text in your documents. And you can retrieve the text. But what if you wanted to embed the image of the document? So then you can capture things like images and graphs and whatnot. So if you need a visual retriever, you might wanna look at ModernVBERT. So it's at the Pareto frontier here, which means up and to the left. That's a good place to be. So there's a generic SigLIP2-large, which is a fantastic model. I've tried that a lot. But it seems like on this benchmark, which I look sideways to read, Vidori, uh, video retrieval, I believe. ColMod and VBERT performs really well, not as, um, high as ColQuen 2.5, but it is more than 10x smaller, so that'll run quite fast on a good modern GPU. So if you need a visual retrieval model, go and check out modern VBERT. So they did a great study here for bidirectional or not directional bidirectional. BERT, remember, stands for Bidirectional Encoder Retrieval Transformer. Is that correct? I may be wrong. Correct me if I'm got— I've got the wrong BERT breakdown. But bidirectional, it means it reads the sequence left to right and then right to left. So they found that multivector, so bidirectional, still outperforms single vector. So that's a cool thing to note. Training makes the difference. I believe the data is open source. Yeah, there we go. Fully open source. All model checkpoints, training datasets, and training represe— uh, recipes are released under the MIT license. There we go. ModernViber. Go and check that one out. Thank you to the team who've released that. Oh, it's also built on the ETTEN architecture, so that's a new set of encoder and decoders, um, for language that we covered in last month's Machine Learning Monthly. So really cool to see the Ettan architectures getting use already. So up next we have Rex Omni, which is a VLM vision language model or MLM, multi-multi-large language model, which turns object detection, object referring, visual prompting, pointing, OCR. And I think there's 10 vision tasks that this does into a next token prediction problem. So Rex Omni is fine-tuned from Qwen 2.5 VL3B and combines 10 computer vision tasks into one model and is able to perform on par with specialist computer vision models. So that means models that, um, are trained specifically on a task. The model— there's another typo, so again, you know it's human written— can handle text-based or vision— visual-based inputs for several kinds of detection tasks. Something of note is the two-stage training pipeline. The first is SFT, supervised fine-tuning, to get the model used to predicting next token like detection coordinates. The second is reinforcement learning via GPRO to get the model to adhere to geometrically correct outputs, e.g., the model is rewarding— rewarded for predicting the right number of boxes. So there's a demo that you can try out on your own images. Here we go here. And then there's— I've already done this, so we'll just copy that. There we go. So I put in here— this is a breakfast I went out and had with my dad the other day. But if I put in here the text-based categories: bagel, almond butter, coffee with milk, butter, Vegemite spread, which is very important in Australia, Vegemite. Uh, predict the— what it's going to do, predict the boxes here. So that's a visualization. Can we go over here? There we go. So the boxes were quite small here, but that's why we can zoom in. So it did incredibly well here. Look at that. Coffee with milk, almond butter, bagel, butter, bagel. I'm probably covering this, aren't I? Move my head. Move me over here. Coffee with milk. And then it can also do points. So pointing is becoming very, very good too. Bagel, almond butter, coffee with milk, butter, Vegemite. So REX Omni is a very cool model and I liked the paper. So if we go to, um, if we go to here, let's have a look at the paper. I'll just move my head back over to where it was. There we go. So if we go in, it's got some good examples of how they do it. What they did is they reshaped the, uh, prediction space, or they used 1,000 different tokens to say the model will when it predicts the bounding box coordinates, because models are really good at next token prediction, they turned coordinates into tokens. So if we go down to see what a data sample looks like. Yeah, there we go. So this is examples of current detectors in MLMs. So Grounding Dino, red apple, everything's a red apple. QuEN 2.5 VL, it does get the apples, but it does the group of apples. And then the optimization difficulty between detectors and MLMs. So the prediction box is classification loss for ML, LLMs, whereas in detectors it's a regression loss. So this box in a regression would be a low loss, but in a classification it basically got every coordinate wrong, even though the box itself is only a little bit off. If we had to classify these as tokens, 32, 34 is wrong, but in regression it's only 2 off. In classification, 66 is not 69, but in regression it's only 3 off. Does that make sense? So classification is just wrong, 0 or 1, whereas regression, it's only a couple off. So hard to optimize for that task. But if we go down here, here's what they did. Design philosophy of Rex Omni for coordinate prediction. So it illustrates our chosen approach: A, adopting a direct coordinate prediction strategy, and B, employing a quantized relative coordinate format represented by special tokens for efficient and robust spatial encoding. So in their relative coordinates with special tokens, they use 1 box to 4 tokens. So, and they use direct prediction, so person at these coordinates. Task formulation, 1000 categories. Yeah, here we go. Text prompt: Please detect pigeon, person, truck, snow in this image. Return the output in box format. And this is what comes— or visual prompting, sorry. You can do not only text prompting, you can do visual prompting. So you can say, hey, here's an example box, gimme back boxes that are— or items in the image that are similar to this example box. That's really cool. And then we have the output format, which is phrase and the coordinates. And then, yeah, there we go. Example of a task for outputting bounding boxes. So this is object reference start. So the reference task item or the reference target item, person, the box is going to start at these tokens and then you can just decode those into actual image coordinates. But these are special tokens that have been replaced in the model the last 1,000 tokens of the vocabulary to be specific for bounding box coordinates. And this is really cool. Here we go. Overview of the Rex-Omni model architecture. Rex-Omni is constructed upon the Quen 2.5 VL3 build backbone with minimal, minimal architecture modifications. Notably, the last 1,000 tokens of the original vocabulary are repurposed to serve as dedicated special tokens representing quantized coordinate values from 0 to 999. So we have an image, we put it through a native resolution VIT or vision transformer, then we pass it to Quen 2.53B along with a text tokenizer, and then it will output specialized tokens which are replaced the last 1,000 tokens of the original vocab, and then we get the output there. So really cool release here. I like the way vision and language models are really colliding now. Essentially, it seems like everything that can be phrased as a next prediction or next token prediction task is getting turned into that. Up next, we have an absolute paradigm shift for OCR. Like last month, it was, it was Christmas for OCR. It's almost actually Christmas. But over the last month— this should probably be months— we've continued to be blessed with high-quality open-source OCR models. So these are the trends that I've started to notice. We have— we use synthetic data to create artificial documents, which allows a model to learn with perfect ground truth. So one of the hard things with OCR data in general is that getting old documents and writing exactly what's on them is quite hard. So what you can do now that models are quite good as baseline models is just take a bunch of existing documents and build synthetic versions of them which have perfect ground truth. So what I mean by that is you take— maybe we go attention paper, right? We have a look at this paper, we pass this, the attention paper, and we'll see an example of where Alma Xia did this. We pass this to a model like Claude or ChatGPT or Qwen or something like that, or Gemini. And we say, hey, break down the structure of this page. And I might go, there's a title at the top, there's some authors' names, there's some text on the side, there's an abstract. And again, the actual information is not too dramatically important because we want to do this for any document type. And then we say, hey, generate us a HTML structure of this document in, but with different text or something like that. And so what it'll do is it'll create another document and it'll be kind of like this, but not exactly the same. But the beautiful thing about that is that we can do that at scale. So then we can export that HTML document via Google Chrome to PDF. But what do we have now? We have the perfect ground truth. So we have an image, a PDF that's been exported by Google Chrome, but then we have the exact text and HTML code that that document was used to be created. And so we can now bootstrap our model by going, hey, here's some examples of synthetic data with perfect information. And then we can take that base model and fine-tune it on actual documents. So that is what OlmOCR2 did. So if we go paper, we'll get that out there in a second. OlmOCR-bench, that's a benchmark. But before we do that, another trend is that we build layout detection and recognition into the same pipeline. For example, where do headers go and what's the reading order? Because yeah, if we go to this document, What's important to this? Do you have to read that first? No, you don't necessarily have to. Do you have to read this first? No, you don't necessarily have to. But for a lot of documents, say there was two columns, then there's a reading order generally, right? You read the left column and then you read the right column. So I wonder if that's here. No, this doesn't have two columns. So this is pretty straightforward. Like you can read this in a linear fashion, but if you have documents that have three columns, you generally read column 1, 2, 3. And so what you do in the model, you build in that this is a heading, you detect that as a heading, you detect this as a side note, you detect this as a footer, you detect this as a, as a header, you detect this as a subtitle, you detect this as a paragraph. If you get what I'm saying, you detect all the different elements of the page and then you can also predict reading order. So you might go reading order. This could be 0, this could be 1, 2, 3, these will all be 3, maybe 4, 5, 6, 7 or something like that. Reading order. So now when you're replicating your document or you're turning it into pure markdown text, you've got section detection. So then you can do the same for images or checkboxes or something like that. And then you've got reading order. So then instead of just getting a big glob of text, you now can just order that text as well. So that's trend number 2. And then trend number 3 is you take an existing VLM like as we saw for Rex Omni, you take QuEN 2.5 VL3B, which has been trained on an internet worth of text and images. So it has a good representation, hopefully, of vision and language, and then you fine-tune it directly for OCR tasks. So as we saw before in that case study of basketball players, when you fine-tune a model directly for a specific task, the error often drops dramatically. Woo! That was a mouthful. Now there's 5 models here or so that I've got, and I'm not going to go through each one because they very— all of them kind of follow the similar paradigm. And this is what I've put out here, but I'll link them all just for your perusal pleasure. Excuse me, I'm getting so excited I'm burping. Let's go to the paper for Almostia. Paper page. And then if we go to archive page— oh no, PDF, that's what we want. So if we have a look at the openness of these models, it's a release date October 2025 and the freshest stuff here. Everything's open source: model weights, training data, training code, inference code, model license, which is not the same for— although some of these have some good open ticks, um, this is almost— yeah, it's probably the most open. They actually even make the benchmark. So you could argue, well, should we trust the people that make the benchmark for their results? And look, I do. Always test the models on your own data though. And then what did they do? They created unit tests. And this is cool for reinforcement learning. So in OlmOCR Bench, we measured the performance of OCR systems by defining a set of unit tests, test cases for each document. These test cases can check for any of the following properties. Text presence checks that text phrases appear exactly in the document. Text absent checks that certain phrases do not appear. Headers, footers, or page numbers. Natural reading order checks sentences for reading order correctness. Table accuracy checks the relative position of cells within specific values in a table. So that's a big one. Table accuracy. I know in the past OCR models have been horrible at tables, but they're getting a lot better. Math formula accuracy checks that a given math formula visually renders the same way with LaTeX. Baseline robustness checks that long repeated n-grams or non-target language characters do not appear. And so these are examples of their unit tests. So if we have the original equation, so an OCR model reading this math equation, Model A might have a high math distance. Right, sorry, an edit distance. So edit distance is if we compare, compare the pure token to token, so text to text, it has a high edit distance. So in traditional systems, you might say a high edit distance is poor performance, but when you render it, it looks better than the one that has a lower edit distance. So I know that might not necessarily make too much sense, But when you, when you drill down into it, what are you trying to optimize for? Are you trying to optimize for the exact edit distance or are you trying to optimize for how the document looks? So that's something you might want to think about in your system. So if we keep scrolling down, yeah, this is the trend that we discussed before of the HTML generation pipeline. So here, this is the GPT-4o paper, I think, or maybe I don't know what paper this is from, but this is— then they generate a HTML version of this. So if we look, Figure 3, HTML page generation for our Olm OCR2 synthetic data pipeline. We sample a page from a real document, left, and prompt a general VLM to generate a highly similar HTML page, right. So there we go, we get a real document. So this is still real data, and then they generate this perfect ground truth on the right. So the rendered HTML page image paired with the raw HTML serves as supervision. So perfect ground truth for our OCR-specialized VLM. And so they scaling— they scaled that, sourced different PDFs, PDF to HTML conversion. So they do layout analysis, content rendering. This was in a loop and then output refinement. So they do it, I think, a few times over. So they lay out analysis and they render the content and then they say, hey, given this rendered content to another VLM, make sure this HTML is improved. Can it be improved even more? So they use Claude 4 Synthet for that, costing approximately $0.12 per document page. And then we see this is their open source data mixture. And then training. If we get down to the results, oh yeah, this is the unit testing. So check header is absent, check model above GPT-4 Turbo and table, check 0.6 above 0.57 and table results. If we go down here, the OLM-OCR bench, where is it? Oh, this is— so this is the different documents that are in the OLM-OCR bench. Archive documents, which is what we're reading now. Old scans for math tables. Old scans in general, headers and footers, multi-column, long, tiny text, and base. I'm not sure what base is there, but if we have the first release, so we get quite a significant improvement over the first release. The first release is 68.2 overall, and then the most recent release, which is this model, Oma OCR v2, is 82.4. Um, and then if we go back, do they have a demo here? I wonder if the demo is running the v2 version. Oh my, OCR2 demo. Okay, let's use our favorite test, attention paper. Excuse me, I'm going to use the Machine Learning Monthly. Why don't we do that? This is pretty hard to read, or might be pretty hard to OCR. I'm just going to take a screenshot of this. Right. Nice post on LinkedIn. Who's that guy? He looks pretty cool. So we're going to test the demo, see how it goes. Page 1. Holy smokes, that was quick. Oh, we get page metadata here. So let's check. That's correct. That's correct. It even got the— oh no, it did a celebration emoji instead of the ring emoji. Might not have that in the vocab. So zero to mastery, one course, new course. Did it get LinkedIn? Oh, there's the link. Does the link work? It does work. That's cool. Watch the video, fire it up to learn. And then down the bottom, learn Hugging Face, get hired. Okay, so that's pretty good, right? So of course, try it on your own documents. But that is OLM OCR. And then we've got a few more that are following the same sort of pattern. So let's just jump back into here. We've got LightOn OCR-1B, which focuses on an end-to-end pipeline, and the model is capable of handling, handling 5.71 pages per second on a single H100 GPU, which is about 500,000 pages a day, or less than 1 cent per 1,000 pages. So that's, that's a model if you've got a lot of pages you need to OCR. Paddle OCR VL, so combines a NavVis Native Resolution Vision Transformer with a lightweight ERNIE. So 3.5 language decoder supports 109 different languages. So if you need a lot of different languages, maybe have a look at Paddle OCR-VL. There's the architecture there. Chandra, which is very similar to OLMoCR2, supports 40+ languages, 9 billion parameter model. So quite a bit larger than the other ones we've been looking at the moment, or actually it's on par with the OLMoCR2. I believe this is a 7B parameter model. And then if we go down, Nanonets OCR 2-3B, slightly smaller, but this one has support for image tags, signature detection, watermark detection, flowcharts. So flowcharts, it'll even output that as Mermaid code. I'll let you look that up. So if you want to, if you need something with a lot of checkmarks and signatures, maybe have a look at Nanonets. And then we have DeepSeqOCR. Sam Witteven has a great video breakdown of that paper, by the way. And what they actually found is more than OCR, they found that you can use vision as compression for text tokens. So the DeepSeq researchers found that even with the compression ratio of 10x, so 10x lower vision tokens compared to pure text tokens, you can get a 97% recovery rate. And even with the compression rate of 20x, so compress the vision token space 20 times, so from 1,000 down to, what's that, 50, the OCR accuracy is still about 60%. So not perfect, but it just shows that it's not terrible either. So you can do 10x. So maybe in the future, long context models will compress tokens into the vision space, which kind of makes sense. When I, I remember an Apple paper recently in which they used data augmentation in the vision space to increase the token vocabulary of a large language model. Anyway, that's something unrelated. If you need an awesome OCR model, we are spoiled for choice right now. Go and try these out and see which one might suit your use case best. Now we have plenty, we have plenty of upgrades here to some open source VLMs. April 1.5B-15 Thinker, or 1.5-15B Thinker, sorry, is on par with Gemini 2.5 Flash. If we look here in the Artificial Analysis Index and it's in this magic quadrant up here. So there we go. It's, it's on. This is text only, I believe. Maybe. Anyway, I like this one because they, they did— I believe it's only mid-training only and they start with Pixtral 12B base model, which is very interesting. That model came out at the end of last year or about a year ago now, September 2024 from, uh, Mistral, and they upscale the model to 15B parameters via depth scaling. So go and check that out, the paper's there for more. The Qwen VL family expands. Now last month we mentioned that the big dog got released, the 235B, uh, and the— I think the 30B with Active3B. But now we have— Qwen are just awesome. Look at this, 8B Thinking, 8B Instruct, 8B FB8, 8B, FP8, 4B, 2B, 32B, 2B, GGUF. We got the whole suite. That is just an incredible release. In my experience, I've been trying the 8B model running locally. It works fantastic. So, uh, big upgrades there to the Quen VL series of models. If you need smaller versions, because last month was the bigger versions, if you need smaller versions to run locally, go and check those out. They're available now. Speaking of open source models, Lava 1 Vision 1.5 is a collection of OpenWeight open data. So they use a combination of ImageNet 21K, LAION-CN, DataComp 1B, COYO 700MIL, which is a collection of— well, DataComp 1B is 1 billion images. COYO 700MIL is 700 million images. SA-1B is SAM, so Segment Anything 1B. Maybe one be mask. Open training VLMs. So the 8B model is on par or better than Quen 2.5 VL 7B. So that's already a very strong open source VLM and you can get the code on GitHub, read the paper and try the demo. And the datasets are— when I last checked, they were being uploaded, so they might be more uploaded now. There we go. There's the instruct data, mid-training data, and here are the model weights. So really cool to see that all being open sourced. And this is something I really haven't seen before, is that the end-to-end training cost was also published. So $16,000, still quite expensive if, if you're not a large company or research group on A100 GPUs at roughly $0.60 per GPU hour. Then we've got B8B-SFT, which has also got some open data available. But check the licenses there because some of it was non-commercial. ByteDance also released updated versions of SA-2VA. We saw this earlier in Machine Learning Monthly at the start of the year, which is they, they marry up SAM-2 with a large language model so you can segment things with just text. So if we go— does it have a demo here? Maybe it will on the GitHub. So this is a model where, yeah, you can enter some text, which is, can you segment the person wearing sunglasses on the street? And then the model will return back segmentation mask here. And it's also the mask come out quite quick because it is purely married with SAM-2. So it's not generating tokens, I don't believe. It's just decoding a mask output. So really cool model here that you might be able to use for data labeling if you need segmentation mask at scale. And then speaking of SAM-2, Sansa models were released, which is— it allows you— so if we go here, yeah, you can scribble on it, adapts SAM-2 for semantic labeling. So what you can do is go, hey, I want the grapes, and then it'll label the grapes. And then you could do draw a box on the oranges and then it'll do the oranges in these two images. And in this case, you might do the umbrella and then it gets the umbrella. So what you could do in this case is So you had a large dataset, like for example at Nutrify, we have a lot of food images. You might label 10, uh, label images of apples and then use those 10 labels of apples to label the rest of your apples at scale. So because you've labeled 10, now the Sansa model, which adapts SAM-2 for semantic labeling by training an adapter module, can do that once you've got a few. It can now run over the rest of your similar examples at scale. Yeah. So there's the boat. We get the boat there. So really cool release and you can try that on your own images there. There's a nice demo there. I'll just get out of these. Lovely. We have some lovely small LLMs released by Granite or IBM Granite. We have 3B, 7B, 32B, and even in the smaller range is 1B and 350 million parameters, all under the Apache 2.0 license. So I like this in the fact that, um, Gemma 3, I've been using that model for, uh, a fine-tuning task for a large language model, performs really well. But now Granite 4.0, 300 mil— or maybe I got this wrong, 350— maybe it's only 300 million, might be another typo there, excuse me. Let's have a look. Did I get it right? 350 mil. Yeah, I thought I got it right. Anyway, so the graphic is wrong. This is not my graphic, by the way, but that model is slightly above the performance— oh well, actually nearly double the performance of Gemma 3 270mm-IT on the average accuracy. Now this is something that Andrej Karpathy brought this up in the talk, and we're going to get to that in a minute, which is down here. We might as well cover that now. Andrej Karpathy went on the Dwarkesh podcast, um, and talked about everything from AGI to self-driving cars, what teaching will look like in the future, because he's He is an excellent teacher. I look up to him for teaching and also AI viewpoints. He had a great point of let's have a look at this. How do you separate the knowledge of an LLM from the general token processing capabilities? And so what I mean by that, if you have— because I've been fine-tuning Gemma 3 270 mil for a very specific structured data extraction. Now you don't necessarily need a bunch of world knowledge for that. Or do you need just token processing? I don't know. So when you have a look at a benchmark that has— a model has more knowledge, it would be interesting to see a benchmark of pure token processing. Now, I might not be wording that right, but as in, it doesn't— does it— does— is the knowledge a requirement? Can you decouple the knowledge of the world with the ability to process tokens? And what I mean by that is if you have say this entire blog post and I just said to an LLM, hey, turn this into a JSON of heading paragraph, right? Do we need world knowledge to be able to know what a heading is versus just token processing? I don't know. But that's an interesting thing of like how small can your model get of that ability to just process tokens while retaining the minimal viable knowledge. So that's— I don't know how to formulate that quite just yet, but I'm finding Gemma 3 270 mil is performing really well on a structured data task after being specifically fine-tuned for that. Maybe Granite 4.0 300 mil performs better if I try that out. Anyway, there's some small LLMs. They're open source Apache 2.0. Facebook released MobileLM Pro, but it's a non-commercial license. However, it has INT4 quantization, which I'm noticing is getting more and more common. OpenAI did this with their GPT-OSSS models. It really makes the model quite small. And because I guess these models have seen so much data, the performance loss is almost negligible from the base model. Of course, if you want the best performance, use the full precision. But if you want faster performance, it's generally a trade-off, right? Quantization might be your friend. We've got a couple of cool things to finish off. OpenCode is a version of ClaudeCode, but it allows you to use any model, and the code is of course open source. So go and check out that if— there you go, 100% open source, not coupled to any provider. So that's really important in the future going forward. You don't necessarily want to be coupled with just Anthropic and Claude. You want to be able to use any model that comes out on Hugging Face, or maybe you don't, but OpenCode's available to you. Handy.computer. I love seeing small apps that are built for a very specific task. The code is available for that on GitHub as well. But what does Handy do? Allows you to speak into any text field. So the free and open source app for speech to text, it allows you to run Whisper locally and it's open source so you can fork it and adjust it to how you want. Maybe you build in a specific function for specific types of text. So that's a really cool example of a small app that's using open source AI models. Handy. There we go. GitHub is open source. Give that a star. And Emu 3.5 is an open source image generation model and that's going on Hugging Face, but it's on par with Nanobanana, Gemini, and Quen Image Edit and GPT-1 for a bunch of image editing tasks. So really cool release there. So we're at that stage where we're like 3 to 4 months or 3 to 6 months of like Frontier model gets released, open source version of that pops up. And yes, it is 34 billion parameters. You might not be able to run it on your local hardware, but cool to see that it is open source. OpenAI released two versions of their OSS models, which is for prompt guarding. So you could use this. They are much better than the original versions, specifically for classified text contents based on safety policies that you provide and perform a suite of foundational safety tasks. So what you could do with these is if you had an LLM system and you didn't want certain prompts going in there because they would be detrimental to your system, you could use these to label 10,000 samples or something like that, train a smaller model to go, hey, safe or not safe, send the prompt through. and then go from there. So really cool release there. Yeah, 20B and 120B versions, but of course they are in int4 quantization, so remember that when you're running them. And then we have RICE-Vit, which is region-aware cluster discrimination. So this was a cool paper to embed the region-specific information as well as OCR into an image encoder. So not just text, but also regions. So like, not, not directly object detection, but it knows that, hey, there's an umbrella here and there's OCR here. So this is embedding it right into the encoder. So that was a cool— that's what I thought was a cool, like, way of doing something of embedding information into an encoder. And then they combined the, uh, that Vision Tower, so the RICE-ViT, with an LLM, and they saw a fair few boosts across different benchmarks for OCR and vision understanding versus using something like a SigLIP v2, which is basically the gold standard at the moment for open source VLMs. So really cool approach to vision encoding there from the DeepGlint AI team. And then we finish off with SAM-3 research paper. Now of course This is a caveat. It's not 100% confirmed that it's the SAM-3 paper because it's still anonymous. So if we go to the PDF, yeah, Segment SAM-3, Segment Anything with Concepts. So this is very exciting. Um, Segment Anything 3. Of course, we don't know if it's the actual SAM-3 from the Meta Researchers. It may be, it may not be. It kind of looks like a meta paper, um, but it's still anonymous. So We'll pretend it is SAM-3 for now. SAM-3 improves over SAM-2 on promptable visual segmentation with clicks. So left, so you can do, hey, click on this to plus, click on this to minus, segment the plus, don't segment the minus while advancing promptable concept segmentation. So now you can do the big update to SAM-3 will be text-based segmentation, which you can kind of do now with several open source models like SART SA2-VA, which we saw before from ByteDance, where all users— where users can segment all instances of a visual concept specified by a short noun phrase, image exemplars. So you can put in an image example or a combination of both. So a striped cat will give you that, a kangaroo will give you that. You can do a round cell plus a positive example. I'll just scroll over here a little bit. Window, small window, and then hard hat, it'll get the two different hard hats there. So very exciting release coming up potentially for SAM 3. Hopefully it's still going to be open source. I'm pretty sure Meta just changed up their open source division, but we'll see. We'll see if it is open source. It'll be covered in a future ML Monthly. So yeah, there we go. So what do we do? We can use visual-based or text-based input prompts to obtain segmentation masks. And I really like the training loop, which involves heavy automatic labeling with SAM-2. And then it goes for AI and human verification. So you can go to an AI model, hey, is this a good mask or not? And then it'll go into a loop to go, hey, this might need to be fixed by someone. And the same thing with a human, is this a good mask or not? And then yes, back into the loop, keep running that. So that has been Machine Learning Monthly for October 2025. Finally, of course, the talk. We kind of covered this before. I'm not going to play this, but there is some absolute gems in this entire podcast from Andrej and Dwarkesh. Andrej is one of my favorite voices in the world of AI. But yeah, check that out if you're interested in a great listen. I listened to this on a few walks and found it fascinating. So that, my friends, has been Machine Learning Monthly for October 2025. If you want to learn Hugging Face, I've got a new course out. You can check out the link here. But as I said at the start, I won't be here at— you won't see me for November 2025 for Machine Learning Monthly. I'm getting married and so I'm going to spend some time with my beautiful wife. I will see you in the next video. Happy machine learning.