On evaluating LLMs: Let the errors emerge from the data | AI & ML Monthly

Ladies and gentlemen, welcome to the May 2025 edition of AI and Machine Learning Monthly. I'm Daniel Bourke, machine learning engineer and your host for today. We're going to cover the latest and greatest in machine learning with a specific focus on open source. All the links you need will be in the description below, and if you'd like this newsletter emailed to you at the start of next month, be sure to sign up. Well, let's jump into it. So we'll start off with my work. I wrote a short blog post about Apache 2.0 open source object detection models. Now, why is this important? Well, if you've gotten into the world of object detection, you've probably run into a YOLO model. So you only look once. So YOLO model, that's a cool background. So there we go. Easy to find. But you'll probably notice that if you start to use it, especially in a commercial application, that it's under the APGL— AGPL 3.0 license. Now, I'm not going to dive into that. I'll let you read it. But TL;DR, if you want to keep your source code closed or just generally want a more permissive license, use Apache 2.0 or MIT models. So I've collected 3 of my favorite object detection models. These have been vetted by me to perform on par or better than YOLO models, and there's links here to all of those. So the first is RFDetta, RoboFlow DettA. You can see its Pareto optimal there. We got the results there. Real-time object detection model. There's also RTDataV2, which is integrated into Hugging Face Transformers. So there's RTDataV2, you can use that there. And then there's also D-FINE, which is also integrated into GitHub and Hugging Face Transformers. And in fact, RTDataV2 is the model that we use in the upcoming ZTM object detection pro— project. So if we look for RTDataV2, you'll see it all scattered throughout. In this project, we learn how to create Trashify, which is an object detection model which will identify hand, trash, and bin in an image. And we customize it from RTDataV2 to do this specific task. So there's some Apache 2.0 object detection models if you're working on an object detection project. Go and check those out and use them because they are a really permissive license. Up next, we have the latest and greatest from the internet. So Netflix share how they created a foundation model for recommendations. This is a great technical blog post from Netflix, and my favorite points here were right at the start. Instead of working on— they said, uh, the impetus for constructing a foundation recommendation model is based on the paradigm shift in natural language processing. NLP, to large language models. In NLP, the trend is moving away from numerous small specialized models— so many different models, hard to maintain— large language model— specialized models towards a single large language model that can perform a variety of tasks either directly or with minimal fine-tuning. So similar to ChatGPT, it's one big model, does a lot of different tasks. So the key insights from the shift include a data-centric approach. So shifting from model-centric strategies, which rely heavily on feature engineering, to a data-centric one. This approach prioritizes the accumulation of large-scale, high-quality data, where feasible, aims for end-to-end training. So rather than trying to prioritize many different models and features and whatnot, you just get a whole pile of data and go, hey, model, learn this. And this is Part 2 is leveraging semi-supervised learning. The next token prediction objective in LLMs has proven remarkably effective. It enables large-scale semi-supervised learning using unlabeled data while also equipping the model with surprisingly deep understanding of world knowledge. So that learning objective of just predicting the next token based on previous tokens, that's quite a simple objective. I say simple, but it's— it takes a few steps to do it. But it's a simple objective. Hey, just given these tokens, predict the next one. And so if we go down here, they talk about their data. So to harness this data effectively, we employ a process of interaction tokenization. So Netflix, if they're going to recommend you a movie, you might click on one movie, go back, click on another movie, go back, click on a third movie, then you won't watch that. So that's an interaction piece of data. And so ensuring meaningful events are identified and redundancies are minimized. So if you have this as your reaction interaction history, you might tokenize that into something like this. And then the, the LLM in Netflix's case will go, hey, given this series of interactions, predict what the next thing should be here that the person might want to watch. So if we go down here, more technical information. I highlighted this. Firstly, during the pre-training, pre-training phase of typical LLMs such as GPT, every target token is generally treated with equal weight. In contrast, in our model, not all user interactions are of equal importance. For instance, a 5-minute trailer play should not carry the same weight as a 2-hour full movie watch. A greater challenge arises when trying to align long-term user satisfaction with specific interactions and recommendations. To address this, we can adopt a multi-token prediction objective during training, where the model predicts the next n tokens at each step instead of a single token. This approach encourages the model to capture longer-term dependencies and avoid myopic predictions focused solely on immediate next events. So this is a really interesting point in the fact that rather than if you're watching a small trailer like on YouTube, if you watch a short video, is that worth as much as watching a longer video? So if you watch a 1-hour tutorial, maybe that's worth more than just watching a 10-second YouTube Short. So I'll let you go through and read the information here. Incremental training. Here's their pipeline. So we have an input, create different embeddings, concatenate those embeddings, have a mixing layer at the end, and then you have a final title embedding there. Challenges: scaling. And the results down here, what did they— the foundation model allows various downstream applications from direct use as a predictive model to generate user and entity embeddings for other applications and can be fine-tuned for specific canvases. So that's I really like that when they create, like similar to when you have a computer vision model pre-trained on ImageNet, you can customize that to many different use cases. So now Netflix have a foundation model for recommendations. They can customize that in different areas of their system. Up next, we have Philip Schmid writes about the differences between AI workflows and AI agent patterns. So you're probably hearing a lot about AI agents and agentic workflows. I like this breakdown. AI agents, agentic AI, agentic architectures, agentic workflows, agentic patterns. Agents are everywhere. My favorite definition at the moment actually is a model in a for loop. That's my favorite definition of an agent. So we define here the difference. This is Philip Schmidt's blog, by the way. We dive into 7 common patterns differentiating between more structured workflows and more dynamic agentic patterns. Workflows typically follow predefined paths, while agents have more autonomy in deciding their course of action. So why do these matter? Well, patterns provide a structured way to think and design systems. Patterns allow us to build and grow AI applications in complexity and adapt changing requirements. When should you not use agents? Agents always seek the simplest solution first. I love this. If you know the exact steps required to solve a problem, a fixed workflow or even a simple script might be more efficient and reliable than an agent. That's my favorite point here. If we want to use workflows, use them for predictability and consistency when dealing with well-defined tasks where the steps are known, and use agents when you need flexibility, adaptability, and model-driven decision-making. So that's, that's my favorite definition of agent— model-driven decision-making— whereas a workflow is you've kind of already laid out the steps. So if we look at these examples of workflows, we have prompt chaining, So you query one LLM, one output, one, and then you put that output into another LLM. So you might have one to extract details and then another one that summarizes those details. And there's some great example code here using Gemini workflow routing. So you have a query and then it goes, the LLM router makes a decision and it passes. So you have a simple task, you might pass that off to a smaller model. If you have a complex task, you might pass that off to a bigger model. That can think for a while. Parallel, parallelization. So this is where you put them to two different LLMs at once, maybe to get two different opinions, and then you can combine them into a final output. Tool use pattern. So you can have a query, an agent that decides whether or not it needs to use a tool. Remember, tool is just like a function. So you might— a tool might be a Google search. So say you have a query, How much is this speaker in a photo? It might go and search the internet for similar looking speakers and then return back prices that it's found. This is a planning pattern. So orchestrator, or orchestrator workflow or workers. So the query comes in, LLM plans a task list and then sends those tasks to different workers. So these could all be LLMs themselves. So this is getting more into the planning stage or agent pattern. The LLM is making decisions here. This is an LLM planner. And then finally the results come back. Now these workflows are going to take— I guess with each of these steps you want to make sure your LLM is good because with each subsequent step you could multiply the errors. So if something goes wrong here, then it could really break down here. Personally, I find that my own like agent or AI use is mostly still workflows. I haven't broken into the agentic sort of model making a decision except for like a model classifying something. I don't know if I count that as a model making a decision. I'm using it as a classifier. User query. This is a coordinator. So this is making basically LLM as a product manager. So then you're— this is getting really deep here. So then you'll have a special agent 1, specialist agent 2, final output. So this is, yeah, like you're— this is the agent. You're putting a lot of responsibility, I guess, on it. And so my— I haven't found a workflow yet that requires this for my personal work, but I just see this and go, wow, even when I'm using an LLM, like even just getting a structured PDF out of one these days is still pretty hard for direct use. So I imagine there you'd have to put in some redundancy here if one of the steps breaks down. And then I really like the acknowledgments down the bottom, which is a whole bunch of different resources. I like this website, agentrecipes.com. So Together AI have some really nice graphics and how you can run those. So in prompt chaining, LLM goes through here, LLM call 2, 3. So if it passes, goes through multiple LLMs. There's a really nice simple website too. So go and check that out. Philipp Schmidt's blog and agentrecipes.com. Let's go back to the blog post. So Spotify up next have built text-to-tracks. Now this is one of my— I haven't read in industry too much yet of like fine-tuning an LLM for a specific— to produce a specific custom token set. This is one of the first times I've read that. So Spotify want to do, hey, you want to put in some text into your app and it'll produce you a playlist or a list of tracks based on, hey, play some old school rock ballads to relax, or Sunday afternoon barbecue Australian barbecue songs. That's the kind of stuff I would play. Um, now where the current methods fall short, so this makes kind of sense to me. Song titles are not descriptors. So if you have a song For example, one of my favorite songs is Saint Pablo by Kanye West. Now that doesn't necessarily reflect what's in the song. There are many different songs like that, right? So if you're trying to search for songs similar to the title name, that's not going to be very helpful. And even two songs with similar names can have completely different vibes. Song titles are ambiguous, slow, and costly. So to overcome the challenges of just putting in a song title to an LLM and going, hey, give me songs similar to this, They developed text-to-tracks. So they have a playlist dataset with title and track names, ID strategy preprocessing. So they create a list of track IDs, then they fine-tune an LLM to produce those track IDs. So this is what I really like is you find— because an LLM is, is really good at predicting next tokens. So if you're introducing new tokens, that might be— you might want to have quite a big dataset to do that. But Spotify obviously have a large track database. And so they put in a prompt, I'd love some relaxing bossa nova music. And the fine-tuned LLM produces track IDs. So rather than song titles, it maps it back to a track ID now. And so they use generative retrieval. So I like when they talk about here the role of track IDs. So one of the most innovative aspects of text-to-tracks is how it assigns IDs to songs. We experimented with 3 different strategies to find the best way to represent tracks. So context— content-based IDs. So use metadata like artist names and song titles to build the ID. Artist track name. While simple and robust as a baseline, this is probably the simplest way you could do it. Artist track name in a string. This method faces the challenges mentioned above, why the current method falls short. And then integer-based IDs. So assign unique integers to songs, uh, either in a naive way where each track corresponds to different integer track ID, or in a structured way making use of the artist-track hierarchy, artist-track integer. This approach is easy to implement. So it'd be like, uh, Kanye West, Saint Pablo, 003 integer. This approach is easy to implement, but many miss more fine-grained relationships between similar tracks, as it only relies on the track's artist and metadata. And then finally, semantic learned IDs, which is of course embedding style. Generally in machine learning projects, when you can go to a learned way, that's usually the best. So use vector decretization techniques to create IDs based on song characteristics, genre, genre, mood, and style. For example, two holiday songs might share part of the ID 0 1 0 or 0 1 and 0 2, indicating their similarity. These identifiers can be seen as zip codes identifying portions of a vector space where a track lives. We tested two different vector spaces, one based on text embeddings, i.e., embeddings, titles of playlists where the track appears in the training set, and one based on collaborative filtering embeddings, i.e., built on mining patterns of songs appearing together in playlists so that if, if existing playlists have a song in it, that's where you can use collaborative filtering. So the long story short is that the embedding-based approach with semantic IDs worked the best, but a close second, which is a great way to sort of start experimenting with your own problems if you want an LLM to generate custom IDs, is the artist track integer pattern. Integer, integer, integer. So that actually didn't perform too far off a filtering-based approach, which probably required a bit more setup than just assigning a numeric value, a numeric integer. So really cool. And of course, their program, their model outperformed traditional keyword indexing by over nearly triple actually, and then even outperformed an embedding-based search by well over double. So if you want to read how Spotify built Text2Tracks and a cool way of embedding tracks into a custom ID and then fine-tuning an LLM, go and check out that blog post. Up next, we have LinkedIn building Jude, so Job and Understanding Data Expert. So continuing with the trend of turning everything into an LLM, it's just a simple workflow, right? Like you get your data organized, but then you have one big model. And rather than training several different models for classifying what job is which or which person should go for which job, you have a giant foundation model that knows your entire database and the embeddings there because it's had so much data in there. It's, it's following that trend of Hey, let's just put everything into one big model. If you have the resources to do that, that's fantastic. I still really like small models as well. So the, the problem for LinkedIn is they have job postings and then candidates. So if I have a candidate profile on LinkedIn and I'm a machine learning engineer in Brisbane, Australia, that's where I live, which are the best jobs for me based on the job postings that are on LinkedIn? So maybe in my recommendation system it won't have California-based jobs, because I don't live in California. So that's their problem. But then if they go down here, they had a whole bunch of different models. And what they did was, yeah, drawing from our experience with Pensieve, our previous generation embedding platform, we knew that Jude's design could support several key enhancements, including operational efficiency, replace standardized features with LLM-derived representations. So yeah, one of the big problems I saw with a lot of these sort of problems is they had several different re-ranking models. So somewhere in there, but they had— let's imagine LinkedIn, they have 100 different models doing all different predictions. And when you have to update them, that's a lot of work to update 100 models at once. Whereas if you have, as Netflix did, one giant foundation model, you can then just use those base embeddings to fine-tune later on. So eliminated dependencies on imprecise smaller ML models, hard to maintain taxonomies, That's another thing. If you're using smaller models, you might need to define terms in structured databases, whereas an LLM can kind of work around, um, more general features. Rigid upstream pipelines. Then they improve their system architecture due to migration from Lambda to Kappa architecture, so easier resolution of time travel issues, cost-efficient GPU inference, and simplifies maintenance by eliminating the need to monitor and recover failed scheduled jobs. So what they do is they do fine-tune representation learning pipeline. I believe they use Mistral. Yeah, there we go. For example, the embeddings coming out of the Mistral family models clearly benefit from the Hadamard product. That's an operation here. Hadamard product is known as element-wise product. So just multiplying these two. Let's go Hadamard product versus dot product. See if Gemini can give us an overview. The dot product and Hadamard product or element-wise product are distinct operations in linear algebra. The dot product results in a single scalar value, while the Hadamard product produces a vector or matrix of the same dimensions. Beautiful. And then we go down, they use 3 different types of loss. So our model's performance significantly benefited from the extensive loss function engineering. The optimal architecture combines 3 complementary loss functions: binary cross-entropy loss, so that's essential for the core classification task, —so classifying, hey, does this job match or not—contrastive InfoNCE loss, which is commonly used for retrieval and semantic search tasks, and VP matrix loss, which provides robust outlier handling as well as effective utilization of weak convergent mechanisms. So if we go down right to the bottom, there's a whole bunch of different information, but the results here are the Jude embeddings have ramped online, have been ramped online, replacing the overlapping standardized features in both job recommendation and search and search L2 rankings, leading to +2% qualified applications, -5% dismissed to apply, and plus, call it 2%, total job applications. This is our highest metric improvements from a single model change. The team supporting talent initiatives has observed during that half year. So that's really cool. We're now starting to see LLM-based models integrate into industry and are providing results. So really good blog post there from the LinkedIn engineering team. Now up next, we have one of my favorite AI companies, Mixed Bread. And this is a great blog post diving into the hidden ceiling, how OCR quality limits RAG performance. So if you're a large company, you have many different documents, you might run OCR over them, optical character recognition, to process PDFs. But the best results you're going to get with a RAG system, so Retrieval Augmented Generation, are always going to be based on how good your retrieval is. So if your OCR text is not that good, your retrieval is going to be hurt. And then if your retrieval is hurt, your generation is going to be hurt because you don't have the right information. And so I'll let you, let you read through the entire blog post if you want. But I like the takeaways here. So OCR creates an invisible performance ceiling. Text extraction errors significantly limit both retrieval accuracy and generative— generation quality in RAG systems. Benchmarks reveal a substantial gap. Even leading OCR solutions fall 4.5% short of ground truth text performance, particularly with complex document layouts. Vision-only generation is not ready yet. Despite rapid improvements, multimodal models still cannot reliably generate precise answers directly from multiple document images. And multimodal retrieval beats perfect text. So this is something that I found pretty cool. So our vector stores outperform even perfect text by 12%. So even if you have perfect text, because what's a document, right? A document has figures, images, text. If you had perfect text, you'll get great retrieval. But there are things, questions that can't be answered generally from straight text. You need the document or the figure or something or the graph. By 12% on retrieval accuracy and recover 70% of generation quality lost to OCR errors while simultaneously simplifying architecture and enhancing future compatibility. So if we go right down to the bottom, I loved the— there we go. Some examples. Recall, retrieval. So MixBread Vector Store got incredible results there. They tested generation results. So if you have better retrieval, you get better answers. So using Gemini 2.5 Flash, they tried to generate— or sorry, they retrieved some documents, used Gemini 2.5 Flash as their generation model, and then used GPT-4.1 as the judge. So based on this question and this answer, which is better. And then of course, Perfect OCR and Perfect— so this is the ground truth dataset. If you have the perfect information, so the Perfect OCR and Perfect Retrieval, that got the best results. But then the MixBread vector store, which is multimodal, and the MixBread OCR got second best scores here, even beating out Perfect OCR and Retrieval. So that's really cool. Down the bottom of the article here, they have some illustrative examples of where handwritten data or where OCR only kind of fails. So there's examples of forms that are filled with handwritten data. So if the OCR doesn't recognize the figures here, so that's pretty hard to even for me to— I can read it, but I'm going to read it a little bit slower. But an OCR system might fail to retrieve this, whereas a multimodal system embeds the whole page as a screenshot and then can retrieve that. And then you try to generate from that. So the multimodal approach. In contrast, the multimodal system processes both the structured form and the handwritten financial figures by analyzing the document's visual layout and handwriting patterns. This holistic understanding allows it correctly to identify the total revenue as 2.75— or 775,060, along with component values of $2,325,472 for operating revenue and you get the points here because this is— yeah, when processed by a standard OCR engine, the crucial handwritten— the crucial handwritten financial data is often missed or entirely garbled. So check that out. Question marks there. So really good blog post. If you're building any kind of RAG system, I'd highly recommend you go through this and see how multimodal embeddings can help with your retrieval step. With the generation step, text is still better. That was another finding that they found here. So retrieval multimodal works best. Generational text only generally works better for now. Up next, we have two blog posts from NVIDIA. The first one is running Hugging Face models instantly with day zero support from NVIDIA Nemo Framework. So the NVIDIA Nemo Framework is designed to speed up your Hugging Face models, and you can also export to VLLM, which, if you haven't seen, is one of my favorite inference engines for large language models and actually many different kinds of models as well, such as classification models. I used this to— VLLM to speed up my Qwen 2.57B model by about 5x on my local GPU. So go check out VLLM. And now NVIDIA is bringing their Nemo framework to help speed up it even more. So if you want to speed up your models on NVIDIA GPUs, check out this blog post. There is also a grandmaster pro tip from winning first place in a Kaggle competition with stacking using QML. So we covered QML in a previous video, but QML essentially allows you to bring clustering algorithms and other scikit-learn-based algorithms. Is scikit-learn here? Scikit-learn. There we go. QML is compatible with scikit-learn, so bringing scikit-learn algorithms to the GPU. And this is from a grandmaster— how they won a competition was they stacked together 75 different models and then filtered those models to, um, in level 2. So level 1 was, hey, let's get 500 models run them with QML because it's so fast. You just train models on a whole bunch of different stuff, filter the best ones, and then combine them even further with a weighted average. And that led to the best result in a Kaggle competition. There is also the way to different, different ways of predicting features. So if you have a target here, the problem was predicting how long will a person listen to a given podcast. There you can break this down. You can predict the target directly. You can predict the ratio. So an alternative to predicting the given target is to predict the ratio between target and feature. You can predict the residual. You can predict the missing features, and you can predict pseudo labels. So really cool breakdown here of how a Kaggle Grandmaster will break down their approach. So not just trying to predict the target, looking at the problem from many different angles, and you can look at it from many different angles in terms of data and many different angles in terms of models. And that's where QML really comes to help, is because it speeds up your models, you can try more out. Up next, we have two present— we have a presentation and a blog post from Benedict Evans. Benedict Evans is an economist. So if we go through here, there are some great graphics and points about will generative AI be the next big thing. And I'm sure A lot of tech nerds like me agree, using it almost every day. But then there was a problem here of do they have product market fit and who for? So if you're in the tech world, you're probably using them all the time. But a lot of people daily active. How many people use generative AI chatbots in the USA? So daily active, about 10% of people, whereas weekly active, it's closer to 30%. So something to think about is that You may, and you and me may use them every day because we're on the internet, on computers all the time. But people who aren't in the industry, the tech industry, maybe not are using them as much for now. It may be, it may take a while for new tech. And that's another one of the points Benedict Evans sort of argues, is it may take for new technology, it takes quite a long time for it to get to every single industry. And that's where he goes on, expands on this point here in this blog post here. So if you want to check it out, highly recommend just for an interesting thought piece about GenAI and how its adoption is going. Actually, there's one more graph I wanted to show there. So Deepseek just proves the point. So this is model quality by release date. If we go back to December or the end of 2022, of course OpenAI's model was by far the best. And then they kind of had the lead for about, let's say, a year and then, or maybe just over a year. And then now we've got probably half a dozen models which all perform on par from Deepseek to Gemini to Claude to Groq. All of them are performing quite, quite on par. This is a great write-up of 5 Steps to Analyze Your LLM Application Failures by Alex Strick Van Lischothen. I love Alex's blog, one of my favorite blogs. So go and follow him if— or read his blog if you want to learn about just machine learning and developing systems in general. But I really like this one on how to evaluate your LLM system. So it's one thing to build, and a lot of the steps, if you've built a traditional ML system, it's a lot of the similar steps there. It's the repeat and analyze process. So You make some predictions, you measure them, you improve them, and you analyze again. And I like this, this graphic here, nice and simple. So you start off, you bootstrap a dataset, whether that's synthetic or not. One of my clients that I'm talking to for a computer vision project is, um, the first thing I basically said was, how do we evaluate? What's the, what's the goal here? We can train a model, but how do we evaluate that on your system. And so they want to deploy it to a mobile device. I have plenty of experience doing that. And so what I said was, hey, we're going to create a dataset, we're going to read and label, we're going to find failure modes, label traces. This is for an LLM, but I'm using computer vision as an example. Quantify and iterate. So we build the dataset and then we try it on where we want it to deploy. So real experiments where you want it to go. And then we've, we've subsequently improved that over time. So we always do end-to-end testing. We train the model, we see how it goes on a phone, and then we bring it back. Or we, we not necessarily on the phone, but we recreate that scenario, whether that's through a Gradio demo or a custom demo or something like that. Always, always test on where you want it to go. So create your initial dataset, look at your data. I really love this. So look at about 100 examples. That was my tip to them as well. We want to create a really good test dataset of at least 100 items of different things that we want to classify with our computer vision model, or in the case here of your LLM. And then you want to continually run tests over those. So pitfalls to watch out for and then discussions here, reflections. I really love the— is there a custom one here? Yeah. So emphasized that custom vibe coded interfaces were also recommended, especially when you have a lot of metadata and tools calling that you might want to present in a certain way to foreground certain elements, etc. And this is where you'll spend 80% of your time for 100 traces. So 100 traces is 100 LLM inputs and outputs. Could take something on the order of an hour. Read each trace, write some brief descriptive notes about the observed patterns or problems in actions where things are going wrong or are unexpected. Importantly, you let the categories of errors emerge from the data rather than coming up with preconceived ideas of what the categories already are. So I love that point of let, let the native errors emerge from your data. So rather than going, hey, I'm building this model, I think this is where it's going to error, build the model and then run through your traces and then see what happens there rather than trying to imagine what's going to happen. Of course you can imagine what's going to happen, but always test in the real environment if you can. So great blog post there. Great blog overall, ml-ops.systems. Oh yes. Up next, we have Daniel Bourke's open source AI of the month. And first up, we have FastVLM, which is— FastVLM is a vision language model by Apple. It's open source. There's the architecture there. And they— one of my favorite things is they have a demo app where you can run it directly on your iPhone. So this is running on my iPhone. Can I just go copy image address, put that up there? This is running on my iPhone 15 Pro. Time to first token. This is where FastVLM really improved over previous models is it gets quickly. What's a big thing in user experience on the device, right? You want it to be fast. You can— if you can see something happening quickly, then you're willing to kind of wait. So I just said, hey, what's written in the image in my prompt? Welcome to Machine Learning Monthly. It missed the typo that I did. I didn't put an N there for Machine Learning Monthly for May 2025. And then it output the rest pretty well. So that's really cool. But if you want to go and have a look, we have a GitHub repository here from FastVLM from the Apple team. And here's some examples of it running directly on device counting number of fingers reading and examining which emoji is which. And then if we go to the demo app, see the app folder, you can run this on your local device. So if you want to see how a model like VLM gets run on device, go and inspect the code here. We got FastVLM Xcode project. FastVLM app. There we go. VLM app. We're a whole bunch of different things here. I'm not a Swift developer. My brother is, so he knows more about that, but I'm trying to learn more. Oh, I just went into the same one, didn't I? The model. How does the model get loaded? So that's a project, a side project for me, is to go through this and understand how to load one of these models directly on device. So really cool release from Apple. And on the trend of small VLM models, NanoVLM is, uh, has two things here. We have a GitHub repository of pure PyTorch code. We have written a tutorial on NanoVLM, that's this one here, the blog post. I'm going to give that an upvote. And model vision_transformer.py, which is 150 lines, a language_decoder.py, 250 lines, and modality_projection. So if we go and have a look at the— if you— the TL;DR of this is if you want to learn about how to create a vision language model in PyTorch from scratch, Then we've got a code here. This is how you get your data ready. This is how you generate stuff based on an input language model, modality projector, vision language model, vision transformer. So they combine in the— this is becoming quite the standard architecture for a VLM. Have an image input. This is your vision encoder backbone. Then we fuse that together in a language model. This is a text input tokenizer. Then we have some kind of output. So if we dive into one of these files, We might look into models, Vision Transformer and VLM. So the VLM, Vision Language Model, we have the vision encoder and the decoder, which is the language model. And then we're going to go through here, the FORD. So we have an image embedding and then we have a token embedding and then they combine them, generate. So image embedding, token embedding, combine them. So replace image tokens with embedding, and then we go pre-fill phase self-decoder. So we're going to decode here based on the embeddings. That is really cool, and it's all written in PyTorch code. So a good side project if you want to learn how to make VLMs would be to replicate this from scratch. I would personally— maybe I'll actually do this. Let me know if you want to see a livestream of just writing this from scratch. Modality Projector. So this is going to combine or turn, I believe, image— yeah, vision to language. So this is this part projecting the vision input into a language input. Speaking of small VLMs, here's a small VLM demo that runs directly in your browser. So if you want to see how to run a small VLM model directly in your browser, I think this is where the industry is going. Is have these models run directly in Google Chrome or on your device. I eventually see in maybe 1 to 2 years we've got GPT-4 level models just running natively in an app on your rather than an API call. So we're getting very close to that. Next, we've got D-FINE. So a series of state-of-the-art object detection models. This is where I kind of mentioned this at the start in my blog post Collecting Apache 2.0. But these are all different backbones for state-of-the-art object detection, real-time models. So they perform better or on par with YOLO at all different values. Yeah. So that's, that's some incredible predictions there. So person, person, person, person, person, tennis racket, clock. I got the clock wrong, but it was close enough. You could filter that with prediction probabilities. So really cool release there if you're into object detection. And then we have OpenVision. OpenVision is a series of 20+ open-source CLIP-style models. So a lot of different things here. And that was a link to Hugging Face. Beautiful. So we have— this is completely open-source data, completely open-source training pipeline, and completely open-source weights. So if you want to train your a CLIP model, or if you want to use one as your backbone, you've got a whole bunch of different ones here. And then if we go— yeah, see the graphic here. Model: OpenVision. Data: Open. Training: Open. Evaluation: Open. Model numbers: over 25. And so they perform the best model here, uh, 80% at top 1 on ImageNet. And that's what's available in JAX and PyTorch. That's so cool. I believe for the dataset they use DataComp. Yeah, DataComp-1B. So if we look up Recap DataComp-1B, this dataset is available on Hugging Face as well. So what they did with this dataset was they went through DataComp-1B, which is about 1 billion images, and they recaptioned the images using an advanced, uh, VLM. Lava 1.5 Llama 3 AB to enhance the alignment and detail of textual descriptions. So it's quite a large database if you want to download it all, but you could definitely use this dataset as a way to extract your own custom images. And because the captions are really enhanced, you might be able to filter for the text quite, quite a bit better. So Open Vision, a lot of open source CLIP models there. Big shout out to the research team for that. So this is a release from Meta, which is Track Anything in Video on Device. So if we go to the demo and the code and models are on GitHub, EdgeTAM, I believe it's Apache 2.0. It's SAM-2, so Segment Anything 2 sort of style model, but it's 22 times faster on an iPhone. So I really like this. This is, I'm, As you can tell, my bias is towards edge-based models. I love small models that you can run on your own hardware. So they have some demos here. I made a little movie of tracking the dog in the image. Let's do— how about we do the cup? I want to track— can I track this cup? And I'm going to exclude these two cups. And the ball. I want to exclude that. So let's see what happens. Okay, so we see the reference mask here and then we got the video. How does this go? Oh wow. It even worked despite them moving it around. That is cool. And then of course there's a few more demos here. I wonder how well it works on this bird. Okay, so I'm going to include the bird and its wings, and I'm going to exclude whatever's over here. So it's got most of the bird in the reference marks— mask. That's great. I'm going to exclude that, make sure it gets most of the bird. Okay, let's track. We got the output video here. Wow. So if you wanted to, if you were making a video editing, you could really apply some effects to that now because you've got the mask of where to put it. And it's even getting the wings flapping. That's incredible. And those wings are moving quite fast. So you can imagine an app where you just tap on, hey, track this, don't track that. That is really cool. So really cool release from Meta there, EdgeTan. Up next, we have Trackers, which is an open source— well, speaking of tracking things in a video, which is an open source multiple or open source library for tracking multiple objects in video. So this is from the amazing RoboFlow team and we have Trackers. And if you want to track things in video so you can plug in an object detection model and then use a tracking algorithm to go, okay, I've detected these— how many planes is it? 8 planes. Now track their movement in this video. So if you're making security cameras, you might detect a person and then track their movement throughout a shopping center or something like that. So really cool pipeline and it works well with Supervision, which is another open source library from RoboFlow. So if we go Supervision for all of your computer vision tool needs. So if you need to display boxes on an image, if you need to display masks, This is— Supervision is a really great library. So big shout out to the RoboFlow team for the release of Trackers and of course Supervision. So up next we have Osmosis, which is another beautiful example of a fine-tuned model for a specific use case. So unstructured LLM output will output directly text, whereas Osmosis is a fine-tuned version, I believe, of Quen 3, Quen 3 6B. To output structured data. So built on top of QuEN 3, we first established a baseline format using 10 samples of randomly generated text and their JSON representations. We then applied reinforcement learning to approximately 500,000 samples of JSON natural language pairs consisting of either re— reasoning traces and their final outputs or natural language reports with their expected structured formats. So the goal here is to directly generate structured text. So we have a JSON schema and correlation trends, performance, engagement. So that's really cool. So another great example of, hey, we have a specific data type we want to output, going to fine-tune an LLM to output that specific data structure, just like Spotify with text-to-track. And then up next, another one of my favorite demos we have, or favorite models of the month. NVIDIA Parakeet V2, which is a commercially available model for text-to-speech— sorry, automatic speech recognition. So it's speech-to-text, other way around. And it performs— if we go down here, we have some results. Really low error rate. Yeah, 6 average word error rate is 6.05, which is, I believe, lower than Whisper V3 Large. And it works incredibly fast. So if we have, I believe it was 3, 3, yeah, has a real-time factor of 3,380 on the HF-Open-ASR leaderboard. That is incredible with a batch size of 128. Let's go and try it out. Hey, I know there's a demo here. So microphone, let's record something here. Hello, my name is Daniel Bourke and I am testing the NVIDIA Parakeet model. It is for recording, or we're doing a demo for Machine Learning Monthly of May 2025. And the date today is June 7th at 9:40 AM. And we are having a great time. It's a beautiful, chilly morning in Brisbane, Australia. Let's see how this goes. That seemed incredibly fast. Did anything come out? Okay, there we go. So let's read it out. Hello, my name is Daniel Bourke. Spelled my last name wrong, but that's okay. I am testing the NVIDIA ParaKit model. It is for recording or we're doing a demo. Wow. They got that. I mistakenly messed up there, but it got that for Machine Learning Monthly of May 2025. The date today is June 7th at 9:40 AM and we're having a great time. It's a beautiful chilly morning in Brisbane, Australia. Wow. Got that perfectly. I would say perfectly. And it outputs timestamps. So really cool model available for commercial and non-commercial use and automatically. Yeah, punctuation and capitalization as well. So if you need an incredibly well-performing Automatic speech recognition model, 600 million parameters is not that large these days, so it could run on many different devices. Go and check that out. Big shout out to NVIDIA for that release. Up next is LLMDET, which is an open vocabulary object detection model capable of detection, detecting objects in images and using natural language. I really love this. Grounding Dino is a similar model. I actually just created a little demo for this one. There we go. There's me. Thanks to Daniel Bourke for your valuable contributions. Let's check out the Gradio demo. So you can see on several different benchmarks it outperforms Grounding Dino, which is another open vocabulary object detection model, which is really great. LLMDET seems to be the best, or if not the second best on many different benchmarks. And so we— a bunch of different sizes here, but let's just test it out on the demo. You can see that here, LLMDET. And if we try one of these images, there we go. So that was just based on text-based input. So we have bin, water bottle, hand, shoe. We've got the bin, water bottle, hand, and shoe. And then this is what I had for lunch the other day. So this is all based on text-based input. So we got lettuce, beautiful orange slices, delicious eggs over here. We got some red cabbage, we got some cheese, and we got the pear slices there. So it's done incredibly well there. So fusing, fusing, fusing, fusing large language models with detection models for grounded object detection. Thank you for the release there from IC Laboratory. And yeah, go and check out the demo if you want to try it out. Built by yours truly. Up next, we have two new releases from Deepseek. We have an upgraded version of R1, their flagship model, and then we have a version of Qwen 3 8B that has been distilled by their flagship model. So across the board, the new model outperforms the original model, and it's open source, but it is quite hefty—685 billion parameters. There are many, uh, inference providers if you want to run that via an API. I personally don't have the amount of GPU power to run this on my own hardware, but it's competitive with OpenAI's O3 and Gemini 2.5 Pro, the recent release, as well. So that is incredible. And of course, Quen 3 235B. But then if you look at the original Deepseek R1, it has leapfrogged that quite substantially. So incredible release from the Deepseek team. And then we also have, uh, Quen— the Quen 3 version, which if we come down here, we have Deepseek Quen 3, which is similar to Gemini 2.5 Flash 5.4 reasoning plus 14B, and it's, it's even close to Qwen's largest model at only 8 billion parameters. So really cool release there, fine-tuning Qwen 3 8B model 4. While this is again on specific benchmarks, always test these models on your own data. You might find that perhaps Gemini 2.5 Flash performs better on your tasks rather than— I believe these are math benchmarks. So always check it out on your own data. But really cool release from Deepseek R1 or Deepseek team. Up next, we have Xiaomi releasing open source MIMO LLMs and VLMs. I love this trend of all the large tech companies. So Xiaomi, the mobile phone maker. So if we look up Xiaomi, Xiaomi, do they make— oh, mi.com. That's cool. IPads, or they make similar— the Xiaomi Pad and phones and whatnot. They've released two. So MIMO-VL, which is a vision language model, which I believe outperforms Quen 2.57B, which is probably my favorite VLM at the moment. I'll have to try this out. I haven't tried this out, um, it's only just released, so I haven't tried this out personally, but it looks like on their benchmarks, the MIMO version is outperforming Qwen 2.5 VL7B. If you haven't tried that model, very powerful model for 7 billion parameters on basically all of the different benchmarks. So really cool release there. That's— they've used RL to improve its outputs. And then we have the MIMO versions, which are actually just got updated as well. Which is— well, only updated one day ago. That's really cool. Which is text generation. So pure text only. So if we come down here, RL, it's improved upon its previous self. So this one was only just released and now this latest one, which was the 30th of May, I believe, is surpassing Deepseek R1. Must be the first version of Deepseek R1. So that is really cool for such a small model and they are available under the MIT license. So big shout out to Xiaomi for those releases. Thank you very much. Up next, we have ByteDance release Dolphin. So the document extraction models are getting really good. So Dolphin is a relatively small model. I believe it's 398 million parameters. So definitely capable of running on, on local hardware, many different GPUs. And what it does is, is extracts information from documents in a streaming fashion, or parallel. And it also outputs different— so it'll parse a document at different, how would you say, levels. So it'll parse it on a page layout, it'll parse it on text paragraph, it'll parse it on a table, and it'll parse it on formula as well as I believe they have 16 or something different types of information. And so the page layout goes, hey, which, which order should I read this content in? And then it'll, it'll read the text paragraph. So it'll go, okay, now let's translate this text to or read the, read the PDF and turn the text into legible text, editable text. Then we'll take the table and turn that into HTML, and then it'll take a formula and turn that into LaTeX. So that's really cool. So yeah, there's your layout. Layout and reading order. So if you have a heading here, paragraph here, which way should you read that document? So it performs incredibly well. Do they have any benchmarks here? Page level parsing. I believe it might be in the paper. Let's take a look at the paper. So if we look at the results of Dolphin, it's basically the best or competitive on every benchmark and even outperforms all of the proprietary models. So it falls short to expert VLMs. So these are experts being specifically trained for Tabpedia, I'm imagining, is for table extraction. And it's, it's quite close there. So really cool to see such a small model be so powerful, outperforming all of these here, Gemini 1.5 Pro, GPT-4o, and much larger models there. So really cool release. If you're looking to extract information from your documents, go and check out Dolphin. By ByteDance and it's available on Hugging Face under the MIT license. So really cool. Up next, we have some papers. So we have Count-GD, which is a paper, but it's also open source. So let's have a look. What Count-GD is supposed to do is it can count items in an object— in an image, sorry. So you can put in a text-based prompt Or you can put in a visual-based prompt and it will count different items in the image. So in this case, strawberry counts the number of strawberries in the image. And then all the code and models are available on GitHub. And then, oh, this space is sleeping. That's okay. There's a demo here that is usually available on Hugging Face Spaces, but I'm just reawaken it up. So if you want to go and check out that there, I've got the link to the demo there. You might have to re-wake it up. So beautiful release there. If you want to count objects in images, check out Count-GD. And then we also have Seed 1.5 VL, which is a new closed source model from ByteSource— ByteSource— from ByteDance. But I really liked the paper. So this is their flagship VL model. You can access it via API. But if we look at the grounding demo, I'm loving how a lot of these new VLMs can detect in images. So detection, creating a detection dataset, usually a long process. But with these foundation models, yeah, this model, so it uses Count-GD to count objects and images too. Do we have counting? Object counting. Here we go. Chain of thought counting. So count all the strawberries. 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12. Has it got 12? There are 12 strawberries in the picture. How cool is that? So really cool release here. Can even detect text. So it's got visual grounding capabilities, but I liked the paper specifically for— if we go to the PDF specifically for— there's one section here where they talked about data mixtures. And now this was really relevant to me on a project that I'm working on, on how do I take a large open source dataset of, say, 21 million images? How many do I filter down to use to get kind of good results if training on all 21 million is not available for me? And here we go, Section 3.1.1. So these are the results. If furthermore the distribution of visual concepts within the raw text-image pairs adheres to long-tail pattern, to empirically test this observation, we conduct a sandbox experiment using Biotrove. So Biotrove is a large-scale dataset for species classification containing 161 million images spanning 366,600 species. So a very long-tail dataset. We train a 1.1 billion active parameter variant of our VLM using 3 distinct data distributions. So randomly select 46 million images. Max 1K, randomly get 41 million samples selected with a maximum of 1,000 samples per species, ensuring inclusion of rare species. Max 100, 15 mil. So 15 million species samples with a maximum of 100 samples per species. Providing greater relative exposure to rare species. So this is a cool breakdown here of if you did the random, it performs really poorly on rare species. So 10% and the average is 44. If you did the maximum 1K, 46 million for one epoch, you got great performance on balanced 10K dataset. You got okay performance on rare 2K and an average of 62. But if you did the max 100 samples per species, so the least amount of data, you got a poorer performance on the balanced dataset, but a really good performance on the rare dataset with an average overall higher of 74. So this to me tells me that if you want to balance your dataset, of course you have to test it on your own data. But if you want to, if you have a really large amount of classes and you want to make sure that it gets the rare ones really well, you should probably cap the number of samples at a certain value for all of the samples. But if you want to make sure it does the best on the samples which make up the majority of your database, of your dataset, you might want to increase the number of images per sample and then you may take a hit on the rarer ones. But can you sort of say, hey, they are rare, so it's okay that we take a little bit of a hit there. What's— in other words, where is it better to take a hit? Is it better to take a hit on performance on the more common samples? Or is it better to take a hit on the rarer samples? So really cool. I like that way of balancing a dataset. And this was a good example because it was quite large, 161 million images and very diverse. So 366,000 classes. So really cool release there from ByteDance. We also are now up to Some company releases. Now there's been a lot of coverage on this, so I'm going to spend probably less time here. Google have enough resources to market their own I/O event, but basically Google's I/O event 2025 was Gemini is everywhere. My favorite release. We've got a list of highlights here. I've linked the highlights there, but my favorite release was Signjammer. One of them. Really cool. Of taking in a visual input of sign language and then uses Gemma to transcribe that to text. So Gemma is an open source version of Google's Gemini. So SignGemma, really, I think it'd be really cool to go the other way as well. So text input, output a video of sign language. That would be really cool. And then of course Gemma 3N, which is two open source models that are available on device. So this is 4B parameters of Gemma 3N. And it's optimized to run on, yeah, your local mobile phone. So that's what the 3N stands for, nano. And then there's a great video breaking down how Gemma 3N works. And then if we go to Hugging Face, Gemma 3N is available on Hugging Face as open source as well. So go and check that out if you're looking at open source models along with FastVLM that can run on device. This is a great example of, of where my— of course, it's going to be the large scale models that probably always work the best because they just have more compute power. But I'm really excited to see where different models go on device. Mistral have released the Mistral Agents API. My favorite part of that was the cookbook. So if you want to go see the cookbook, there are some great examples on how to build different agents that really ties in with what we talked about at the start with the agentic patterns. So remember, my definition of an agent is a model in a for loop for now. And then if we go OpenAI, have Codex, which is their software engineering-based agent that can basically plug into your GitHub repo and make pull requests and whatnot. I haven't personally tried it out. I haven't had a use case for it just yet. And but Yeah, you might want to see if Codex suits your workflow. And then of course we have Claude 4, Sonnet, and Opus. They perform quite well. I like Claude basically as my daily driver for chatbot interface. I go to Claude.ai and I found Claude 4 performs quite well. And I definitely think that in my experience, Claude is the best coding for my needs. I know that Gemini 2.5 Pro beats it out on benchmarks, but to me it's between— I usually use Sonnet. Opus is a bit slow for me. O3 and Sonnet 4 for me have been the most helpful in terms of different programming things. Gemini 2.5 Pro, sometimes it's good, but it just misses things for me. I'm not sure, just from a purely anecdotal standpoint on my behalf. So go and check out Claude for if you haven't. And then we have media, podcasts, and more. Microsoft released a free AI agent course for beginners on YouTube. So if you want to learn agents, Microsoft have a free course that you can go through there. Which agent AI for AI agent framework to use, Agentic RAG, AI agent planning. So go and check that out. A bunch of different videos there. I really liked these two podcasts from Logan Kilpatrick. And the Google Gemini team. So this is Google for Developers deep dive into long context. So it's an hour interview with Nikolai Savinov and about how Gemini built the long context window from a research perspective and then deployed that into production. And then this is another one for interview with Sergey Brin, the co-founder of Google and computer scientist working on Gemini. In reaction to a year of progress with Gemini. So Sergey Brin, co-founder of Google. My favorite takeaway from that was that he said, this is the most exciting time for computer science in my whole career. And that's pretty big coming from the person who founded Google. So that has been your issue of May 2025 for Machine Learning or AI Machine Learning Monthly. If there's anything you'd like to see in a future episode, let me know. Don't forget, all links will be in the description. And if you want to sign up to get this article as an email in your inbox at the start of every month, be sure to go and check out the link. Otherwise, happy machine learning, and I'll see you next month.