Open-source AI models are surpassing closed source (fast) | AI/ML Monthly

Hello, Daniel Bourke here, machine learning engineer. And in this video, we're going to do a video walkthrough of my newsletter, AI and Machine Learning Monthly, which covers the latest and greatest, but not always the latest, in the world of machine learning and AI, with a specific focus on open source. So if you want a link to the text version, it'll be below. If you want timestamps, they're also there too. Let's dive right into it. I've got a few things that I've been working on here. The first is a guide to bounding box formats and how to draw them. Next is a reveal of a project that's coming up, building a custom object detection model with Hugging Face Transformers, a small app to calculate how much memory a machine learning model needs, a blog post on beginner's guide to embedded machine learning. In other words, how to get machine learning models to run on device. And there's a video version of ML Monthly January 2025, which was from last month. So be sure to check that out if you haven't seen it. So the first is, A guide to bounding box formats and how to draw them on learnml.io. I want to release more tutorials and blog posts on here. So this is the one that I published a few weeks ago. But essentially what it does is it guides you through different bounding box formats. And so we have XY, XY, XYWH, CX, CYWH, where they come from, where to use them, and what different frameworks use which. So PyTorch, Hugging Face Transformers, YOLO type models, or Google Gemini. And the goal of this blog post is to introduce the different formats and how to draw them on an image. So say we're got this image of someone putting a piece of trash in the bin, and then we want to draw a bounding box around the bin because we're building an app like Trashify, which I'll show you in a second. But we have this box that— this one was drawn nicely, and then this is the same box but drawn in the wrong format. So there's nothing worse than having a good object detection model but then drawing boxes in the wrong format. So if you want to learn more about object detection, different bounding box formats, check out this blog post. That leads into the work in progress on learnhuggingface.com. Not affiliated with Hugging Face, just a tutorial website where I'm working on different projects or open source tutorials. And this one is creating an open source object detection model or fine-tuning an open source object detection model for your own use case. It goes along with the text classification project that is live. This one is— the code is all done, but I'm just adding all the extra materials that go along with it. And if we go to the demo, this is what we're going to do. We start with data, we find a model, and we train it, and we finish with a demo which is Trashify, which is a machine learning model with the goal of encouraging people to clean up their local area. So given an image like this, it will detect hand, trash, and bin. And if it gets those 3 things, you get a +1. So you could imagine this could run on an iPhone or a smartphone, detect these 3 items in the camera frame. And then if it detects a hand, trash, and bin, you get +1 point to a global leaderboard of people in different areas. And you can see who has a high score for picking up the most trash in their local area. So if we try it on a custom image, Let's see this one. Well, this is in the test set. You'll have to believe me here that this came from the test set. So if we predict on this, and there we go. So this is without NMS filtering, which is non-max suppression. And then if we have here, we have NMS suppression. So we learned this in the project is +1 because we have a hand, we have a trash, and we have bin. And then we have the ML model memory calculator. So what if you're like me and you'd like to run open source models? And I have an RTX 4090 in that computer over there, which has 24 gigs of VRAM. I want to see which models are capable of running on there. So 7B tends to be about the— or 7 billion parameters tends to be about the sweet spot for me. But there are larger models becoming more and more available these days. And of course, smaller models. So with the help of Claude, I made a simple app to help you calculate how many or how much memory you'll need for your machine learning model. So if we have 20 million parameters, we've got plenty of room there, but different levels of model there. So go and check that out. And then if you want to learn all about embedded machine learning, how different models get run on device, why Tesla cars might want to use models on device or do use models on device rather than the cloud, Go and check that out. An in-depth blog post about embedded machine learning. Anyway, that is my work for the past month. Work in progress, but we'll get updates on that soon. So let's jump into Daniel's open source AI resources of the month. And so the first one here, we have segment anything with text using Saatva. Now, I don't know if that's pronounced, but it's like S-A-2-V-A. So this combines SAM-2, which is Segment Anything from Meta, which is the goal of drawing boxes, or not boxes, sorry, like pixel by pixel segmenting different items in images and videos. That was what was the upgrade with SAM-2. But the team from ByteDance have released SAVA-2. We've got a demo that I've, like all good machine learning cooking shows, prepared earlier. So if we install, uh, upgrade a version of Transformers, flash attention, I am running the bigger model, so I do have an A100 running in here. We have 40GB of VRAM, and the model I'm using, the 8B version, so 8 billion parameters, requires 27GB to run. So I've already loaded that. And say we have this image here, and at Nutrify we like to do, uh, food intelligence. So I'd like to segment all the different food objects here. So if we keep going, this was my lunch the other day. I've already done it. The beautiful thing is you can prompt with text. So if I go the main subject, it doesn't give me that good results because if you look at that image, there really is there a main subject or not? I'll leave that up to you to decide. But then when we get more specific with food. So if we have this food list here, a portion of dried meat sticks there, fruit salad with watermelon and cantaloupe. And salad with red onion, bocconcini, rocket leaves, cucumber slices. We have that over there. So if we run that and then we plot the masks, how cool is that? So we get it segmented. And then of course, because we've got segmentations, we can turn those into boxes. This is the type of thing you'll learn in the blog post that I showed before. We now have bounding boxes around our food items just from text. So that's the power of the SA-2VA model. Which comes in 4 different sizes, 1B, 8B, 4B, 26B. And that's where you can use the ML model memory calculator to figure out if your hardware can run it. Now, a really cool thing that I found is because it has been trained with Segment Anything V2 plus an LLM, it has semantic understanding. And so let me get specific with this. If we didn't want the masks around the plate or the bowl here, we just specifically wanted the food items. Let's see if we can do that via prompting. What I'll do is I've got another, the same list here, this list, but I've replicated it. But now I've specified at the end, only, no plate, only, no bowl, only, no glass bowl, no fork. So if I change this to food list only, and we run the prediction, it's pretty quick because it's only using the SAM-2 decoder. And then if we have a look at our visualization, look at that. They are some great looking boxes. Well, sorry, uh, masks. So there's a portion of dried meat, a little bit pixelated, but that's okay. We can see that it's fairly pixel perfect. Now, not again, actually not pixel perfect, but for our use case, it certainly serves its purpose. And then if we want boxes, our boxes should be now much tighter around the food. There we go. So that is really cool. So a big shout out to the ByteDance team for publishing that. There is also a research paper that goes along with it. And of course, a GitHub somewhere, the link, yeah, the link is on the Hugging Face repo if you want to see that. And then if we go to the keynote, this is an example of segmenting the main subject. So if there is clearly a main subject, you can just get it masked out. and then we've just seen the example there of if you get specific of just submitting the target thing that you want. There's the hard-boiled eggs. And what I found really cool in the paper is the data annotation pipeline. So this is getting to be more and more of a theme, is creating large-scale synthetic datasets for your specific use case. And they start by creating an object-level expression, uh, leveraging a large VLM. So InternVL-2 76B. Then they create a scene-level expression, and then finally a video-level expression, and they merge that all into one. So check out the paper a bit more there if you want more information, but otherwise, if you want to try the models out, they are available on Hugging Face. Next up, we have Google's release of SigLIP2. So I've got another demo here that I've tried out. So if you put an input image, well, let's start with SigLIP. So SigLIP1 is probably my most used computer vision model outside of my own models that we built for Nujify, because it marries up language and image. And that's what SigLIP stands for, Sigmoid Language Image Pre-training. And so SigLIP-1 was already incredible. SigLIP-2 has improved SigLIP-1 across almost every benchmark. Well, across every benchmark. So if you put in this input image here and then a list of text descriptions, SigLIP-2 is designed to match those image with text. And the more specific the text or the better match the text, the higher score SigLIP will assign to that text input. So if we have a plate of food, a plate of breakfast food, a plate of eggs on toast with sides, and then a plate of food with eggs on toast, grilled sausages, smashed avocado and cabbage, red cabbage, wilted spinach, mushrooms and tomato chutney. Oh my goodness. So this is a breakfast that I had the other day at a company, a cafe nearby where I live called Good Company. Shout out. If you're on the Brisbane, north side of Brisbane, Epic Cafe. And what we can see here is both SigLIP-1 and SigLIP-2 did an incredible job of matching the most specific text. Now, the thing is, all of those text inputs could describe this image, but they are ranging from simple to more detailed. And here's the thing. If you want to get really specific, you'll get a higher score. But if you just stay nice and simple, you'll still get a score. But in comparison to the text that better suits it, it is a lower score. So that is the job of SigLib, zero-shot image classification. And you'll also notice that in many open-source VLMs, SigLib— not SigLib 1, SigLib 2 is brand new— SigLib 1 is the vision encoder for many VLMs. So vision encode, LLM encodes a text, we marry them up. I'm assuming SigLib 2 is going to be a drop-in replacement for all of those models, open-source models that are using SigLib. 1 as a vision encoder. So if you want to see the blog post, there's a blog post here on Hugging Face, goes through the architecture. If you want to see the paper, that's also available here. That's on Hugging Face, but the paper is also on Archive. And if you want a demo, you can try out the demo here on Google's Hugging Face, SigLIP-1 versus SigLIP-2. As you'll see, the more specific the text to the image, the higher the score. So all 3 of these could describe this image, but the most specific one gets the most or the highest score, which is generally what you want. And then we have SigLIT2. All the model weights are available in OpenClip as well. So whole bunch of different variants there. So big shout out to Google there. Thank you for open sourcing SigLIT2. Up next, we have OVIS-2, and OVIS stands for Open Vision. So this is an open source VLM. So if we jump into here, there are 6 different variants ranging from 1 billion parameters to 34 billion parameters. There is a GitHub available with all of the code there. And this is the architecture I was just talking about. So what OVIS-2 does is it combines vision and text, and then they pass those embeddings into an LLM to produce an output. So if you have image input, we embed the image and you have text input, you embed the text. Combine that in the embedding space of an LLM, and then produce text as an output. So if we want to go to the demo, OVOS-2 16B. So if we upload an image, there we go, we can take this from Trashify. Please write a caption for this image. And then we can let that run. It's also capable of not only captioning images, but thinking about different step-by-step processes. So there's a lot more advanced use cases here, but there we go. Recycling in action. A handful of paper waste ready for disposal. So there we go. Nice and simple caption, but all of these models are open source. I believe they use Apple's AIM V2 backbone. Yeah, there we go. So previously in OVIS 1.6, it was SigLIP-1. So if we go to Obus 1.6 and we jump into, here we go, used SigLIP 400ml, so SigLIP version 1 with a paired LLM. So this new version uses Apple's recently released AIM V2. So if we look up that, we go to Apple here, is a vision encoder which scores incredibly well on a lot of different benchmarks. I am excited to see if this they substituted this one for SigLIP2 and what the performance looked like there. And if we check out the performance, OVIS-2 series performs the best on the Pareto curve of all of the OpenCompass benchmarks. Even the 34 billion parameter model performs nearly on par with a 72 or 78 parameter model. So really cool release there from AIDC AI. So thank you very much for that. Onto the next. So speaking of open source VLMs, we're continuing the trend here with Microsoft launching Phi-4 MultiModal and Phi-4 Mini. So mentioned in last month's Machine Learning Monthly, January 2025, we had Phi-4, which is a 14 billion parameter LLM. So text only, whereas Phi-4 MultiModal and Phi-4 Mini, Phi-4 Mini is text only again, but a smaller size. Phi-4 Multimodal can do everything. So speech, text, audio. If we go here, Mini Instruct, we go to Microsoft's page, Phi-4 Collection, Multimodal Instruct. Beautiful. So now we have language, vision, and speech. And what really impressed me is that not only is it capable of 23 different languages, it's that if you, uh, It's got a bunch of different use cases, but if we go to the speech recognition, and this is where I'm starting to get really impressed by a lot of models being so good at speech recognition, is Phi-4 Multimodal Instruct performs the best or has the lowest error rate. So lower is better in this graph across many different, 3 different benchmarks, except for this one here where Gemini 1.5 Pro outperforms it slightly. But in terms of word error rate. So versus even models like Whisper v3 from OpenAI, which is a model specifically trained to turn speech into text. So that's really cool. But then if we go down here, I've taken the example, I'll let you read through all the benchmarks if you want. It's a 5.6 parameter or 5.6 billion parameter model, but I've taken this audio processing code. And again, like all good machine learning cooking shows, have got an example already. So if we have transcribed the audio to text and then translate the audio to French, there's an example audio file here that they've used. Use SEP tags as a separator between the original transcript and the translation. So I recorded my own audio file. I'm going to play that over the top of this text here and we'll see if that works. You can see if how well the transcription is. Hello, Daniel Bourke here. I am testing the new Phi-4 multimodal model from Microsoft, and it is capable of taking in audio inputs and turning them into text. It's capable of taking image inputs and turning or returning text from the image input, and it is also capable of just straight text-to-text. It's also quite small, so it can run on many devices locally, and it competes with models that are much larger than it and are also only available through a closed source API. So a really cool open source release from Microsoft here. So basically instead of, uh, the only thing it really got wrong was how to spell my last name, but that's, uh, this is probably a more common spelling of Bourke than B-O-U-R-K-E. And then I don't speak French, but we have the transcription here straight afterwards. And that's just from a prompt. So audio file in and then transcription and translation out. That is awesome. So if you speak French, please tell me how this translation is compared to the text input here. And then not only do we have the multimodal model, which is 5.6 billion parameters, if we go back, we also have Phi-4 MiniInstruct, which is a text-only LLM, but that is 3.84 billion parameters and it performs well above its weight class. So even at the smaller size, so these models are 2x bigger on the right-hand side here. We have a 63.5% performance or average score, sorry, across all these benchmarks. And then the only model, only 2 models to outperform it or 3 here are Qwen, so 7B, so that's twice the size. GPT-4o Mini outperforms it as well, but that's a closed-source model. And then Gemma 2, which is 9 billion parameters, so This is 3.8, so 2.5 times bigger is only slightly higher than it again. So incredible release from Microsoft there. If you're interested in small LLMs or small multimodal LLMs, including speech now, which is awesome, check out the Phi-4 versions. You have a blog post, technical report, and a cookbook on the Microsoft Hugging Face page, and they are all MIT licensed. Now with so many OpenVLMs coming out, there How do you keep track of them? Well, that's where the OpenVLM leaderboard comes in, which aggregates 31 different benchmarks. I'll let you go through these in your own time. But if you're searching for an open source VLM model that you can use in your own workflow, for example, I'm really impressed by the OVOS-2 models. So I'm looking to use them in Neutrophize workflows. And if you want to, you can sort by date here of the last most recently evaluated models. So there we go, Gemini 2.0 Pro, which was recently released. We have an eval date of a few days ago, and you can sort by average score, you can sort by model size. So if you're like me, you have one GPU and you want to run it locally, you might look for a smaller model. But what really impresses me here is that the open models are— there is now many open models that are outperforming even the closed source models. So Claude 3.7 CNet, which was recently released as well, doesn't score as high across all of these benchmarks averaged as OVIS-2 34B. Even OVIS-2 16B is performing better, or even 8B is performing better than Claude 3.7 CNet. So that is really cool to see, is that open source models are getting better and better. So go and check that out. You can even— we see which one's at the moment, Step 1.0. But generally I've found the ones with the weights listed are open source. So that's the OpenVLM leaderboard from OpenCompass. Up next, we have SmallVLM v2. So now SmallVLM, which was also mentioned in last month's Machine Learning Monthly— I love how fast the open source space is improving— now has video understanding built into the model. So it comes in 3 different sizes, 2.2B, 500 mil and 256 mil. And it's also MLX ready. So if we go MLX GitHub, MLX is Apple's machine learning framework or one of them for running on Apple Silicon. So this is really, it's really helpful for running if you want to run it locally on your Mac or other Apple device. So if we go down, the model is very performant for its size. And there's also a 500 mil video instruct version. And the cool thing is that the 500 mil Video Instruct version is now running open on an iPhone. Sorry, running, not open, on an iPhone. So if we have a look at what's going on, how many plants do you see? And it's going to answer the question. This is all running on device. So that is really cool to see. That's what I'm really excited about is models like this running on device. So it's just with you all the time. And we'll see, we'll see a model like that shortly again as well. So here, left or right cup for espresso. There we go. Beautiful. Now, of course, that's been sped up, but this is, of course, a V1 version. So it's going to improve over time. If you want to use that or try it out, there's a form there to fill out. And then you can also use the model for video highlight generator. So it's got a plugin for VLC media. And then you can pass it in. I think it's, yeah, long-form video up to 1+ hour and automatically extracts the most significant moments. So that, that at the moment, that's really important to me because my father is in hospital at the moment and he's getting taken care of by some wonderful nurses. However, their, their main job is to give him care, right? But the other day, my mum and I went up and we found out that he had a rough night. But we didn't get much information because they of course were prioritizing providing the care rather than writing it down. So it would have been really nice for us to have that information. And so I'm thinking, uh, in that case, of course you would— it's a healthcare space, you'd need privacy concerns. But this small VLM2 could run locally on the hospital, record footage of, um, throughout the night, surveillance footage, and, uh, write a report every day of when something happened. Um, so for example, at 3 AM, woke up and had to be delivered care by the nurses. That's— so rather than the nurses spending their time, their precious time, writing and doing paperwork, they can provide the care. So this is just, I think, over the next couple of years, video inference is going to be incredibly useful. So you can see all the demo code there. There's also an opportunity to fine-tune the model. So the 500M parameter variant, uh, you can probably perform full fine-tuning. So you can see the notebook there, but otherwise, um, there's a blog post, there's the collection for SmallVLM, uh, there's a demo here which is describe this image because it's got multimodal, and much, much more. Of course, you can sign up to try the app out later on too. And up next, we have, continuing the trend of open source VLMs, we have IBM release Granat 3.2 models under Apache 2.0. And so we're now entering the era of having models that are designed for purpose-built business use cases. So that's what Granat is designed for, uh, business use cases, engineered from the ground up to ensure trust and scalability. So the models are available as Apache 2.0 on Hugging Face. and what I love about the IBM Granite series is that they've built out documentation quite substantially for how to use the models. So if you want to perform multimodal RAG, you can process documents, and they've got tutorials on how to do that. So this is really important, I think, for any future— if anyone wants to open source their model, documentation on how to use it is probably just as important as the model itself. So if we— I shouldn't have closed those. If we look at 8B Instruct, which is a pure text-only model in many different languages, we have thinking capabilities, summarization, text classification, extraction, Q&A, RAG, and a few more here. But then we also have the vision. Oh wait, that's the Guardian. So if we look at the vision model, there we go. So it's a compact and efficient vision language model, which has around 3 billion parameters. And it's specifically designed for business use cases. So extracting information from tables, charts, infographics, plots, diagrams, and more. So if we have a look at the benchmarks where it's a business use case, so Document VQA, Document Vision Question Answer, Chart QA, so questions from charts, Text VQA, questions from text. Granoid Vision performs the best across almost all of these benchmarks. It does slightly fall on the M-M-M-U, which is general knowledge. But if you're mostly using a model to extract knowledge from your own data, it doesn't necessarily need general knowledge. So of course, take all benchmarks with a grain of salt. Always test it on your own data. But there is a paper associated with that. I found that a good read of how they used synthetic. If we look here, syn— have I got this here? It's not mentioned here, but it is mentioned up here. Synthetic data. Yeah, there we go. So creating synthetic data-specific capabilities including document understanding tasks. So there is also a GitHub beautiful open source release from IBM for the Granite model. So if you're using, uh, LLMs or want to use LLMs or VLMs for a business use case, be going sure, be sure to go and check out the Granet series. Up next, we have an open source release from the ARC Institute, which is Evo2, which now instead of predicting language tokens, the next token in a series like the cat will jump over the fence, Evo2 is trained on 100,000 species worth of DNA data. So LLMs are starting to introduce or starting to enter the biological field. And so I believe it is, yeah, 9.3 trillion nucleotides. So base pairs, which are A, C, G, and T, like in DNA. And so what this makes this model capable of is that it can accurately identify disease-causing mutations in human genes. So that is a really cool use case. And the one that they highlighted here was, for example, in tests with variants of the breast cancer uh, associated gene BRCA1. Evo2 achieved over 90% accuracy in predicting which mutations are benign versus potentially, potentially pathogenic. So my blessed grandmother had surgery on last Friday for breast cancer, and so having this potential use case for something in the future to screen someone, uh, and go, hey, this, um, this sequence that we found may be benign, or it may be potentially pathogenic. We could probably investigate that further. Is really close to home for me at the moment. So, uh, incredible release here from the Ark Institute. And then there's two papers associated with it. We have the biology-focused one, which is Genome Modeling and Design Across All Domains of Life with Evo2. And then we have the machine learning focus one, which is— they talk about how they designed the algorithm, which was Striped Hyena 2, and they used that for, uh, next token prediction rather than the dense transformer. Because what they wanted to do with, I believe, does it have it here? Million a sequence. Yeah, there we go. So the model can process genetic sequences of up to 1 million tokens at once. So this was really important to be able to go over if we have a gene that ranges from I'm not sure the exact range in a human genome, but I believe it's probably from like 100 nucleotides to maybe even millions long. So you want it to be able to process a long sequence at a time. And so what they did is they created StripeTayena2, which is much more efficient as the training sequence increases. So the Dense Transformer scales here quadratically with sequence length. And so if you want to train it in less time, you you want to make a more efficient model. So that's how they created this. That's a machine learning focused paper. Finally, there is also a GitHub. So if you're in the biological field and you want possibly the most state-of-the-art performing model for genome modeling at the moment, go and check out Evo2. And now we have Hugging Face release Open Deep Research. So over the past couple of weeks, Google and OpenAI, and of course Perplexity, have their own version of deep research, which is essentially, uh, like you tell an agent or an LLM to go and find information on the internet, searching all through those links, taking steps to sort through that information, and then returning back a report. And it may take a bit of time to do that, but the beautiful thing is, is that it does it automatically. That's the whole goal anyway. So if you go into the blog post, Open Source Deep Research, performs very well for an open source model, open source version, and they replicated the workflow in about 24 hours. So that is incredible work for such a short time frame. But if we go down here, the way I imagine it is that there's an LLM in the pilot seat, and what it does is a code agent. So the LLM, you give it an instruction, say as I've given one here, What are all the ingredients, nutrition information in Domino's Pizza in Australia? And I know, one second, let me move my head so you can see that. This is the instruction I gave it and it's going to generate code steps. So if we look through here, this is running live at the moment. I just put this in. It takes a few minutes per step or thereabouts. And this is what happens is it will let the agents express the action it wants to take in code. So what this means is that you have an LLM and you go, hey, do this thing for me. And then it's going to go, oh, I need to use this code step to take the next step. And then I use this code function to do the next step, such as search the web, click on pages, read information. And so what this paper showed, which is what they used to implement their system, was that code actions require 30% fewer steps than JSON. So really cool read-through here of how agents are definitely evolving. They aren't perfect, but they are getting better and better. So read the blog post announcement there. This is all built on top of Hugging Face's open source library, SmallAgents, which is designed to help you build agents, which combines an LLM with tools. And then this is the Deep Research GitHub. So you can see the workflow that they went through to reproduce these steps. And then of course, here's the demo, which I've gone, what are all the ingredients nutrition information in Domino's Pizza in Australia? So it's got a web search here. It's searched for Domino's Australia menu nutrition. Then it's visited a page here. Domino's, is this an actual webpage? Let's find out. There we go. That's exactly what we wanted. And what's it done next? So it's visit page, gone there, page down, thought, scroll down, page down, visiting URL, Ctrl+F on a page. This is really cool. So I'm going to let this run for a bit. We may check back on it in a few minutes. We may not and see where it ends up. But that is a really cool open source version of an agent, and this field is just evolving very quickly, so it's awesome to see such cool projects being replicated within a day. Up next, speaking of the Small Agents library, there is also a course to go along with it. So if you want to learn more about open source agents or just agents in general, check out the Hugging Face, uh, open source course for agents. It's got 3 sections there so far. So introduction to agents, frameworks to use, and the Small Agents Framework. So the Small Agents Framework is what the Open Deep Research— oh no, it's got an error. Okay, so not perfect yet, but it looks like we can start to see the steps that it was taking. And that was similar to— oh wait, there we go. It worked. A few popular pizzas, margherita pizza. Here's the nutrition information. Meat lovers pizza, pepperoni pizza. So, that's pretty cool. Pretty cool to go from a text prompt to information backed with their official nutrition page. Does this— it works. Margherita. Is that correct? So, classic crust. Does it say anything here? One piece, 833 calories. What do we get? Okay. Maybe it's not perfect. 116 calories, 8 servings per pizza. Okay, so maybe that's times 8. Anyway, we're not going to fact check that for now, but that's just to show the workflow and that the field is rapidly evolving. If you want to learn about AI agents, check out the open source Hugging Face AI agents course. Now up next, we have an open source 1.6 billion parameter transformer model for text to speech. I had to think about that then. I nearly said speech to text. But they released two versions of it, which is a hybrid and a transformer-based. The hybrid is more efficient because what they're trying to do is make it faster than real time, which means if you have one second of speech, it can be generated faster than one second. So yeah, the hybrid version, I'll let you read the blog post for more information on the specific architecture. But the important thing to take away is that it's faster and more efficient or slightly faster than a transformer. There's an architecture, and that's the goal here, is to, um, the real-time factor. So if you're using an RTX 4090 locally, I'm lucky enough to have one of those over there, it can generate speech faster than real time. So that's beautiful, that's where we want. Now it is hard to evaluate text-to-speech models, so they've got some examples here of audio generated by the model. So Let's just have a listen.

I don't really care what you call me. I've been a silent spectator watching species evolve, empires rise and fall. But always remember, I am mighty and enduring. Respect me and I'll nurture you. Ignore me and you shall face the consequences.

So that was pretty good. But if we try another one.

I don't really care what you call me. I've been a silent spectator watching species evolve, empires rise and fall. Fall, but always remember, I am mighty and enduring.

So to me, the Zonos one sounded a bit more natural versus the ElevenLabs, which sounded a bit more robotic. I won't go through all of them. You can, of course, go and try them out for yourself, but the models are available on Hugging Face under the Apache 2.0 license. And then if you want the demo, we can try some information here. So, okay, so I've got some text here. Let's see how this sounds. And now this is with the hybrid model. I've turned off all the emotions. I haven't really played around with the settings. This is just the default version, I guess.

Yo, my name is Daniel Bourke, and I am a machine learning engineer at Neutrify, and I live in Brisbane, Australia.

Okay, I think Kokoro 82ml from last episode was a little bit better than that. I'm not sure where that voice came from, but yeah, if you want to open source text-to-speech model, go and check that out. Up next, we have AI2 release Olmo for iOS, free to use on-device LLM. And so Olmo stands for Open Language Mixture of Experts. So I'll let you read through the blog post if you want to check it out. The takeaway is that it's now very possible to run large language models. Well, this is a relatively small one compared to other big models, but it is an LLM. Running on your device without an internet connection. So no API calls. It's just running locally on iPhone and iPad. So if we go down here, they reveal how they made it. And it's also an open source model, which is available on Hugging Face under Apache 2.0. So it is open source. And then the app is available on the App Store as a free download. And even more impressive is that the app code itself is available on GitHub, So if you want to build an app that runs locally on device with an LLM, an open source one, you can see exactly how that's built. So really cool. And I tried it on my own phone, which is an iPhone 15 Pro. And as you can see, airplane mode is turned on. So I asked for a simple hummus recipe. I did speed this up by 3x to just for brevity, but it was pretty quick, like faster than what I could read. And then I said I was allergic to tahini, which I'm not. I was just trying to test it out, and it gave a good fix. So this is really cool to see of, um, like where the future is going and more and more powerful models running on device. This is the trend that I'm loving throughout this year so far, and I think models like this should potentially replace the assistant that's on most iPhones beginning with S. I'm not going to say that out loud. Up next, we have Local AI and their GitHub repository of many different open source tools. So, uh, Local AI is an organization and they focus on creating efficient open source AI software that runs on your device. So this is a background removal one, um, for OBS. So I'm using OBS to record this right now. If I didn't have a green screen, you can use, um, OBS to replace your background. So it would just be a cut out of me, but I kind of like my nerdy computer background. So check that out if you're using OBS and you want to see how, or just all the code really, of how an on-device AI system runs. And then we also have Local Vocal, which is speech AI assistant in OBS. So you can run OpenAI's Whisper in real time to create speech captions appearing on your OBS recording. So that is really cool as well. And then that's not all. There is a lot more here, but awesome collection of different software that you can run AI on your own device. Next up, we have OpenRLHF. So RLHF stands for Reinforcement Learning from Human Feedback. It's how you align large language models with human preferred outputs and starting to be the same in computer vision as well. So if we go through here, there's a whole bunch of different methods or reinforcement learning algorithms that are implemented in OpenRLHF, such as REINFORCE++ or PPO, which is Proximal Policy Optimization, and then GP-RO or GRPO, sorry, which is what DeepSeek R1 used to improve their model. And then there is also a great blog post that I found, and that is Unraveling RLHF and Its Variants: Progress and Practical Engineering Insights. So This goes through a bunch of different RL methods and non-RL methods such as direct policy optimization and explains how they work and the different advantages, pros and cons of each. So really cool library to see if you want to use some reinforcement learning on your own models. Go and check it out. The documentation is also very comprehensive as well. So quick start, a whole bunch of different yeah, it's built on top of Ray. So it's very performance efficient. So go and check that out. OpenRLHF on GitHub. Up next, we have a series of case studies. So these are examples where a company has taken something and they've built it into the real world and then they've written about how they did it. So first up, we have FineGrain. So they share how they create a promptable high-resolution image segmentation. So we saw a promptable segmentation model at the start of the video. and that is SAR2VAR. But this is fine-grained, have deployed this in an API that you can use. So we have the segment, segment anything model. They actually didn't use that one. They used a different architecture. So how we built our own box promptable segmenter. So they use Grounding Dino as part of the pipeline, which is Grounding Dino detects boxes based on text. And yeah, they use the MV4Net, MV4Net. MVA, sorry, MVA-Net architecture. So if we go to this, there's the archive paper for that. And that's a really cool example of how research gets translated into production. And so someone has done a comparison here. We're trying to segment the chair. Photoshop still has all the background. Canva has a few missing artifacts down here. And the fine-grained object cutter does a really nice job of capturing all the details. Probably not 100% pixel perfect, but it's doing this with machine learning, so that is incredible. And their goal was to, uh, get rid of this or improve this, uh, cropping with things on the edge. So they discuss how they did it. They create 5 different channels of input, and after the release, you can see the improved model. So FineGrain, they create pixel perfect image editing API, and then you can even check or try out the demo. And this is a really cool use case or cool example of how building an open source demo on Hugging Face has turned into a company. So they got a lot of feedback from people using the space and tried to improve the product here. So there we go. What should we erase? Erase the soap. Does a pretty good job there. And what else? Erase the potted plant. Wow, that is cool. It even does it in the mirror. Okay, that's really cool. So go and check out, uh, Object Fine Grain Object Eraser. And if you want to learn more on how these different types of models get built, go and check out their blog post. Up next, we have the makers of the Zed code editor, open source Zeta. So this is a great case study of Oh, where's their blog? Blog. And then we have Zed. So Zed now predicts your next edit with Zeta. So this is a really cool example of taking an open source model, which is Quen 2.5 Coder 7B, which is available on Hugging Face, and they fine-tuned it using their own dataset. So in one way, they use supervised fine-tuning with UnSloth and LoRA, or low-rank optimization or low, low rank adaptation. And but they found that direct preference optimization with just 150 carefully selected examples were significantly enough to improve Zeta's behavior. And then there's another. So that was just making the model better. But there's two parts to making a product, right? You have to make the model good to begin with, but then you have to serve that model. So Z's priority is fast inference or fast. The app should just feel fast. So what they did was they set aggressive performance targets and the predictions are delivered in under 200 milliseconds for the median case and under 500 milliseconds for the 90th percentile. So really cool example of how they used speculative decoding to where the model writes, like you have a smaller model writing a draft and then you go over the top. And they also served it with Cloudflare workers to make sure that when someone pings a model, so they send information from their editor to the cloud, it selects the closest data center. Because when you're moving information on the internet, it is the speed of light, but that still takes time to move around. So you need to ideally send information to the closest compute cloud to you geographically. And then of course, the Zeta model is also available, the one that they fine-tuned from Quen 2.5 Coder 7B is available for testing on Hugging Face. So really cool use case to see an open source model getting deployed into a product thanks to fine tuning and serving. So go and check that out from the Z team. Up next, we have LM Studio introducing speculative decoding and then a couple of bonus extra resources on the topic of speculative decoding from Google. So LM Studio is an app for running LLMs on your own device. So if we go here, you can run it, LLMs on your Mac. So type to them natively. There's a whole bunch of different open source variants. There's also an SDK available now, but this is a great case study on how they made their models faster. So speculative decoding is a technique that can speed up token generation by 1.5 to 3x in some cases. So that is really cool. So it works by using a draft model first, quickly predicting the next few tokens as a draft generation. And then right afterwards, because you've got the draft, you can go over it with a bigger model and they either get confirmed or rejected by the main model. So the main model being a bigger model. So kind of like in, if you were writing an essay, you might write the first draft and that takes you, you type it out really quickly because you're just going over. but then you go back through it again and you edit it and you make it better. So performance stats, when you have a main model that is 32 bil, and you have a draft model that is much smaller, so 0.5 billion parameters, it goes from 7.3 tokens per second to 17 tokens per second, or close to 18 tokens per second. So a speedup of 2.43. And then we have some other stats here. But if you want to read about how they did it, please go through the blog post. Really cool example of how research goes. So this paper is what introduced speculative decoding. So 2022 or 2023, and they show how they improve the inference of a transformer model here. So that goes to show like it takes a little while for, it often takes a little while for research to go into a product that you can use. And then if you want to learn more about speculative decoding, there is a specific use case here on the Google blog post, which is called Speculative RAG. So Retrieval Augmented Generation speeds up RAG. And then there's a great breakdown of different speculative decoding techniques and how they're being used. See, much faster generation there. It's almost probably 2.5 times faster. Of where, of how speculative decoding works and where it's being used. So go and check those out on the Google blog, and big shout out to LM Studio for shipping an amazing new feature. And now we're going to do a quickfire round on these releases because they are generally from bigger companies and they have enough broadcast reach themselves. So I wanted to focus on all of the open source stuff a bit more here from smaller players, but Google have released Google Gemini 2.0, uh, Flashlight and Pro and Flash. So 3 variants. And from my personal experience, they all perform quite well, uh, especially, um, at their price point. I think they're about 1,000x or maybe 10,000x. Gemini Flash is 10,000x cheaper than, that may be too high. I'll put it on screen versus GPT-4.5. Um, a great blog post here from Phil Schmid on how to get structured outputs from PDFs with Gemini 2.0. And then there's another Gemini— oh, sorry, Imagen 3 release in the Gemini API. So Imagen 3 generates the best images I've ever seen. So I've got this workflow here, example. So if you were to want to create synthetic data for training your own models, you could have an input image. This is a real image. We saw it before up here when we segmented the eggs. but then I use Gemini 2.0 to caption this image. So with a detailed description, and we have a generated image. So that's pretty good for a generated image. And then we could feed that. Now we have two samples for the price of one. And we could also write in the prompt to reorder the order of some of these or make it on a different background or make it on a different plate. And so all of a sudden we can improve our data augmentation pipeline with new samples, entirely new samples. So rather than just augmenting this, we just create a new sample and tweak the text. So if we go back to where we're at, so incredible fleet of releases from Google there. We have Anthropic release Claude 3.7 as well as Claude Code. So Claude 3.7 is their latest flagship model, Sonnet, and it performs from my testing. Claude is, is incredible. It's probably my favorite writing LLM. and also very good at code. And then Claude Code is an agentic coding tool. So it lives in your terminal and it can take steps. But there is a video, um, you can see the example here in the video example. xAI introduced Groq-3, which is, we're starting to get to that point where LLMs are all sort of converging on being really good. So there's probably 5 or 6 LLMs that are GPT-4o and above now. And for the most impressive thing to me for Groq-3 is how quickly they got to state-of-the-art results. So I believe the xAI team was only formed about 18 months ago from the time of recording this. And so they've gone from starting a new team to having the best or most performant LLM in the world according to the benchmarks in about 18 months. So that is really cool. OpenAI released GPT-4.5. The most impressive release there for me was about a 2x reduction in hallucination on the SimpleQA benchmark. So instead of GPT-4.0, which was 61.8% hallucination, GPT-4.5 apparently is 36%, 37%. Yeah. So the downside is of this model is that it's quite expensive. Yeah, so if we look at the pricing of GPT-4.5, it's $75 per 1 million input tokens. But if we go to Gemini pricing, and if we look into this, for example, yeah, Gemini 2.0 Flashlight, probably not as good a quality model, or Gemini 2.0 Flash, which is a great quality model from my experience, is $0.10. So if we do $75, $75 divided by $0.1, 750 times more expensive on GPT-4.5 on input tokens than Gemini 2.0 Flash. So you've got to think about that. Is it worth the 750x increase potentially for your use case? Not for me personally. And then to finish it off, we have AI2 release ScholarQA. So this is a really cool example of a RAG app across a whole bunch of different research. So if we go to the demo, we can go, how does magnesium help sleep? What this is going to do is it's going to search over, I believe it is, we're just going to leave that run in the background, 8 million academic papers from various fields of study. And they've created this as an index using Vespa. And they've used a combination of BM25, which is an indexing algorithm, as well as dense embeddings. And then if we go to here, it's going to search through or create a workflow of finding different papers which relate to the query that I put in, which was, how does magnesium help sleep? And then it's going to aggregate all of those sources and give me an answer with link backs to the the research paper. And now, as we can see, a few minutes later, it's generated an answer with references. So, biological mechanisms of magnesium and sleep regulation, effects of magnesium on sleep parameters, uh, clinical evidence for magnesium sleep benefits. Of course, all of these are cited, and it is used, I believe, Claude 0.5 or 3.5 CNET to basically take all of the information from the research papers and create it into this beautiful display. And then if we wanted to jump back in and check all the sources, we can check the papers here. So really cool for doing some scientific research here and just aggregating it. And then of course, if there is a chance of hallucination, we can go back into the original paper and see if that information is correct. So a huge month for February 2025. If you'd like to see anything in a specific, in a future episode, leave a comment below. But otherwise, I'll see you next month.