This New AI Vision Model Beats Everything (Molmo Ai)

So in today's video, we're diving into momo, a groundbreaking family of state of the art multimodal AI models that are actually setting new standards. MOMO doesn't just interpret images and text like other AI models. It actually takes things a step further by learning how to point at what it perceives, unlocking a whole new level of interaction with both the physical and virtual world. Whether it's outperforming models 10 times its size, or closing the gap between open and proprietary systems, Malmo is paving the way for next gen AI applications that just don't understand their environment, but actively interact with it. Take a look at this quick demo and then I'm going to dive into some of the rather shocking benchmarks.

Is this option vegan?

No, this option is not vegan.

Count the number of people.

Counting the number of people shows a total of 21.

Convert this table to JSON.

Here's the table converted to JSON format.

I want to sell my bike on Craigslist. Write me a description for it.

Schwinn bike for sale. Blue with white accents. $300 or best offer.

Do you want to buy it?

It's Monday at 1pm Can I park here and for how long?

Yes, you can park here on Monday at 1pm you have six hours left to park.

I really like electronic music. What show should I go to?

If you're into electron music, you should definitely check out the show featuring Taeko and Zed at the showbox.

Soto, can you write me a knock knock joke?

Knock knock. Who's there? Rocket. Rocket who? Rocket to the moon and back, baby.

Can you point to the model that takes the least time to train?

Nice.

That is really good. Which of these models uses reinforcement learning?

The image shows a table of different models.

Polymers.

Hey Momo, order me a pumpkin spice latte from Starbucks.

I'll place the order through Uber eats your pumpkin spice latte is on its way box.

So we could see just then and.

There that we actually do have a variety of different things that this open source model is able to do. And I got to be honest with you guys, this is remarkably surprising. So one of the things that we did see there was the incredible vision capabilities. We also managed to see the fact that this model can point at certain things within an image. You can see right here that it was able to point at certain things within the table and then of course convert that to JSON, which was rather incredible. So all of this is really, really effective. And one of the things that they also somewhat teased in this, they don't speak about too much on their blog, but we can see here that they spoke about how wedge web agents powered by MOMO can now browse the Internet for you. In here you can see someone ordering something from UberEat. So I'm wondering how effective this is going to be. And as I do know that web agents are some of the hardest thing to do with the complexity of agents running into multi step reasoning.

So the most surprising thing about MOMO is the fact that it's got incredible vision capabilities that are on par with state of the art models that are ten times its size. You can literally see here that the human evaluators actually rated MOMO 72 billion parameters higher than every other closed source model and basically put it on par with GPT4O. You can see our human ELO preference evaluations used 15,000 image and text prompt pairs and they queried each visual language model for responses and presented the resulting image to text response triplets for all VLM pairs to a set of 870 human annotators who gave the rankings. So you can see right here that this is incredible because it manages to surpass Gemini 1.5 Pro, Claude 3.5, Sonnet 1.5 Flash, and recently, even though Llama 3.2 was released, it also does crush Llama 3.2 in terms of these visual analysis benchmarks. So it's absolutely incredible in terms of its size, in terms of its speed, and of course its effectiveness. So this is not a model that you want to count out at all because it is really, really effective. Now you might be wondering, okay, how on earth did they do this? Well, they actually have taken some tips probably from how they did Phi three point, which is high quality data is all you need. We can see here that they state that data quality wins over quantity. So essentially with this model that they were building, usually people just train these robots on millions and billions of different pictures and descriptions that they find over the Internet. But the problem is that these descriptions are usually messy and confusing, which makes it hard for the AI to actually understand the right things and of course to learn the right things. And because of this, sometimes robots can hallucinate, and which is what is the biggest issue in AI right now. But instead, these guys actually used really, really good data, even if that data was much less, which is basically they used a few good pictures instead of a million blurry ones. So they use about a million image description pairs, which is a lot less than others. But because their data is actually more cleaner and more details Their robot actually gets a lot better, better at analyzing these images. Now, they also have Pixmo, which is the specialized way that they gather their data, which is in two steps. So, well, two types of data. So first they got the dense captioning, which means that they get people to describe pictures in lots of details. So not just a dog, but a brown dog sitting under a tree with a red bull next to it. And then of course, part two, which is of course fine tuning, which is helping the robot do specific things like answer questions about things in images, like reading documents or even point at things in an image. Instead of asking people to WR what they see, which can be slow or incompete, they ask people to speak their descriptions out loud for a minute. And it turns out that when people talk, they explain more about the picture and they describe all the little details and where things are. Like the dog is sitting left of the tree. And they collected all of these detailed speech descriptions for thousands of pictures, making sure that the robot learns from people and not other robots. And this makes the. I keep saying robot, but I'm talking about the neural network slash the AI here. And it just makes it a lot more effective. And that's why this model just manages to crush the others. Now, one example that I did see on Twitter was the fact that this model, you know, someone tweeted out that there is only one AI model that can tell the time from a clock on a face. And it's the most overlooked model out there. It's not GPT, Claude, Gemini or Loma. It's Momo by AI2. And of course that is true. GPT4 gets it wrong. Gemini 1.5 Pro, the most recent model, gets it wrong. Claude gets it wrong too, but Momo manages to get it. Now, what's crazy about this is that they also have some really small models. So they have their most efficient model, which is Momo Mitcher of Experts1B, which is based on their fully open source mod mixture of experts LLMs, which nearly match the performance of GPT4 vision on both academic benchmarks and human evaluation. And of course, you can see they refer to their best in class Malmo. Model MOMO 72B achieves the highest academic benchmark score and ranks second on human evaluation, just slightly behind its GPT4O. So right now I'm going to show you guys some more demos. One of the demos that they have here is really cool, which is where they've integrated this into a Vision Pro, showing you guys how advanced AIs with vision capabilities actually manage to transform our understanding of our environment.

Hey, I'm going to talk about how we could use Momo, our multimodal AI model with an Apple Vision Pro to enable us to better understand and interact in the world. This is a model that can answer questions based on an image and it can also augment and explain its reasoning through pointing. So let's get into it. Okay, let's start off with something basic. Hey Momo, what is this machine used for?

This is a sophisticated coffee station featuring a large stainless steel machine with a digital display surrounded by coffee cups, creamers and sugar packets.

Nice. That was pretty cool. Now let's have it augment our vision by predicting points as visual tags on the image. Hey Momo, can you give me an overview of these options?

The image shows a coffee machine with various drink options. Coffee is available in different strengths including a coffee strong x2 option for those who prefer milk based drinks. There are specialty choices like cappuccino and latte. Flavored specialty drinks are also offered such as Cafe Mocha and Vanilla Latte. Choc van options are available for chocolate and vanilla lovers. Lastly, there are extras like hot water and chocolate shot for customization.

Nice. That worked pretty well too. Now remind me again, what is espresso?

Espresso is a concentrated, strongly flavored coffee made by forcing hot water through finely ground coffee beans under high pressure.

Point to espresso.

Pointing to espresso.

Point to maximum strength.

Pointing to maximum strength.

Point to double shot.

Pointing to double shot.

Why won't it let me start the machine?

The machine is likely not allowing you to start because the cup hasn't been placed on the designated cup holder or dispensing area yet.

Okay, that was great. Now let's get a cup and fill it up. And we can interact with Momo as we walk around in the world here. We could quickly summarize information that were presented. So I'm going to pick up this sign and ask hey Momo, can you quickly summarize this sign for me?

The weekly lunch menu for September 23 to 27 is displayed featuring five days of diverse cuisine options including Bayrek and Baguette Cafe, Turk, Cedars, Indian, Bloom Fusion and Cameron Catering.

Finally, let's try one more example here. I'm going to go into a conference room and I'm going to ask how many chairs are in the scene and so MoMA will be able to give me an answer much quicker than I would if I were just counting. And so it's really augmenting my ability to See and interact in the scene. Hey Momo, how many chairs are in this scene?

Counting the chairs shows a total of 17.

Then we have another example here of where they're using Momo and robotics in order to aid robotics with their vision. Robots are pretty good, but sometimes the vision models aren't that good. And I think that this is going to help out quite a lot if robots know where exactly to place them. Things where exactly things are. And a good onboard vision model is going to be essential for ensuring progress in robotics. So take a look here.

We can see the what the robot is observing and we are going to ask a couple of questions that will ask the normal model and we'll see what it says. So one of the first questions we can ask is what are the objects in this scene? Just to get a sense of what are the objects that we can see here. As you can see, the video is being updated.

The scene contains several objects. A white countertop with a wooden base. A variety of snacks including Lays chips, gummy bears and Cheez Its. A blue water bottle and a white thermos. A laptop with stickers. A white coffee mug. A tray with napkins and paper plates. A kitchen area with a white table, chairs and a refrigerator. A microwave. A garbage can. A string of flags.

A window.

A blue chair. A water fountain. A man wearing a gray sweater and black pants. A woman with brown hair wearing a white tank top. These items and people create a busy kitchen or break room atmosphere with a mix of food, drink and everyday objects.

So now let's try to ask the robot how we can pick up one of the objects. So let's say we want to grab the white bottle. So let's say point to the white bottle just to see what we can get. And you can see that the point is shown here in the image. But let's say we want to be more precise. We want to grab the object by the handle. So we can say point to the handle on the white bottom. And as you can see, it is pointing to the white handle, the handle of the white bottle. Then at this point we are able to issue a command for the robot, a low level controller on the robot to go and perform the task. Now, after the bottle is moved, now we want to find a place in the scene that we can put the dirty bottles in. So let's say I point to where I should put the dirty dishes in. And you can see that it points to the sink. So the robot, then again the low level controller can take over. The robot will move towards the sink. Okay, so now the robot is here and we want to put the bottle that is in hand in the sink. We need to find the free spaces in there so we can just ask the robot, ask Momo, point to the free space in the sink and then press Enter. And as you can see, it's pointing to a free space in the sink. Now again, a low level controller can take over. Just ask a couple of other questions from the robot. For example, we can ask the robot where point to the tea bags. Or actually we can even ask the robot point to things I need to make TV bit. And we can see that the Momo model is pointing to all the T dice that we have in the scene. Point to the machine that I can use to make copy with. And you can see that it's pointing to the copy machine.

So now that you've seen all of that, let me know what you think about Momo. This was definitely a rather surprise. I didn't expect this just after the Llama 3 release. This was certainly a surprise. But this is why many individuals have consistently said that AI is rapidly advancing faster than you can imagine. I can't imagine how many other companies there are out there, you know, training their own models, doing their own things that are going to surprise us in a few months. If you enjoyed this video, I'll see you all in the next one.