Microsofts New 'KOSMOS 2' Multimodal Takes Everyone By SURPRISE! (Now RELEASED!)

So Microsoft have released another large language model called Cosmos 2. And this is a multimodal large language model that is very, very interesting. Now if what I said just confused you, please understand and I'll explain. Multimodal large language models are essentially language models that you can basically use with other modalities other than text. So for example, with this large language model, which is actually a working product and not just a research paper, you can actually submit images and get back a response. And this is the very big next step in artificial intelligence. As you know, ChatGPT has taken the world by storm, but every person right now who's working in artificial intelligence is trying to move the needle by looking at image recognition. And this is what Cosmos 2 aims to do. Now Cosmos 2 by Microsoft is, is a little bit different because of how they tackle certain problems. And I do think the way they've tackled a multimodal large language model in this research paper and with the live demo, I will show you one later in the video is arguably going to be how it's going to be done in the future. So let's take a look at the abstract so we can pretty much understand what they're talking about. So it says we introduce Cosmos 2, a multimodal large language model, enabling new capabilities of perceiving object descriptions and, and grounding text to the visual world. They also state that this work lays out the foundation for the development of embodiment AI and sheds the light on big convergence of language, multimodal perception, action and world modeling, which is a key step towards artificial general intelligence. And essentially, if you don't know what artificial general intelligence is, that is an AI system that's going to be an AI system that's capable of pretty much any task and it's going to be better than humans at any literally everything. And this is a key step towards that. So what exactly is Cosmos 2 and what can it do? So what we're going to do is we're going to take a look at some of the examples from the research paper because they showcase just how good it is at recognizing images, categorizing them, and then of course grounding them in reality. So what we can now see are three separate images. Okay. And what we're going to do is we're going to look at those images, the questions submitted with those images, and, and see if the AI can actually recognize those images. So right here you can see that we have a picture of an emoji. And the question is asking the, and the question asks the large Multimodal AI. Can you locate the left eye of the emoji? Then, of course, we can see that the completion is a small box around this left eye of the emoji. Now, essentially, what we're looking at Here is how Cosmos 2 actually works. You see, in the abstract, they talk about how they use bounding boxes to pretty much identify objects in that image, and that's how this AI labels and categorizes different things, which will be interesting further on in the demo, because you'll see it identify real things in real life with the same bounding boxes. Then, of course, we have a second image here, which is of course a picture of two cows in some kind of jungle or near some vegetation. Then, of course, we have the question, how many cows are here? Please answer. Then, of course, you can see two cows present in the image. And of course, it actually does this right here. It bounds the two boxes over the two cows, which is definitely going to be really interesting for those potentially with vision problems or just a multitude of different applications. For the third picture, we do have something that says, welcome to Carnaby Street. You can see that the question says, what does the sign say? And of course, the multimodal AI Cosmos 2 output welcome to Carnaby Street. And then of course, balance the box around this. Now, of course, you might think, why is it using a bounding box? Why is this not highlighting everything differently? I'm not exactly sure why they've chosen this. Maybe it's just the most effective method, but. So from these examples, we can see that Cosmos 2 clearly understands how to locate certain things which is in the picture. And of course, these different questions highlight different understandings. And what we can see from the first One is that Kosmos 2 can clearly locate certain things within an image. From number two, we can also see that Kosmos 2 is able to locate how many things there are within a subject image. And then of course, in subject three, what we do have is Kosmos 2 being able to read text from said image. So although you might just think that these are three random things chosen at random, they this isn't random at all. It's clearly demonstrating an ability to perform three separate tasks across three separate domains, which is really good for a multimodal AI, because every time a user is going to submit any sorts of image, there's going to be various different applications. Now, further on, we do have three more separate images that do require different sorts of responses, and they do ask them different questions when, which showcases just how good Cosmos 2 is at being a multimodal AI. So one of the first images that we do see here is why is this animal unusual? And what we do see here is a picture of a turtle and then we have a picture of a hare or a picture of a rabbit. And essentially we know that this animal is unusual because turtles are not notoriously fast. And then of course the completion from Kosmos 2 is, is this animal is unique because it is a turtle. Turtles are known for their slow and steady pace, whereas rabbits are known to be agile and quick. It is not common for a turtle to be faster than a rabbit, especially in terms of speed. So this first example shows us that Kosmos 2 has not just image identification, but a level of understanding. And it's important to know that this is completely different from those AIs that do have just image recognition. Because although image recognition is very good by identifying maybe that's an orange, maybe that's a chair, that's a table, understanding the nuances in images and decoding that is a huge step up and take a look at this image right here. Then of course we have Kosmos 2 saying this is a boy attached to the boat. And that shows a level of understanding. Now the example here, the third and last one, is definitely a very interesting one because it shows us once again the the Cosmos two can understand differences and extrapolate certain things from images. So the question says, what is the biggest difference between Bottle one and Bottle two? And then of course the answer is the biggest difference between Bottle one and Bottle two is the label. So that's actually very interesting because as you can see from the labels, we do see that one has a lemon and a strawberry and then of course the other one has a melon on it. And the biggest difference is clearly is of course the fruit and the label, which is once again, like we said, this large multimodal AI that Microsoft have just released presents a level of understanding which is not really seen in too much AIs. The only time we've really seen this level of image understanding and not image recognition is with GPT4's image identification, which is slowly being rolled out amongst users. If you did watch our recent video, so stay aware for that because that is going to be very interesting. But also later on in the video we will try this and demo it and Compare it to OpenAI's. Now this is one of the last examples that they do have in the paper, but it is one of my favorite because it does show that Cosmos 2 can provide detailed examples. So we can see here that they have an input prompt, and it says, describe this image in detail. And without looking at the description, I describe this as a snowman, see, sitting next to a campfire, probably trying to warm his hands, but at the same time, it's a little bit confusing because he could probably be melting. So it's a snowman at a campfire, probably having a hot meal. And then, of course, we can see the complete description along with all of the things that this AI has managed to recognize and then craft together. So we can see that a snowman sits next to a campfire in the snow. He's wearing a hat, a scarf, and mittens. There are several pots nearby, likely containing a hot meal for the snowman. The scene is serene and peaceful, with the snowman looking at the fire with a warm smile. Now, I got to be honest with you guys, that is a much better description than I did. And probably anyone who was looking at that image, I mean, being able to describe the sense of emotion, the sense of the aura, is definitely something that not everyone is capable to do. And then, of course, being able to locate and identify exactly what every single piece of this image is about is. Is something that is going to prove very, very useful. I mean, countless times on the Internet, we always see images where people are struggling to identify certain things. And think about this in certain applications, where you need to quickly identify what's going on in an image, this is going to prove very, very useful. Then, of course, we have a bit more technical stuff, and this is where they describe how it kind of works. Now, I'm going to pretty much gloss over everything here. And you can see that right here we have a dog in a field of flowers. And essentially what it does is it breaks everything down. It breaks it down into three parts. You can see it breaks it down into a dog, a field, and flowers. So we can see that it then tries to merge this together and drop the substrings, which it's called. Okay, so it drops the flowers, it drops the field of flowers. And then, of course, it says, a dog in a field of flowers. So this is essentially how the AI is working together, getting every single piece of that image and then merging it all into one cohesive prompt. And I think this is very, very effective when it comes to providing us with the information that we need. Look at some of the results compared to other benchmarks. We will start to realize that Kosmos 2 is actually better than these at first glance. We do see the other models like Visual, Bert, Glyp, Fiber and Grill. Lots of these models do have higher results, but but what they don't have is zero shot capability and that's what Kosmos 2 excels in. You can see here Kosmos 2 excels in zero shot where four other models lack. Then of course in this table, once again, Kosmos 2 and Grill are the only visual models that are able to successfully do this with their referring expression comprehension with zero shot accuracy though. So if you're wondering what a zero shot is in artificial intelligence, a zero shot refers to a scenario where a model is capable of performing performing a task without any specific training or examples related to that task. Instead, the model leverages its general knowledge or pre trained capabilities to make predictions or to generate outputs for tasks it has never seen before. So essentially this means that Cosmos 2 is a completely different level. Now, of course, let's get onto some of the live demos with Cosmos 2. Now what's also interesting is that if you're wondering about the text capabilities of this model, because of course, as you know it's it is able to identify images. Well, it's actually pretty good. It's actually on par with ChatGPT 3.5 in terms of its ability to recognize text and to be able to predict the next word. So Cosmos 2 isn't just an image classifier or something that can understand images. It's actually really good at those natural language tasks that we do know and love from ChatGPT. So now of course, here is the moment you likely have been waiting for. This is the live demo that Cosmos 2 presents us with. Now I do want to state that after releasing this video, if the video does blow up, there might be a time in which this page might be down frequently. That does happen when we do release these videos. So please give it some time. Either try and use it early or if the video has a lot of views, just wait sometimes. Because since these pages are demos, what often happens is the page will get overloaded and eventually the requests do take quite a lot of time to be submitted. So I've got to be honest with you guys, this, this software is far better than I thought. Like genuinely, I thought this was going to be a very, very awful demo. But the live demo is absolutely crazy and I've literally been waiting hours because I was greeted with this page where I had 502bad Gateway down for quite some time. And as you can see, I actually tried to grab an image where visibility was quite bad. You can see right here that we have a image of a plane wreck and the plane is rusted. It actually looks like it's part of the plane coral. And you can barely see the divers. But Cosmos 2 is successfully able to identify the divers and quickly identify the plane wreck, which is really, really interesting. So what I'm going to do is I'm going to go ahead and submit this again to see if it's able to generate another response that's far more detailed, as there are two different boxes which give you different responses, one being a brief response and one being a detailed response. So you can see here that after around 60 seconds, we then get this image. And it says, this image features a large, old, rusted airplane sitting on the ocean floor. Divers are swimming around the airplane, exploring its structure and exploring the surrounding area. There are several people in the scene, some of them closer to the airplane and others further away. The divers are spread out across the image, with some closer to the airplane and others in the back. We can see that that's a pretty accurate representation. But of course, I got it to generate this image where it says a leather soccer ball on a field. Now you. What I do like about this as well is that you can easily see which images it's talking about. Lots of the times when we do have these multimodal AIs, they just tell us what's in the image. But imagine you knew what was in the image, but you didn't know where it was. Maybe you have a visual disability. And Cosmos is able to outline exactly where it is and tell you what it's doing. So I think this is really, really useful and we're going to see some more applications for this in the future. Then, of course, I submitted another image where we have a man and his horse and. And Cosmos 2 gets this really cool. You can see it highlights the beach and then, of course, highlights his horse, and then of course, it highlights the man. So we can see that this is going to work on a range of different tasks. But I also wanted to present it more increasing, more difficult images as we went on, just to see how far we could push it. And this was one of the ones that I did think was going to be a little bit harder. But you can see that Kos mos2 handled it pretty easily. You can see an image of a person adjusting the rod with the holder was right there. It's got the person. It can actually put the bounding box over their arms. Then, of course, the rod holder and then of course the actual rod. So this is where you can see that it's able to identify every single piece of the image and then, of course, realize what's in it. Then we did another example of an image that was a little bit more detailed, and you can see right here that it actually managed to get this one as well. It highlighted the baked cod, highlighted all the vegetables around it, and was able to say an image of baked cod and vegetables on a baking tray as. And I did find that even though this was a brief description, it's still a pretty good description because when I did actually search this up, this does look like baked cod. So it is really, really good at identifying things in images. And of course, I didn't actually know this tool was called a grinder. That's why I input it into this image. But it actually got it pretty correctly. And I had to do a double check for a Google search to see if this was right. And you can see right here that it says close up of a worker cutting a granite slab with a grinder and dust is flowing off the pavement. So I don't think there's any image that it can't handle that well. It seems to be performing very well at simply every image I throw at it. And this is why I say Kosmos 2 is really good, because this was one of the images I threw at it, and I thought, there's no way it's going to get it. Now, some of you might know what these are, but honestly, for me, I was very, very perplexed. I was confused. I was like, what are these? Are these skittles? Are these random furry things? I genuinely had no idea. And then it told me that these were colorful pom poms with googly eyes. Now, for some of you, that might be something that you immediately know, but this object is pretty foreign to me. And of course, with a Google search, I'm able to quickly verify that this is what it is. So you could just imagine the amount of applications that this is where you want to immediately know what something is in an image online, or maybe you're trying to buy something and you're instantly able to get there. And these images were generated in around. And these responses were genuinely generated around 5, 5 to 6 seconds. So this is actually pretty quick. And for demos that is exceedingly fast, usually demos take anywhere between 60 seconds to around two minutes to accomplish their task, simply because the amount of users requesting it. Although, as we mentioned, once we release this video, that might change. So this is an image where it starts to hallucinate you can see that it says there are several cars visible in the scene, with, of course, no visible cars in the scene except one. Then it says the parking garage is filled with various vehicles and including a truck parked to the left, a car on the right side and a motorcycle parked further to its right. So I don't see a truck to the left or a motorcycle to the right. So it's clear that when you have the more detailed descriptions, sometimes this hallucination can occur. Then, of course, one of the last things I wanted to do was to test if it could actually handle black and white images. And it actually can, except there was a small hallucination with the left arm, because you can see right here that it says a dog, when it's in fact not a dog, it's just that person's arm. But of course it says a man, a trash can, and of course the street. And you can see here that the further detailed explanation doesn't actually hallucinate as much, but it does describe this as a bench, which is actually not a bench, it's actually a sign. But honestly, this is one of the most promising visual large language models that I've seen so far. And with multimodal AIs coming out soon, I genuinely cannot wait for GPT4's full image release. The only reason I do think it's currently being delayed is because they do want to ensure that it is currently safe. And we do know that OpenAI always maintains their safety routine of rigorously testing large language models, multimodal AIs ensuring that they're safe before public release.