This content was created by LARRUZ. This content was uploaded by LARRUZ. This is a YouTube video titled "Googles 'MultiModal' AI Gato Continues To To SHAKE UP The Industry" (created 2023-06-28, last edited 2026-01-02). The content is a commentary/presentation with two speakers (labelled SPEAKER A and SPEAKER B) reviewing DeepMind's Gato research paper (published 2022) and its multimodal, generalist-agent approach. The speakers summarize DeepMind's background (AlphaFold, AlphaGo), describe Gato’s multimodal capabilities (image captioning, chat/dialogue, Atari gameplay, robot control), note its small parameter size (~1.2B), discuss limitations (superficial or factually incorrect chat outputs, hallucinations), and highlight a recent real-world application (Robocat) to argue for Gato’s potential impact and future improvement via techniques like reinforcement learning from human feedback.
Chapters
In this video, we need to discuss a research paper that was essentially released last year, but it was one of those research papers that since the rise of AI, has been somewhat forgotten. Now, up until recently, there was not really the mention of multimodal AI models, but as you do know, there are certain companies and research teams out there that do try to push the needle on what we do know and, and what we think is capable. Now, one research team that has consistently pushed the needle in terms of what is possible is Google's DeepMind. And if you're not familiar with this team, well, let me just gloss over some of their accomplishments. Now, you might not be familiar with DeepMind, but they are a division of Google, a specific research team that do constantly produce new research papers, new studies that showcase just how far we can go with artificial intelligence. Now, DeepMind are mainly noteworthy for two main projects amongst the other countless other research papers that they've done. The first being AlphaFold, which can accurately predict 3D models of protein structures and is accelerating research in nearly every field of biology. And of course, AlphaGo, which was the first computer program to defeat a professional human Go player, the first to defeat a GO world champion, and is arguably the strongest Go player in history. Now, if you're wondering why this research team is being highly regarded just because of a computer that was able to beat a human this understand that Go is a board game with simple rules but an incredibly large number of possible moves and configurations. To boil things down, the number of possible positions in Go is estimated to be more than the number of atoms in the universe, which makes it difficult for computers to evaluate and to choose the best news configurations of the board.
Is more than the number of atoms in the universe. AlphaGo found a way to learn how to play Go. So far, AlphaGo has beaten every challenge we've given it, but we won't know its true strength until we play somebody who is at the top of the world, likely to die.
And what was crazy about DeepMind's AlphaGo is that the move sometimes appeared unconventional and surprising to human players, which made people think that this AI behaved as if it was thinking and created new strategies that hadn't been considered before. Of course, that just glosses over DeepMind's history. But what this video is about is one of DeepMind's paper that, like we stated before, was released last year. Now, if you're wondering why we're deciding to cover this, it's because this framework was recently used in a recent project called Robocat. Now, what this Paper is, is, it's called gato and essentially it's quite simply a mini AGI or one of the first glances at what an AGI system could look like in its very early stages. So in the abstract DeepMind state that inspired by progress in large scale language modeling, we apply a similar approach towards building a single generalist agent beyond the realm of text outputs. The agent, which we refer to as gato, works as a multimodal, multitask, multi embodiment, generalist policy. The same network can play Atari caption images, chat stack blocks with real robot arms, and much, much more, deciding based on its context whether to output text, joint talks, button presses or other tokens. In this report we describe the model and the data and document the current capabilities of GATE. As many of you know, ChatGPT has taken the world by storm and in doing so it's kind of overshadowed some of the other AI models that were released being researched. And this is one of the frameworks that I really do think is interesting and worth covering now when there's much more of an AI crowd. So essentially GATO is an AI model that is completely multimodal. Now for those of you who don't know what that means, that just essentially means that it can do more than ChatGPT. You see, ChatGPT is simply a text based AI that can generate long pieces of coherent text based on a single or small user prompt. But with gato, essentially what you have is you have varying different outputs based on the user's input. And this means it can handle many different modalities. Like it stated before, a year after this paper was released, there have been a lot more interesting multimodal AIs that have been worked on, such as Microsoft's Visual ChatGPT and Microsoft's Jarvis, which was very, very interesting because it essentially was a multimodal AI. But moving on from jarvis, if we look at gato, the possibilities here are truly incredible. You see, what makes GATO different from other AIs such as Microsoft's Visual ChatGPT and other AIs that are additionally multimodal, including images, video and text, is that GATO can be applied to the physical world. Which means that this kind of AI system, if developed more so, can have real world implications. So we're going to cover some of gato's most impressive capabilities. Number one is going to be the ability to caption images. Now, as you may know, before GPT4 did talk about how they're going to release their multimodal features later on sometime in the year. We aren't sure when this is scheduled for, but we can predict that at least by the end of the year this should be released. And we do know that this does include the ability to describe images. Now remember, this paper was released in 2022. But what we can see here is a bunch of images, 10 to be exact. The image right here that we can see says three different captions under each different image. So it states that the colorful ceramic toys are on the living room floor. A living room with three different color deposits on the floor and a room with a long red rug. A TV and some pictures. There are also some other images right here. And I do think that this is very interesting because further on in the paper we do get to instances where this AI can do things which we really didn't think it was going to be able to. You can also see that along these images, the captions are pretty accurate. A bearded man holding a plate of food. A man holding a banana to take a picture of it. A man holding a slice of cake. Then we have a big truck in the middle of the road. Then we have a baseball player pitching a ball on top of a baseball field. Then we have a surfer riding a wave in the ocean. A surfer with a wetsuit riding a wave. Then we have a white horse with a white and blue silver bride. And it's interesting to see how this AI handled these problems. Now, what's important to note here, although some of you may think that this image captioning isn't that great, you have to understand that in this research paper they specifically decided not to cherry pick any responses because they wanted to showcase the true capabilities. Now understand that with reinforcement, learning, with human feedback, this kind of AI could get incredibly better. For example, when it output all three of these captions, you, a human could then select the best one and give the artificial intelligence a token that tells it this was a good choice. And that's how over time, AI systems become better and better. Then of course we have an interesting feature. And to be honest, I'm not entirely sure why GETO wasn't released by Google or developed into a full fledged product. What we can see here is this is the chat functionality by Gato. So this is called Chit chat with Gato. Dialogues with Gato when it is prompted to be a chatbot, usually Geto replies with a relevant response, but is often superficial or factually incorrect which could be improved with further scaling. And we use the same prompt in another paper. So I find the conversational tone of this to be very interesting because it starts with a user saying, explain the basics of black holes to me. And the artificial intelligence states. So I guess I'm trying to answer this simple question, what's a black hole? The user comments back, indeed, please go ahead. And then it gives a pretty decent explanation of a black hole. But of course we do see some instances where there are some hallucinations. We can see that when it's stated, what is the capital of France? They say Marseille, which of course isn't the capital of France. Now, although you might be thinking, okay, this is an AI that isn't that particularly impressive on any given task. Why are people talking about this AI when there are far better AIs in certain categories? Well, you have to understand that this kind of AI model doesn't use a large number of parameters, such as ChatGPT. This model only used around 1.2 billion parameters in the case of Gates. So this AI, what you have to understand, although it might lack complete depth, it can do a wider range of tasks. And it also can do strange tasks, for example, playing video games like Atari, which is not something that other AIs can do. And remember, the goal here for this kind of paper was to create something that could handle pretty much any task needed. And whilst you might think that these are just simple experiments and research driven studies, this is not the case. You see, DeepMind's robocat, which was released a couple of days ago, proves that Gato's framework can be used in the real world applications that many people may need. Which goes to show that once these AI models do have a base for that, eventually, over time, we're going to be able to build upon these large multimodal models and implement them into the real world.
