Google Unleashes RT-2: The New AI Overlord for Robots!

Ever dreamt of telling a robot, hey, clean my room. And watching it spring to action, understanding every nook, cranny and object it encounters? Welcome to the futuristic realm of RT2, the next gen artificial intelligence shaping the bridge between human instruction, digital understanding and robotic action. So RT2 stands for Robotics Transformer 2, and it's a vision language action model, or VLA for short. A VLA model is a type of AI model that can understand both text and images from the web and translate them into robotic actions. This means that you can give a robot a simple command in natural language like throw away the trash and the robot will know what to do, even if it has never seen that task before. Which, to be honest, sounds pretty cool. Humans naturally learn from many sources like books, videos or personal experiences, and can apply this knowledge in new situations. For instance, if we see an apple on a table, we know its features and what to do with it without being taught every single detail. Robots, however, typically require specific data for their tasks. They need details about their environment and the actions they must perform. If anything changes slightly, they might not function correctly. For example, to make a robot dispose of trash, you'd need to feed it tons of details about trash, its appearance, handling and disposal. That's where RT2 becomes vital. It's a breakthrough in narrowing the learning differences between humans and robots using transformers, a kind of AI model that learns from vast Internet content, like how GPT3 produces text on varied topics. RT2 doesn't just process text or images, it also turns them into robotic actions. So you just need to instruct a robot in simple terms and it'll understand what to do alright. Now, RT2 uses two main a vision Language model and a Vision Language Action model. A VLM learns from text and images on the web, understanding things like what objects are and how they relate. The vla, which is an advanced vlm, not only learns, but can also direct robotic actions. The earlier model, RT1 could do simple tasks like picking up items, but was restricted to tasks it had seen before. RT2 is better than RT1 because it also learns from web data, which lets it do more versatile tasks and handle new situations. In essence, RT2 uses web information to learn in a way similar to humans, benefiting from the vast knowledge available online. Though unlike humans, it doesn't get lost in the rabbit hole of Wikipedia links or end up watching cat videos for hours. The VLM part of RT2 trains using lots of online text and pictures, like Wikipedia articles and news stories. It turns this content into special formats called embeddings which which capture the essence of the text or image. Later, these embeddings can be turned back into text or images. The vla part of RT2 also uses online data, but adds in robot data. This includes images the robot sees, commands given by people, and what the robot does in response. Like the vlm, the VLA turns this into embeddings. But it can also convert these embeddings into robot actions, which are shown as special action words. These these action words describe what the robot can do, like move or grasp, and provide specifics like direction and speed. For instance, move left tells the robot to move half a meter to the left. To make the VLA work, RT2 uses a method called VLM transformation. It adjusts the VLM to predict robot actions instead of just text or images. The process starts with a blank slate for predicting actions, but it leans heavily on what the VLM knows from online data. Because of this, it can do things it hasn't been directly trained on. For example, it might clean your room, even if that wasn't part of its robot training, simply because it knows about cleaning from online sources. So Google has shown that RT2 can do several things. It can sort trash like food wrappers, banana skins, and paper cups, then put them in a bin. But be warned, though, it might judge your diet choices. Also, it's good with various trash sizes and shapes, including crumpled paper and plastic bottles. It can tell objects apart. For example, it knows the difference between apples and tomatoes, or dinosaurs and dragons by how they look and their names. If you ask it to pick up the red fruit or pick up the extinct animal, it understands it's good at tasks with multiple steps. For instance, if asked to move a banana to two one, it figures out that means three, finds three items like cups and puts the banana there. This is because it thinks through steps, breaking big tasks into smaller ones. RT2 can manage situations it hasn't encountered before, using what it's learned online, so it can avoid obstacles like chairs when moving, even if it hasn't seen it on the web. If it makes sense based on what it knows, it can do it, like removing an apple when asked to move it to zero. It can adjust to new settings like different rooms. It can also do things on the spot, like catching a falling bag or cleaning up a spill with a towel. These abilities mean RT2 is helpful in daily life. It learns from what it does, so it gets better and can handle more. It's not just a robot that does things. It learns as it goes to understand the science behind RT2, we need to take a look at several of its innovative features. So first, RT2 uses what's called chain of thought reasoning, which lets it split hard tasks into smaller, simpler steps and tackle them one after the other. Imagine asking it to move a banana to the sum of two plus one. RT2 does it like figure out that two plus one is three, spot three items on a table, maybe cups, and then place the banana by these items. For every step, it uses its VLA model to decide on actions or questions. Like, to find out the sum, it might ask what is 2 plus 1? Then to identify items, it might command scan and later identify them. Lastly, to move the banana, it might decide move banana cup one and then do it. This way, RT2 can do complicated tasks without specific instructions. So in simple terms, RT2 playing Where's the banana is like grandma playing where's my glasses? Except one of them actually finds it. Then second, RT2 controls robots using action tokens. Think of these as simple commands. An example could be move left 0.5, telling a robot to go half a meter left. These tokens are easier for people to understand, can be used on various robots or places, work well with transformer models that deal with token sequences. With action tokens, RT2 can make robotic actions based on online data. It uses something called a VLM transformation, which changes a VLM to a vla so it predicts action tokens. And third, it can turn visual only jobs into robot actions, and that means only needing to see, no language needed, e.g. sorting items by color. And it does this by first using its VLM model to describe what it sees, and then it uses the VLA model to decide on an action from that Description. So if RT2 sees two apples, one red and one green, it might describe them and then decide on an action like move left. This lets it do more tasks even if they don't need language input. RT2 is clearly a major improvement over its previous version. While RT1, a VLA model, could do simple tasks like opening doors or picking up things in an office kitchen, it was restricted. It only did tasks that it had seen before compared to other models using different robotic control methods, like VC1, which uses vision and language, R3M, which relies on reinforcement learning, and Moo, which uses meta learning. Again, RT2 shows much better results in tests that measure a robot's skill in doing tasks based on language commands, RT2 scored 92.3% while VC1 got 85.6%. R3M had 81.4% and Moo got 79.8%. Plus, RT2 is more adaptable and stable than these models, especially in new or unfamiliar situations. Now, now RT2 could have a significant economic impact given the growing valuation of the industrial robotics industry. According to a report by Grandview Research, the global industrial robotics market size was valued at $44.6 billion in 2020 and is expected to grow at a compound annual growth rate of 9.4% from 2021 to 2028. Naturally, introducing robots and AI into our world presents a unique set of challenges and concerns, most of which revolve around the concept of trust. A fundamental question is whether humans will ever feel truly comfortable placing their trust in robots and AI. How can we guarantee their absolute safety, ensuring they never pose a threat to us? It's vital for these machines to consistently adhere to the guidelines and rules we establish for them. Also, a significant weight of responsibility falls on the shoulders of the engineers and developers behind these robots and AIs. They play a critical role in ensuring that these innovations not only function as intended, but also harmonize seamlessly with our societal values and expectations. Okay, if you enjoyed this video and want to stay tuned for more, do me a favor and hit that like button. And don't forget to subscribe to the channel. Thank you for joining me today and I'll see you in the next video.