Google combined language models, image-language models, and self-driving training to build a robot that follows complex spoken instructions. The robot learned from unannotated data and completed navigation tasks based on natural language commands in two experiments.
Dear fellow scholars, this is Two Minute Papers with Dr. Zsolnai. Today we are going to give instructions to Google's new self driving robot and see if it is smart enough to deal with these nasty tasks. Look at that. There is no way, right? Well, with our current tools I am not so sure. But let's be ambitious and think about a possible dream scenario for a self driving robot. For this we need three puzzle pieces. You see, whenever we talk about self driving techniques, the input is an image and the output is a question like ok, self driving car, what would you do next? But note that the input is not a sentence. These AIs don't speak English. So puzzle piece number one language. For a possible solution to that, have a look at OpenAI and Nvidia's AI that can play Minecraft. And here comes the best part. They also understand English. How do we do that? Well, we can use OpenAI's earlier technique called GPT3 which has read a huge part of the Internet. And get this, it now understands English. However, we are not done yet. Not even close. Puzzle piece number two, Images. If we wish it not only to be able to understand English, but to understand our instructions, it needs to be able to connect these English words to images. You see, it knows how to spell stop sign, but it hasn't the slightest clue what a stop sign really looks like. So we need to add something that that would make it understand what it sees. Fortunately we can also do this for instance using the recent Dall EI that took the world by storm by being able to generate these incredible images given our text descriptions. And this has a back and forth understanding between words and images. This module is called Clip and it is a general puzzle piece that fortunately can be reused here too. But we are not done yet. Now puzzle piece number three. Of course we need self driving, but make it really hard for this paper. For instance, train the AI on a large unannotated self driving dataset. Which means that it does not get a great deal of help with the training. There is lots of data but not a lot of instructions within this data. Then finally, let's chuck all these three pieces into a robot and see if it became smart enough to be able to get around the real world. Of course there is no way, right? That sounds like science fiction and it might never happen. Well, I hope you know what's coming. Now hold on to your papers and check out Google's new self driving robot which they say has all three of these puzzle pieces. So let's take it out for a ride and see what it can do in two amazing experiments. First, let's keep things light. Experiment number one, let's ask it to recognize and go to a white building. Nice. It found and recognized it. That's a good start. Then continue until you see a white truck. Now continue until you find a stop sign. And we are done. This is incredible. We really have all three puzzle pieces working together. And this is an excellent application of the second law of papers which says that everything is connected. This AI can drive itself, understands English, and can see and understand the world around it too. Wow. So now are you thinking what I am thinking? Oh yes. Let's give it a really hard time. Yes. Final boss time. Look at this prompt. Experiment number two, let's start following the red black building until we see a fire hydrant. And haha. Let's get a bit tricky. Yup, that's right. No fire hydrant anywhere to be seen. Your move, little robot. And look at that. It stops, turns around and goes the other way. And finally finds that highly sought after fire hydrant. Very cool. Now into the grove to the right. Okay, this might be it. And now you are supposed to take a right when you see a manhole cover. No manhole cover yet. But there were ones earlier that it is supposed to ignore. So good job not falling for these. And now the search continues. But no manhole cover. Still not one in sight. Wait, Got it. And now to the right and find a trailer. No trailer anywhere to be seen. Are we even at the right place? This is getting very confusing. But with a little more perseverance. Yes, there is the trailer. Good job little robot. This is absolutely incredible. I think this description could have confused many humans. Yet the robot prevailed. All three puzzle pieces are working together really well. This looks like something straight out of a science fiction movie. And a moment that I really loved in this paper. Look, this is a scientist who is trying to keep up with the robot. In the meantime, imagine this person going home and being asked Honey, what have you been doing today? Well, I followed the robot around all day. It was the best. How cool is that? So with DALL E2 and stable diffusion, we are definitely entering the age of AI driven image creation. And with DeepMind's AlphaFold, perhaps even the age of AI driven medicine. And we are getting closer and closer to the age of AI self driving and navigation too. Loving it. What a time to be alive. And before we go, have a look at this. Oh yes. As of the making of this video, only 152 people have looked at this paper and that, dear fellow scholars, is why two minute papers exist. I am worried that if we don't talk about some of these works in these videos, almost no one will. So thank you so much for coming along this journey with me and if you feel that you would like more of this, please consider subscribing if you're looking for inexpensive cloud GPUs for AI, Lambda now offers the best prices in the world for GPU Cloud compute. No commitments or negotiation required. Just sign up and launch an instance and hold on to your papers because with Lambda GPU Cloud you can get on demand a 100 instances for $1.10 per hour versus $4.10 per hour with AWS. That's 73% savings. Did I mention they also offer persistent storage? So join researchers at organizations like Apple, MIT and Caltech in using Lambda Cloud instances, workstations or servers. Make sure to go to lambdalabs.compapers to sign up for one of their amazing GPU instances today. Thanks for watching and for your generous support and I'll see you next time.
