Waymo's AI Recreates San Francisco From 2.8 Million Photos! 🚘

And dear fellow scholars, this is Two Minute Papers with Dr. Zsolnai. Today we are going to see if Waymo's AI can recreate a virtual copy of San Francisco from 2.8 million photos. To be able to do that, they rely on a previous AI based technique that can take a collection of photos like these and magically create a video where we can fly through these photos. This is what we call a nerve based technique. And these are truly amazing. Essentially, photos go in, the AI fills in the gaps and reality comes out. And there are a lot of gaps between these photos, all of which are filled in with high quality synthetic data. So as you see with these previous methods, great leaps are being made. But one thing stayed more or less the same and that is the scale of these scenes is not that big. So scientists at Waymo had a crazy idea and they said rendering just a tiny scene is not that useful. We have millions of photos laying around. Why not render an entire city like this? So can Waymo do that? Well, maybe, but that would take Waymo. I am sorry. I am so sorry. I just couldn't resist. Well, let's see what they came up with. Look, these self driving cars are going around the city. They take photos along their journey and. Well, I have to say that I am a little skeptical here. Have a look at what previous techniques could do with this dataset. This is not really usable. So could Waymo pull this off? Well, hold on to your papers and let's have a look together. My goodness. This is their fully reconstructed 3D neighborhood from these photos. Wow, that is superb. And don't forget, most of this information is synthetic that is filled in by the AI. Does this mean that? Yes, yes it does. It means three amazing things. One, we can drive a different path that has not been driven before by these cars and still see the city correctly. Two, we can look at these buildings from viewpoints that we don't have enough information about. And the AI fills in the rest of the details. So cool. But it doesn't end there. No sir, not even close. Three, Here comes my favorite. We can also engage in what they call appearance modulation. Yes, some of the driving took place at night, some during the daytime. So we have information about the change of the lighting conditions. What does that mean? It means that we can even fuse all this information together and choose the time of day for our virtual city. That is absolutely amazing. I love it. Yes, of course not even this technique is perfect. The resolution and the details should definitely be improved over time. Plus it does well with A stationary city, but with dynamic moving objects, not so much. But do not forget the original first NERF paper was published just two years ago. And it could do this. And now just a couple papers down the line. And we have not only these tiny scenes, but entire city blocks. So much improvement in just a couple papers. How cool is that? Absolutely amazing. And with this, we can drive and play around in a beautiful virtual world that is a copy of the real world around us. And now if we wish it to be a little different, we can even have our freedom in changing this world according to our artistic vision. I would love to see more work in this direction. But wait, here comes the big question. What is all this good for? Well, one of the answers is sim2real. What is that? Sim2real means training an AI in a simulated world and trying to teach it everything it needs to learn there before deploying it into the real world. Here is an amazing example. Look. OpenAI trained the robot hand in a simulation to be able to rotate these Rubik cubes and then deployed this software onto a real robot hand. And look, it can use this simulation knowledge. And now it works in the real world too. But Sim 2 Real has relevance to self driving cars too. Look, Tesla is already working on creating virtual worlds and training their cars there. One of the advantages of that is that we can create really unlikely and potentially unsafe scenarios. But in these virtual worlds, the self driving AI can train itself safely and when we deploy them into the real world, they will have all this knowledge. It is fantastic to see Waymo also moving in this direction. What a time to be alive. So what would you use this for? What do you expect to happen a couple more papers down the line? Please let me know in the comments below. I'd love to hear your thoughts. What you see here is a report of this exact paper we have talked about which was made by Weights and Biases. I put a link to it in the description. Make sure to have a look. I think it helps you understand this paper better. Weights and Biases provides tools to track your experiments in your deep learning projects. Using their system, you can create beautiful reports like this one to explain your findings to your colleagues better. It is used by many prestigious labs including OpenAI, Toyota Research, GitHub and more. And the best part is that Weights and Biases is free for all individuals, academics and open source projects. Make sure to Visit them through wnb.com papers or just click the link in the video description and you can get a free demo today. Our thanks to Weights and Biases for their long standing support and for helping us make better videos for you. Thanks for watching and for your generous support and I'll see you next time.