A Two Minute Papers video by Dr. Zsolnai describing a recent Google AI that synthesizes long aerial ‘fly-through’ videos from a single photograph. The presenter explains that the method combines image inpainting, outpainting, and super-resolution, compares it to a prior (drone-trained) approach, and highlights improvements in visual quality, temporal coherence, and controllability. The video also includes a sponsorship mention for Weights & Biases.
Chapters
Dear fellow scholars, this is Two Minute Papers with Dr. Zsolnai. Today we are going to use an AI to fly into this photo. And we will see how much better this technique has become in just one year. It will be insanity. Yes. In a previous video we explored an insane idea. What if we could take just one photograph of a landscape and then we would fly into this photo like a bird? Of course, that is a big ask. Because to be able to do this, we would need to invent at least three things. One is image in painting. Look, when we start flying, the regions between the trees are missing. We need to generate those two. Information is also missing, not just within, but around the photo. This is a huge problem because completely new regions should also appear that are beyond the image. This means that we also need to perform image outpainting, creating these new regions from scratch, continuing the image, if you will. And three, as we fly closer to these new regions, we will be looking at fewer and fewer pixels. And from closer and closer we. Which means this. For this we would need super resolution in goes a coarse image or video. And this AI based method is tasked with this. Yes, this is not science fiction. This is super resolution where the AI starts out from noise and synthesizes crisp details onto the image. So last year, scientists at Google created an amazing AI that was able to learn and fuse all these three techniques together to create this. Wow. So this is possible after all. Well, hold on to your papers because it is not only possible, but the follow up paper is already here. We are now one more paper down the line. Less than a year later. I know you're itching to see it. Me too. So let's compare them together. These methods will all start from the same point and oh my, look how quickly they deviate. The two earlier methods quickly break down. And this is the work that we talked about a few weeks ago. This is clearly much better. However, as every single one of you fellow scholars can see, it lacks temporal coherence. What is that? Well, this means that the AI does not have a long term vision of what it wishes to do, and it barely remembers what it just did a few frames ago. As a result, these landscapes start morphing into something completely different very quickly. And now you know what's coming. So hold on to your papers and let's look at the new technique. My goodness, I love it. So much improvement in just a year. Now everyone can see that these are also not perfect. But this kind of improvement, just one more paper down the line, is nothing short of amazing. Especially that it doesn't only have better quality. No, no, it can do even more. It offers us more control too. Now we can turn the camera around and whenever we see something interesting, we can decide which direction we wish to go. And the result is that now we can create and also control these beautiful long aerial videos where after the first few frames, every single thing is synthesized by an AI. How cool is that? What a time to be alive. And it doesn't stop there. It gets even better. If you have been holding onto your paper so far, now squeeze that paper. Because this new AI synthesizes these videos without ever having seen one. That's right, it had never seen a video. The previous work was trained on drone videos. But training this one only requires a collection of single photographs. Multiple views and the camera position are not required. That is insanity. This AI is so much smarter than the previous one that was published just a year ago. And it requires training data that is much easier to produce at the same time. And I wonder what we will be capable of just two more papers down the line. So cool.
This video has been supported by weights and biases. Being a machine learning researcher means doing tons of experiments and of course creating tons of data. But I am not looking for data, I am looking for insights. And weights and biases helps with exactly that. They have tools for experiment tracking, dataset and model versioning and even hyper parameter optimization.
No wonder.
This is the experiment tracking tool choice of OpenAI, Toyota Research, Samsung and many more prestigious labs. Make sure to use the link WNB me paperintro or just click the link in the video description and try this 10 minute example of weights and biases today to experience the wonderful feeling of training a neural network and being in control of your experiments. After you try it, you won't want to go back.
Thanks for watching and for your generous support and I'll see you next time.
