A YouTube explainer by Dr. Zsolnai (2 Minute Papers) summarizing and comparing three new AI papers that enable text-to-video, text-to-3D (mesh) scene generation, and robust image inpainting/single-image 3D completion. The video contrasts these results with prior tools (Runway Gen1/Gen2), emphasizes trade-offs between visual quality and temporal/spatial coherence, and closes with a clarification about how Google DeepMind’s Gemini demo footage was produced.
Chapters
Today is a good day, fellow scholars. Why is that? Because we are going to talk about this, this and this. Three incredible new papers that will help you unleash your creativity like never before. And a word about Google DeepMind's Gemini AI at the end of the video. First, let's try to become a virtual film director and create videos. But wait, one of our previous episodes was exactly on that. Yes, but here you get to be a director, but not in the way you think. So how? Well, first let's start with a photograph. Now get this. Google's earlier AI helps us to fly into this photo. That is incredible. It even supports curved camera motions and long term videos. However, there is a problem. What is the problem? As you see, long term coherence is not the greatest. The scene seems to deviate further and further away from the origin photo. Of course, because it has to make up a lot of stuff. But there is a follow up paper on this. This is better, but the issues still remain the same. But we learned something important here. Whatever the new solution is going to look like, now we know that we should be evaluating it on quality and coherence. Now let's have a look at a new text to video technique. It promises zero shot scene generation. That is incredible. What is that? It means that it is able to generate scenes with these prompts. Kind of like an artist painting a landscape that they have never seen before. It did not have access to similar videos in its training data. And yet here it is. I love it. And back to quality and coherence. In my opinion, we have higher quality results. And coherence is really good too. All we need is just a piece of text input. So how. How is this even possible? Well, I will tell you in a moment. Dear fellow scholars. This is two minute papers with Dr. Zsolnai. Now let's compare it to previous techniques. This is against Runways Gen1. We showcased this in an earlier video too. Hmm. Do you see what I see here? Yes. It is not your eyes. The output seems a little slower. Gen 1 indeed generates a few duplicate frames. Otherwise not bad. This is a tool that is already being used out there to create movie projects. Not feature length movies. But do not let that fool you, AI. Film festivals are already being held where people put Together videos of 1 to 10 minutes in length. Not 5 second videos anymore. Prizes are also given out. And if that fires you up, you can also submit your own work too. Today everybody can become an artist and take part in this. What an incredible time to be alive, right? So that was Gen 1, I will note that Gen 2 also exist to the best of my knowledge, when without a paper. And now let's compare Gen1 to the new technique. And the new technique. Wow, this is an incredible jump in just one paper. And Gen 1 appeared not years ago, but less than a year ago. Such an improvement in less than one year. And as always, just imagine what we will be capable of just two more papers down the line. My goodness. But we are not done yet. Not even close. Now hold on to your papers, fellow scholars, and let's see how it performs against another technique from this year. And for me, this is once again, wow. Really, this kind of improvement in just one paper is very impressive. Now look here, because this is super important. What is this? Well, fellow scholars, this is the mesh representation of the scene, digital 3D geometry. And I hear you asking. Great. Now we can put it into our video games. That is important. But is that super important? Yes, it is. You see, meshes are essential in understanding this work. And here comes one of my favorite things about this paper. First, it starts out with a mesh structure digital geometry. And over time, this AI is like a sculptor who is working on a clay model. Yes. As the camera moves around, it grows this mesh outwards. Loving it. Now, this was text to video and also text to 3D geometry. But now, wait a second. Text prompts are great, but what if we already have an image of our scene instead? But that's just a 2D image. And we would like 3D geometry instead. Well, have a look at this paper. Oh yeah. It promises that it takes just this one image and it guesses what is around it and what is behind it in a coherent manner. Well, it promises that, but I will believe it when I see it. Whoa, look at that. It's not perfect by any means, but it is real good. And good is an important keyword. Why? Well, take it from the legendary chip engineer Jim Keller.
And there's a website or a YouTube channel called 2 Minute Papers. He says this rule is if it's good, it's just two papers away from being great. And this is going to happen pretty fast.
That was an incredibly good summary of what you see here and a huge honor. Thank you. So, yes, if it's good, two more papers down the line and it will be great. So we can now create these videos from text or a photograph. But get this, you might not even need a photograph to perform all this. Yes, that's right. With the third paper today, you can even have a photo that is almost completely destroyed. And it will be able to restore it really well. And this can then go into the photo to 3D techniques. Can you imagine before watching 2 minute papers that someone says that from this photo and two AI research papers and it will be restored with amazingly high quality and then you can even put it into a video game. No one would have believed it. And yet here we are.
Wow.
Now note that of course image inpainting techniques already exist. There are heaps and heaps of papers on it. For instance, here is a legendary paper that patch match from 14 years ago, all handcrafted human ingenuity. A graphics paper of course that was really impressive. But compared to what we can do today. Wow, I am out of words. What a time to be alive. Now, as promised, a word about Gemini. We discussed it in detail a few videos ago and I made sure that we mainly discuss results not from the marketing materials but from the research paper itself. We are fellow scholars here and paper results are more detailed and easier to verify. However, while I was talking I also showed you some materials like this. In the meantime, Google discussed how this footage was made and it was made not with the AI's real time feedback on a video, but they used still images from the video plus wrote a prompt to give it to the AI. The answers were then showed here. These prompts range from simple like what do you see here? To something more detailed. However, we did not see these prompts. I think best would have been if it were presented in a way like you see here. And second best would have been adding a disclaimer to the footage that still images and additional prompting was used. The paper results that span the majority of our videos are to the best of my knowledge accurate. I wanted to make really sure to revisit this just to make sure that you always get accurate information about these works. That is the way of the scholar. When I was a PhD student, my experiments created lots of data for my deep learning projects. But I was not looking for more data, I was looking for insights. This is what weights and biases does and it is the best everyone is using it. Try it out now at WNB Me Papers or click the link in the description below.
