NVIDIA’s New AI: 50x Smaller Virtual Worlds!

Today we are going to create absolutely incredible virtual worlds with these new papers. First, Nvidia did something here, but if the quality does not seem to be too much better here, then how does this really help? We'll find out together. Then we are going to re sculpt an image with this collaboration between intel and New York University. Then we will become a movie director and give directions to not people. But get this images. Oh yes, and then we won't even need to direct these images. This AI technique directs the video by itself. Dear fellow scholars, this is two minute papers with Dr. So with nerfs, we are able to gather a bunch of photos and have a technique stitch it together into a virtual world. There are models that can do this extremely quickly. For instance, instant neural graphics. This converges in a matter of seconds, which is kind of insane. And its quality is often even better than its predecessors. You can see a great deal more detail in the hair and the sweater. And now let's see the new technique. Ready to be blown away. Wait a second. This looks nearly the same. So. So is this better? If it is better, how? Well, what you see here is quality. But quality is just half of the story. The other half is size. We haven't talked about that yet. The first technique is reasonably sized, but the quality is lacking. Then comes instant neural graphics Quality. Much better, but the size is much larger. And now look at the new technique, which looks roughly the same, but oh my. It packs the same quality, but in 1/5 the size. Fantastic. In this sense, this is even better than the legendary new technique Gaussian splatting, which can create and now even animate virtual worlds. And this new one is 50 times more compact than that. Crazy. Now let's co opt some images. Second paper here. The goal is to take an image, any image, and then convert the people or objects in it into a 3D model. But not to create a video game character from them. Although that's quite nice too. But no, not here. Here we now have knowledge about the backside of this model too. So we can choose a new pose for our character and and apply some more magic to put it back into the image. With the new pose, we can even rotate them, you name it. Shifting these objects to new positions is also possible. And wait, these are 3D models, so we can even apply deformations to them. Carve out that bad boy. And there we go. Apart from some suspect artifacts at the mouth region, this one is almost perfect. Or placing new ducklings or fish into an image. Not a problem. And this concept Gives us a great deal of control over these images. For instance, how many cherries would you like? How about this one? And another one and another one. The consistency between the images is not perfect, but they are nearly the same. And just imagine what we will be capable of. Just two more papers down the line. My goodness, what a time to be alive. And we are not done with magic for today. Not even close. With this other work, we can apply some more artistic direction to already existing images. Just look at the arrows. This indicates our wishes as to how the image should be moving. And bam. We get a video. This works great for camera movement. You know what? I wonder what happens if I instruct this horse to move? That is so much more complex than just camera movement. So what happens then? Now hold on to your papers, fellow scholars. And my goodness, look at that. The AI understands how a horse should move and synthesizes exactly that. It is not perfect, not even close. But this is once again an excellent opportunity to invoke the First Law of Papers. What is that? Well, the first Law of Papers says that research is a process. Do not look at where we are, look at where we will be. Two more papers down the line. Remember what Dall E1 could do in terms of text to image, and then Dall? E2 dropped and blew it out of the water. And just imagine what a Dall E2 moment for this kind of video synthesis could be. Wow. And now check this out here. This AI technique looked at videos of people in real conversations. And then all we need is our audio input. Then get this. It creates virtual characters, mouth movements, and even gestures automatically, so we can have conversations in virtual worlds more easily. Wait, this avatar is animated only by my voice? That's cool. Yeah. Let me show you how it works. I have to say, the synthesized movements are often expressive, I give you that. But also sometimes a little stiff. Mouth movement is not that accurate yet. But it is very impressive that from just the audio, all this can be synthesized once again. Just two more papers down the line and you might start seeing this out there in the real world. And I think this work is a really good showcase of how difficult this problem is. You see, our brains are wired to look at each other and read each other's expressions. Thus, if even a little hesitation, just a tiny smirk, if just the slightest things are off, we immediately know that something is wrong. We are wired for that. So making this work properly will be incredibly difficult. And. But if something human ingenuity and the power of AI will be able to do that. If you're looking for inexpensive cloud GPUs for AI. Lambda now offers the best prices in the world for GPU cloud compute. No commitments or negotiation required. Just sign up and launch an instance and hold on to your papers. Because with the Lambda GPU cloud you can now get on demand H100 instances and and they are one of the first cloud providers to offer publicly available on demand H100 access. Did I mention they also offer persistent storage? So join researchers at organizations like Apple, MIT and Caltech in using Lambda Cloud instances, workstations or servers. Make sure to go to lambdalabs.compapers to sign up for one of their amazing GPU instances today.