This is a Two Minute Papers YouTube video (Dec 23, 2021) by Dr. Károly Zsolnai-Fehér presenting "nerfies"—selfie-derived 3D portraits that can be rotated—and a new method called HyperNeRF that handles animations and moving subjects much better than the original 2020 nerfie approach. The video demonstrates HyperNeRF’s ability to freeze and select specific moments via a described "hyperspace", highlights the central role of accurate depth maps, shows challenging examples (coffee being made, melting chocolate), and ends with a sponsor message from Weights & Biases.
Chapters
Dear fellow scholars, this is Two Minute Papers with Dr. Zsolnai. Today we are not going to create selfies. We are going to create nerfies instead. What are those? Well, nerfies are selfies from the future. The goal here is to take a selfie video and turn it into a portrait that we can rotate around in complete freedom. The technique appeared in November 2020 a little more than a year ago, and as you see, easily outperformed its predecessors. It shows a lot of strength in these examples and seems nearly invincible. However, there is a problem. What is the problem? Well, it still did not do all that well on moving things. Don't believe it. Let's try it out. This is the input video. Aha. There we go. As soon as there is a little movement in time, we see that this looks like a researcher who is about to have a powerful paper deadline experience. I wonder what the nerfy technique will do with this. Uh oh, that's not optimal. And now let's see if this new method called hypernurf is able to salvage the situation. Oh my. One moment perfectly frozen in time. And we can also move around with the camera. Kind of like the bullet time effect from the Matrix. Sensational. What is also sensational is that of course, you seasoned fellow scholars immediately ask, okay, but which moment will get frozen in time? The one with the mouth closed or open? And hypernerf says, well, you tell me. Yes, we can even choose which moment we should freeze in time by exploring this thing that they call the hyperspace. Hence the name hypernerf. The process looks like this. So if this can handle animations well, then let's give it some real tough animations. This is one of my favorite examples where we can even make a video of coffee being made. Yes, that is indeed the true paper deadline experience. And a key to creating these nerfies correctly is getting this depth map right here. The colors describe how far things are from the camera. That is challenging in and of itself. And now imagine that the camera is moving all the time. And not only that, but the subject of the scene is also moving all the time too. This is very challenging and this technique does it just right. The previous nerfy technique from just a year ago had a great deal of trouble with this chocolate melting scene. Look, tons of artifacts and deformations as we move the camera. Now hold on to your papers and let's see if the new method can do even this. Now that would be something. And wow. Outstanding. I also loved how the authors went the extra mile with the presentation of their website. Look, all the authors are there, animated with this very method, putting their money where their papers are. Loving it. So with that, there you go. These are nerfies, selfies from the future. And finally, they really work. Such amazing improvements in approximately a year. The pace of progress in AI research is nothing short of amazing. What a time to be alive. This video has been supported by Weights and Biases. Look at this. They have a great community forum that aims to make you the best machine learning engineer you can be. You see, I always get messages from you fellow scholars telling me that you have been inspired by the series but don't really know where to start. And here it is. In this forum you can share your projects and ask for advice, look for collaborators and more. Make sure to visit WNB me paperforum and say hi or just click the link in the video description. Our thanks to Weights and Biases for their long standing support and for helping us make better videos for you. Thanks for watching and for your generous support and I'll see you next time.
