A Two Minute Papers YouTube video by Dr. Zsolnai (June 17, 2023) surveying recent AI advances relevant to games and virtual experiences: NVIDIA’s system for generating in-game character dialogue (text, TTS, lip sync), NVIDIA’s ‘Neuralangelo’ photogrammetric neural surface reconstruction for high-fidelity 3D scene capture, and an AI animation paper simulating gibbon locomotion (University of British Columbia; code released). The video also includes a short sponsored segment about Weights & Biases (experiment-tracking tool; free for individuals/academics/open source).
Chapters
Dear fellow scholars, this is Two Minute Papers with Dr. Zsolnai. Today you are going to see glimpses of the future, many of which come from the amazing AI research papers that appeared not so long ago. First, Nvidia just showcased their new system which can automatically come up with dialogue for a virtual character in a video game. First, the text that the character needs to say will be synthesized by an AI. Then a text to voice AI will make it actually say it in a human voice. And the mouth movements will also be synthesized. Look.
Hey Jen, how are you? Unfortunately, not so good. How come? I am worried about the crime around here. It's gotten bad lately. My ramen shop got caught in the crossfire.
Wow, it sounds a little stiff. It could be a bit more natural than. But this is only the first version of something that was previously nearly impossible. And now here it is. And it may be part of the video games and perhaps even the virtual assistants of the future. This can be combined with Unreal Engine 5 and MetaHuman, which is an amazing system for creating virtual humans. And look at this. What is this? This is Nvidia's Neural Angelo. This performs something that researchers call photogrammetric neural surface reconstruction. It means that you can scan your environment with your phone camera and out comes a 3D mesh that you can use to create a virtual world for video games. Or have a video conference that actually takes place in this environment. Not just a 2D image, but actually being there. Previous techniques really struggled with this. Look, here is an input scene with a screw clamp. This is the neural implicit surfaces paper from two years ago. And as you see, it was unable to reconstruct the details on the screw. A more recent technique, Neural Warp from just one year ago, provides smoother results, that much is clear. However, the high frequency details are unfortunately still lost. Now with Neuro.
Look.
Wow. Now that is what I call a high quality reconstruction. Screws finally look like screws. And the paper reveals that this particular scene was not just a fluke. The new technique is so much better than the previous ones on a number of different scenes. So high fidelity neural reconstruction is becoming more and more of a possibility today. Loving it. We will be able to create a digital version of any place on this planet and play with our friends there. How cool is that? What a time to be alive. Okay, so generating speech is a possibility. And generating amazing virtual spaces is a possibility too. However, what about animation? Here is an amazingly fun paper that I would love to show you. And it is about simulating locomotion for virtual Gibbons. This is an AI based method which can jump from vine to vine. Wait a second, this can't do anything? Well yes, but this is just the initial part of the training. He doesn't quite know how to do this yet. Note that to the best of my knowledge, some of this is footage that is exclusive to us, so you can only see it here on two minute papers, nowhere else. But this doesn't seem to work too well, does it? However, this is a siggraph paper, so you fellow scholars immediately know that this has to be real good. So you know what, let's wait an hour for it to learn now, an hour later. My goodness, look at that. It can now jump from vine to vine and hold on to your papers, fellow scholars, because it gets even better. Look, it now understands that if it gets a couple more swings before jumping, it can even do this. Wow, that is incredible. Make sure to have a look at the paper in the video description. It is a ton of fun and I would like to send a huge thank you to the scientists at the University of British Columbia for also publishing the source code of this project for all of us for free. So as you see, AI research is progressing so incredibly fast, I may have to showcase these works, not one but three per video to be able to keep up. What do you think? Did you enjoy this? Let me know in the comments below. And for now, let the experiments begin.
Weights and Biases provides tools to track your experiments in your deep learning projects.
What you see here is their Tables.
Feature and the best part about it is that it is not only able to handle pretty much any kind of data you can throw at it, but it also presents your experiments to you in a way that is easy to understand. It is used by many prestigious labs including OpenAI, Toyota Research, GitHub and more. And the best part is that Weights and Biases is free for all individuals, academics and open source projects. Make sure to Visit them through wnb.com papers or just click the link in the video description and you can get a free demo today. Our thanks to Weights and Biases for their long standing support and for helping us make better videos for you.
Thanks for watching and for your generous support and I'll see you next time.
