Károly Zsolnai-Fehér shows how Unreal Engine 5's Metahuman Animator lets you capture and animate realistic virtual humans in minutes, with neural networks now running body and garment simulations in real time instead of seconds per frame.
Dear fellow scholars, this is 2 Minute Papers with Dr. Today we are going to talk about metahumans. What are those? Well, see here for yourself. This will be a culmination of a ton of papers that we talk about here. For instance, the hair has to be simulated, often strand by strand. The appearance of skin has to be correct and. And deformations to the skin also have to be simulated correctly. And here comes the most challenging part. All this on a shoestring computational budget. Or in other words, yes, in real time. Is this even possible? Well, let's see it together. Yes, this is metahuman, part of Unreal Engine 5, a popular game development platform where we can create virtual humans like we do in a video game. With the difference that these are incredibly high quality models. Today, millions of these metahumans already exist, but that is nothing compared to what you are about to see now. Now they have introduced metahuman animator, where we can scan ourselves and enter a virtual world, so they say. So let's have a look together. First we capture ourselves, and by that I mean our appearance and gestures too. Then, after less than a minute of processing, now it is able to locate and track facial landmarks like the eyes, eyebrows and mouth. And now it can show us the DNA of our virtual character. This can mimic our gestures really well already. But yes, I hear you asking, ok, doc, but it doesn't look like the test subject at all. So now hold on to your papers, fellow scholars, and watch this. Wow. Now we're talking. That is simply incredible. This did not require the expertise of a group of artists, a huge studio with a huge budget, and it did not even require months of work. It is done right, as we are talking, in minutes. What a time to be alive. And if that is at all possible, it gets even better. We can also transfer these gestures to a variety of different characters. That is excellent in and of itself. But just imagine how we could make completely virtual characters and life forms come to life with our own gestures. How cool is that? And here are a few more features that I absolutely loved. We can design the full body of our characters. We can even choose clothes for them. And if we are not happy with our character, we can even make small adjustments to it. Now, after creating or scanning these characters, they are now allowed to enter a virtual world. However, when they move around in this world, their movement and as a result, deformations to their muscles have to be simulated. This can be done with a core simulator that runs in real time. However, those are not nearly as good as these ones. Now, these are Absolutely amazing. However, these simulations take forever, and I really mean forever. But these simulations are measured not in frames per second, but in seconds per frame. And it gets even worse, sometimes even minutes per frame. This is the price to be paid for these super accurate simulations. Until now. You see, these simulations can also be done super quickly with learning based methods. And now these also run in real time. Just look at how realistic this movement is. And this will run on our computers and later even on our phones. And get this, this also applies for garment simulations as well. I have highlighted the relevant regions where the new neural network based technique generates these really cool folds on the cloth. Wow. And as you see, we are living incredible times and. And this will put gaming, animation, movies, virtual worlds, and perhaps even conference calls to the next level.
Weights and Biases provides tools to track your experiments in your deep learning projects. Using their system, you can create beautiful reports like this one to explain your findings to your colleagues better. It is used by many prestigious labs including OpenAI, Toyota Research, GitHub and more. And the best part is that Weights and Biases is free for all individuals, academics and open source projects. Make sure to Visit them through wnb.com papers or just click the link in the video description and you can get a free demo today. Our thanks to Weights and Biases for their long standing support and for helping us make better videos for you.
Thanks for watching and for your generous support and I'll see you next time.
