NVIDIA's Audio2Face maps voice recordings to animated characters with emotion control and face-swapping. The tool runs on modest hardware, works with Unreal Engine, and can transform a human face into a rhino or any other character.
Dear fellow scholars, this is Two Minute Papers with Dr. Zsolnai.
Today, through the power of AI research, we are going to see how easily we can ask virtual characters to not only say what we say, but we can even become an art director and ask them to add some emotion to it.
You and your men will do that. You have to go in and out very quick. Are those Eurasian footwear? Cowboy chaps or jolly earthmoving headgear?
Probably the most important thing we need to do is to bring the country together. And one of the skills that I bring to bear.
So how does this happen? Well, in goes what we say, for instance, me uttering dear fellow scholars, or anything else. And here's the key. We can also specify the emotional state of the character, and the AI does the rest. That is absolutely amazing. But it gets even more amazing. Now hold on to your papers and look. Yes, that's right. This was possible back in 2017, approximately 402 minute papers episodes ago. And whenever I showcase results like this, I always get the question from you fellow scholars asking, yes, this all looks great, but when do I get to use this? And the answer is right now.
Why?
Because Nvidia has released Audio2Face, a collection of AI techniques that we can use to perform this quickly and easily. Look, we can record our voice live and have a virtual character say what we are saying. But if it doesn't stop there, it also has three amazing features. One, we can even perform a face swap not only between humanoid characters, but my goodness, even from a humanoid to for instance, a rhino. Now that's something. I love it. But wait, there is more. There is this. And this too. Too. We can still specify emotions like anger, sadness and excitement. And the virtual character will perform that for us. We only need to provide our voice. No more acting skills required. In my opinion, this will be a godsend in any kind of digital media, computer games, or even when meeting our friends in a virtual space. Three, the usability of this technique is out of this world. For instance, it does not eat a great deal of resources, so we can easily run multiple instances of it at the same time. This is a wonderful usability feature, one of many that really makes or breaks when it comes to a new technique being used in the industry or not. An aspect not to be underestimated. And here is another usability feature. It works well with Unreal Engine's Metahuman. This is a piece of software that can create virtual humans. And with that, we can not only create these virtual humans, but become the voice actors for them without having to hire a bunch of animators. How cool is that? Now I believe this is an earlier version of Metahuman. Here is the newer one. Wow. Way better. Just imagine how cool it would be to voice these characters automatically. Now the important lesson is that this was possible in a paper in 2017 and now in a few years it has vastly improved. So much so that it is now out there deployed in a real product that we can use right now that is a powerful democratizing force for computer animation. So yes, the papers that you see here are real, as real as it gets. And this tech transfer can often occur in just a few years time, in some other cases even quicker. What a time to be alive. So what would you use this for? I'd love to know what you think, so please let me know in the comments below. This episode has been supported by Cohere AI.
Cohere builds large language models and makes them available through an API so businesses can add advanced language understanding to their system or app quickly. With just one line of code, you can use your own data, whether it's text from customer service requests, legal contracts, or social media posts, to create your own custom models to understand text or even generate it. For instance, it can be used to automatically determine whether your messages are about your business hours, returns, or shipping. Or it can be used to generate a list of possible sentences you can use for your product descriptions. Make sure to go to Cohere AI papers or click the link in the video description and give it a try. Today, it's super easy to use.
Thanks for watching and for your generous support, and I'll see you next time.
