Ubisoft's new AI generates realistic gestures for game characters from speech. The model works seven times faster than previous methods, learns from just two hours of gesture data, and handles unseen speakers and languages without retraining.
November 26, 2022.
About two hours of gesture data.
About seven times faster.
Weights and Biases.
Dear fellow scholars, this is two minute papers with Dr. Zsolnai. Earlier we talked about AI based techniques that can learn to clone your voice. And then we can perform text to speech. So it would learn this.
I think they have to change that. Further details are expected later.
And then the AI would generate this.
This is a voice line generated by an AI. The first law of papers says that research is a process.
Yes. This means that we can write something and the AI says it in our voice. So that is text to speech. But get this. What about speech to gesture? What is that? Well, scientists at Ubisoft had a crazy idea. They said, let's create a dataset of characters where the AI can see gestures for agreement, disagreement, being relaxed, the neutral, scared and more. Learn from it and apply it to new virtual characters intelligently. Sounds good, right? But not so fast. Ubisoft is not the first to try this idea. Here's a technique from just two years ago.
Look, I check my phone all the time. I hope that somebody has reached out.
There is something here. But we are humans, at least most of us anyway, and our eyes are highly attuned to the gestures of other humans. Which means that if even the smallest part of the animation is off, we will immediately notice. And I think every single one of you fellow scholars indeed noticed that something is off here. And based on the quality of the animation that is required today to keep the illusion up. And I am not sure if this is coming to fruition anytime soon. Just look at these examples. We are so far away. But in any case, let's have a look at the new technique together. The concept is the same. First in goes a speech sample. And it will be able to generate long sequences using a style of our.
Choosing, like society has left behind. What? Nobody listens to you anymore, Nobody thinks about you anymore. And yet it is you who populate our shopping malls. It is you.
That looks incredible. Let's compare it to the previous method.
I check it. I check my phone all the time. I hope that somebody has reached out.
Yes. My goodness. A night and day difference. Such incredible progress in just two years. Yes, this is from only two years ago. And that would already be pretty cool. But it gets crazier than that. Much crazier. For instance, we can plug in a new speaker the AI hasn't heard about yet. And this happens.
Biggest fake smile to match his. Asked him how his day was. His answer, perfect.
This is really good. But that's still nothing. It generalized to not only new speakers, but hold on to your papers. To new languages too. Listen.
Wow.
And the list of features just keeps on going. For instance, we can also exert some artistic control here. If we feel that the hands are too high up, we can lower them or we can even ask for more or less hip movement during the monologue. And I have to say, it does a lot of things really well, from the lowest energy gestures up to the highest, and it still doesn't stop there. It is not only better than the previous technique, is not only more controllable than the previous technique, but but it is also about seven times faster. And even better, it is done with a neural network that is significantly simpler than that one. So how many hours of these gestures did the neural network get access to to learn this? How big was the training set size? Actually it requires very little information. All it was given was about two hours of these gestures. And it is not copy pasting, but it can continue these gestures for a long, long monologue seamlessly. And as a cherry on top, even some creative control is allowed. So these amazing virtual worlds are going to be even more full of lifelike characters. And with this work, this process is going to be even easier and more accessible for all of us.
What a time to be alive. Weights and Biases provides tools to track your experiments in your deep learning projects. What you see here is their tables feature. And the best part about it is that it is not only able to handle pretty much any kind of data you can throw at it, but it also presents your experiments to you in a way that is easy to understand. It is used by many prestigious labs including OpenAI to Toyota Research, GitHub and more. And the best part is that Weights and Biases is free for all individuals, academics and open source projects. Make sure to Visit them through wnb.com papers or just click the link in the video description and you can get a free demo today. Our thanks to Weights and Biases for their long standing support and for helping us make better videos for you.
Thanks for watching and for your generous support and I'll see you next time.
