Dr. Zsolnai explains a new technique that turns 2D sketches into animatable 3D models in near real-time, skipping traditional modeling and rigging. The method accepts SIGGRAPH Asia and has source code available, though it struggles with depth, proportions, and hidden textures.
Dear fellow scholars, this is two minute papers with Dr. Zsolnai. If we wish to create an adorable virtual monster and animate it, we first have to engage in 3D modeling. Then, if we wish to make it move and make this movement believable, we have to specify where the bones and joints are located within the model. This process is called rigging. As you see, it is quite laborious and requires expertise in this domain to pull this off. And now imagine this process with no 3D modeling and no rigging. This is of course, impossible. Right? Well, now you know what's coming, so hold on to your papers, because here is a newer technique that indeed performs the impossible. All we need to do is grab a pencil and create a rough sketch of our character. Then it will take a big breath and inflate it into a 3D model. This process was nearly seven times faster than the classical workflow. But what matters, even this new workflow requires zero expertise in 3D modeling and rigging. This means that with this technique, absolutely anybody can become an artist. So we noted that these models can also be animated. Is that so? Yes, that's right. We can indeed animate these models by using these red control points. And even better, we get to specify where these points go. That's a good thing, because we can make sure that a prescribed part can move around, opening up the possibility of creating and animating a wide range of characters. And I would say all this can be done in a matter of minutes, but it's even better sometimes even within a minute. Whoa. This new technique does a lot of legwork that previous methods were not able to pull off so well. For instance, it takes a little information about which part is in front or behind the model. Then it stitches all of these strokes together and inflates our drawing into a 3D model. And it does this better than previous methods. Look. Well, ok, the new one looks a bit better. Where the body parts connect here. And that's it. Wait a second. Aha. Somebody didn't do their job correctly. And we went from this work to this in just two years. This progress is absolute insanity. Now, let's have a look at a full workflow from start to end. First, we draw the strokes. Note that we can specify that one arm and leg is in front of the body and the other one is behind. And BAM. The 3D model is now done. Wow, that was quick. And now add the little red control points for animation. And let the fun begin. Mr. Your paper has been officially accepted. Move the feet, pin the hands, rock the body. Wait. Not only that, but this paper was accepted to the Siggraph Asia Conference, which is equivalent to winning an Olympic gold medal in computer graphics research, if you will. So add a little neck movement too. Oh yeah, now we're talking. With this technique, the possibilities really feel endless. We can animate humanoids, monsters, other adorable creatures, or can even make scientific illustrations come to life without any modeling and rigging expertise. Do you remember this earlier video where we could paint on a piece of 3D geometry and transfer its properties onto a 3D model? This method can be combined with that too. Yum. And in case you're wondering how quick this combination is, my goodness. Very very quick. Now this technique is also not perfect. One of the limitations of this single view drawing workflow is is that we only have limited control over the proportions in depth. Texturing occluded regions is also not that easy. The authors proposed possible solutions to these limitations in the paper, so make sure to have a look in the video description and it appears to me that with a little polishing this may be ready to go for artistic projects right now. If you have a closer look, you will also see that this work also cites the flow paper from Mihaly Csikszentmihalyi Extra Star points for that. And with that said, when can we use this? And here comes the best part. Right now the authors really put their papers where their mouth is. Or in other words, the source code for this project is available. Also, there is an online demo. Woohoo. The link is available in the video description. Make sure to read the instructions before you start. So there you go. Instant 3D models with animation without requiring 3D modeling and rigging expertise. What do you think? Let me know in the comments below. This episode has been supported by Weights and Biases. In this post they show you how to use transformers from the Hugging Face Library and how to track your model performance. During my PhD studies I trained a ton of neural networks which were used in our experiments. However, over time there was just too much data in our repositories and what I am looking for is not data but insight. And that's exactly how Weights and Biases helps you by organizing your experiments. It is used by more than 200 companies and research institutions including OpenAI, Toyota Research, GitHub and more. And get this Weights and Biases is free for all individuals, academics and open source projects. Make sure to Visit them through wnb.com papers or just click the link in the video description and you can get a free demo today. Our thanks to WeightsAndBiases for their long standing support and for helping us make better videos for you. Thanks for watching and for your generous support and I'll see you next time.
