This content was created by LARRUZ. This content was uploaded by LARRUZ. This is a YouTube video titled "This AI Creates A Moving Digital Avatar Of You" (published 2019-12-11). A single host/narrator (the Two Minute Papers-style presenter) explains a research paper on reconstructing 3D volumetric data (hair, skin, garments, smoke, toys) from input images or video using ray marching and learned foreground/background separation. The narration covers the method, ablation tests showing the importance of background learning, animation of the reconstructed latent space (real-time avatar control with a magenta dot), current limitations (resolution, missing fine detail), and a call-to-action to subscribe. The episode also includes sponsorship messaging for Lambda (Lambda GPU Cloud) and advertising details for researchers/startups needing GPU compute.
Chapters
This episode has been supported by Lamda. Dear fellow scholars, this is two minute papers with. In this series, we talk about research on all kinds of physics simulations, including fluids, collision physics, and we have even ventured into hair simulations. We mostly talk about how the individual hair strands should move and how they should look in terms of color and reflectance. Creating these beautiful videos takes getting many, many moving parts, right. For instance, before all of that, the very first step is not any of those steps. First we have to get the 3D geometry of these hairstyles into our simulation system. In a previous episode, we have seen an excellent work that does this well for human hair. But what if we would like to model not human hair, but something completely different? Well, hold on to your papers because this new work is so general that it can look at an input image or video and give us not only a model of the human hair, but human skin garments, and of course, my favorite, smoke plumes and more. But if you look here, this part begs the following question. The input is an image and the output also looks like an image. And. And we need to make them similar. So what's the big deal here? A copying machine can do that, no? Well, not really. Here's why. To create the output, we are working with something that indeed looks like an image, but it is not an image. It is a three dimensional cube in which we have to specify color and opacity values everywhere. After that, we simulate rays of light passing through this volume, which is a technique that we call ray marching. And this process has to produce the same 2D image through Ray marching as what was given as an input. That's much, much harder than building a copying machine, as you see here. Normally this does not work well at all because for instance, a standard algorithm sees lights in the background and assumes that these are really bright and dense points. That is kind of true, but they are usually not even part of the data that we would like to reconstruct. To solve this issue, the authors propose learning to tell the foreground and background images apart so they can be separated before we start the reconstruction of the human. And this is a good research paper, which means that if it contains multiple new techniques, each of them are tested separately to know how much they contribute to the final results. We get the previously seen dreadful results without the background separation step. Here are the results with the learned backgrounds. We can still see the lights due to the way the final image is constructed. And the fact that we have so little of this halo effect is really cool. Here you see the results with the true background data where the background learning step is not present. Note that this is cheating because this data is not available for all cameras and backgrounds. However, it is a great way to test the quality of this learning step. The comparison of the learned method against this reveals that the two are very close, which is exactly what we are looking for. And finally the input footage is also shown for reference. This is ultimately what we are trying to achieve and as you see, the output is quite close to it. The final algorithm excels at reconstructing volume data for toys, smoke plumes and humans alike. And the coolest part is that it works for not only stationary inputs, but but for animations as well. Wait. Actually there is something that is perhaps even cooler. With the magic of neural networks and latent spaces, we can even animate this data. Here you see an example of that where an avatar is animated in real time by moving around this magenta dot. A limiting factor here is the resolution of this reconstruction. If you look closely you can see that some fine details are missing. But you know the saying, given the rate of progress in machine learning research, two more papers down the line and this will likely be orders of magnitude better. And if you feel that you always need to take your daily dose of papers, my statistics show that many of you are subscribed but didn't use the bell icon. If you click this bell icon, you will never miss a future episode and can properly engage in your paper addiction. This episode has been supported by Lamda. If you are a researcher or a startup looking for cheap GPU compute to run these algorithms, check out Lamda GPU Cloud. I've talked about Lambda's GPU workstations in other videos and am happy to tell you that they are offering GPU cloud services as well. The Lambda GPU Cloud can train ImageNet to 93% accuracy for less than $19. Lambda's web based IDE lets you easily access your instance right in your browser. And finally, hold on to your papers because the Lambda GPU Cloud costs less than half of AWS and Azure. Make sure to go to lambdalabs.compapers or click the link in the video description and sign up for one of their amazing GPU instances today. Our thanks to Lambda for helping us make better videos for you. Thanks for watching and for your generous support and I'll see you next time.
