Dr. Zsolnai reviews a 2019 paper on capturing photorealistic digital humans using a rig with hundreds of LEDs and depth sensors. The method models how light reflects off skin, hair, and clothing, then reproduces fine details and motion more realistically than earlier techniques.
Dear fellow scholars, this is Two Minute Papers with Dr. Zsolnai. Today we are going to look at a paper with two twists. You know what? I'll give you twist number one right away. This human isn't here. This human isn't here either. And neither is this human here. Now you're probably asking, what are you even talking about? Now hold on to your papers because I am talking about this. Look. This is the geometry of this virtual human inside a virtual world. Whoa. Yes. All of these people are in a synthetic video and in a virtual environment that can be changed with a simple click. And more importantly, as we change the lighting or the environment, it also simulates the effect of that environment on the character, making it look like they are really there. So that sounds good, but how do we take a human and make a digital copy of them? Well, first we place them in a capture system that contains hundreds of LED lights and an elaborate sensor for capturing depth information. Why do we need these? Well, all this gives the system plenty of data on how the skin, hair and the clothes reflect light. And at this point, we know everything we need to know and can now proceed and place our virtual copy in a computer game or even a telepresence meeting. Now this is already amazing, but two things really stick out here. One, you will see when you look at this previous competing work. This had a really smooth output geometry, which means that only few high frequency details were retained. This other work was better at retaining the details. But look, tons of artifacts appear when the model is moving. And what does the new one look like? Is it any better? Let's have a look. Oh my. We get tons of fine details and the movements have improved significantly. Not perfect by any means. Look here. But still, this is an amazing leap forward. Two, the other remarkable thing here is that the results are so realistic that objects in the virtual scene can cast a shadow on our model. What a time to be alive. And now. Yes, you remember that I promised two twists. So where is twist number two? Well, it has been here all along for the entirety of this video. Have you noticed? Look, all this is from 2019, from two years ago. Two years is a long time in machine learning and computer graphics research and I cannot wait to see how it will be improved. Two more papers down the line. If you are excited too, make sure to subscribe and hit the bell icon to not miss it when it appears. This video has been supported by weights and biases. They have an amazing podcast by the name Gradient Descent where they interview machine learning experts who discuss how they use learning based algorithms to solve real world problems. They've discussed biology, teaching robots, machine learning in outer space, and a whole lot more. Perfect for a fellow scholar with an open mind. Make sure to visit them through WNB MEGD or just click the link in the video description. Our thanks to Weights and Biases for their long standing support and for helping us make better videos for you. Thanks for watching and for your generous support and I'll see you next time.
