Dr. Károly Zsolnai-Fehér covers new AI that uses 16 cameras to capture and edit dance performances. You can rescale dancers, copy-paste people into scenes, retime movements, and stabilize shaky footage to create synthetic music videos.
September 2, 2021.
Dr. Károly Zsolnai-Fehér.
16 cameras.
Roughly 1 decibel of signal per camera removed.
Dear fellow scholars, this is Two Minute Papers with Dr. Zsolnai. Today we are going to make some crazy synthetic music videos. In machine learning research, view synthesis papers are in the rise these days. These techniques are also referred to as NERF variants, which is a learning based algorithm that tries to reproduce real world scenes from only a few views. It is very challenging. Look in go a bunch of photos of a scene and the method has to be able to synthesize new photorealistic images between these photos. But this is not the only paper in this area. Researchers are very aware of the potential here and thus a great number of NERF variants are appearing every month. For instance, here is a recent one that extends the original technique to handle shape, shiny and reflective objects better. So what else is there to do here? Well, look here, this new one demands not a bunch of photos from just one camera, no, no. But from 16 different cameras. That's a big ask. But in return, the method now has tons of information about the geometry and the movement of these test subjects. So is it intelligent enough to make something useful out of it? Now, believe it or not, this in return can not only help us look around in the scene, but even edit it in three new ways. For instance, one, we can change the scale of these subjects, Add and remove them from the scene and even copy paste them. Excellent for creating music videos. Well, talking about music videos, do you know what is even more excellent for those retiming movements that is also possible. This can for instance improve an okay dancing performance into an excellent one. And three, because now we are in charge of the final footage. If the original footage is shaky, well, we can choose to eliminate that camera shake game changer. It's not quite the hardware requirement where you just whip out your smartphone and start nerfing and editing. But for what it can do, it really does not ask for a lot. Especially given that if we wish to, we can even remove some of these cameras and still expect reasonable results. We lose roughly a decibel of signal per camera. Here is what that looks like. Not too shabby. And all this progress. Just one more paper down the line. And I like the idea behind this paper a great deal. Because typically what we are looking for in a follow up paper is trying to achieve similar results while asking for less data from the user. This paper goes into the exact other direction and asks what amazing things could be done if we had more data instead. Loving it. And with that, not only neural view synthesis, but neural scene editing is also possible. What a time to be alive. This video has been supported by weights and biases. Being a machine learning researcher means doing tons of experiments and of course creating tons of data. But I am not looking for data, I am looking for insights. And weights and biases helps with exactly that. They have tools for experiment tracking, dataset and model versioning, and even hyper parameter optimization. No wonder this is the experiment tracking tool choice of OpenAI, Toyota Research, Samsung and many more prestigious labs. Make sure to use the link wnb, me, paperintro or just click the link in the video description and try this 10 minute example of weights and biases today to experience the wonderful feeling of training a neural network and being in control of your experiments. After you try it, you won't want to go back. Thanks for watching and for your generous support and I'll see you next time.
