Károly Zsolnai-Fehér explains how to edit images and videos by reconstructing 3D models from selected objects, then applying physics simulations to change gravity, mass, or stiffness. You can animate a melting watch in a Dali painting, add wind to a clothesline, or bend a bridge by adjusting its mass.
Dear fellow scholars, this is Two Minute Papers Withehir. This work is about changing already existing images and videos by adding new objects to them or editing their physical attributes. These editable attributes include gravity, mass, stiffness, or we can even add new physical forces to the scene. For instance, we can change the stiffness of the watches in this Dali painting and create an animation from it. Physically accurate animations from paintings. How cool is that? This is approaching science fiction levels of craziness. Animating a stationary clothesline by adding a virtual wind effect to the scene or bending a bridge by changing its mass is also a possibility. The first reaction I had when I've looked at this work was are you kidding me? You can't edit a photograph. Especially that. I've seen plenty of earlier works that tried to do something similar, but each time the limitations were just too crippling for real world use usage. And the ultimate question is how much user interaction does this need? Is this trivial to use or is it a laborious process? What we need to do is roughly highlight the outline of the object that we'd like to manipulate. The algorithm uses a previously published technique to make sure that the outlines are accurately captured and then tries to create a 3D digital model from the selected area. We need one more step where we align the 3D model to the image or video input. Finally, the attribute changes and edits take place not on the video footage, but on this 3D model through a physics simulation technique. A truly refreshing combination of old and new techniques with some killer applications. Loving it. The biggest challenge is to make sure that the geometry and the visual consistency of the scene is preserved through these changes. There are plenty of details discussed in the paper. Make sure to have a look at that. The link to it is available in the video. Description as these 2D photo to 3D model generator algorithms improve, so will the quality of these editing techniques in the near future. Our previous episode was on this topic. Make sure to have a look at that. Also, if you would like to get more updates on the newest and coolest works in this rapidly improving field, make sure to subscribe and click the Bell icon to be notified when new two Minute Papers videos come up. Thanks for watching and for your generous support. Na See you next time.
