Károly Zsolnai-Fehér describes a method that generates animations from sound by matching audio to a database of rigid-body simulations. The system runs on a graph encoding event sequences without machine learning, but works only on flat surfaces and takes several minutes to process.
Dear fellow scholars, this is two minute papers with. This is going to be absolutely amazing. Earlier, we had some delightful discussions on synthesizing sound from animations. The input would be a sequence, for instance, a video depicting the complete and utter destruction of plates, wooden bunnies, or footage of bubbling water. And the output should be a physical simulation that yields appropriate sound effects for the observed phenomenon. In short, input animation. Output synthesized sound effects for this animation. And get this. What if we would turn the problem around where we have sound as an input and we try to synthesize an animation that could create such a sound? Hmm, I like it. Very spicy project indeed. And however crazy the idea may sound, given the richness of sound effects in nature, it may actually be easier to generate a believable animation than a perfect sound effect. It is extremely difficult to be able to match the amazingly detailed real world sound of, for instance, a sliding, rolling or bouncing bolt with a simulation. The more I think about this, the more I realize that this direction actually makes perfect sense. And no machine learning is used here. If we look under the hood, we'll see a pre generated database of rigidbody simulations with dozens of different objects and a big graph that tries to group different events and motions together and encode the order of execution of these events and motions. Now hold on to your papers and let's check out the first round of results together. Sa. Wow. I think it would be an understatement to say that they nailed it. And what's more, we can also add additional constraints like a prescribed landing location to the object to make sure that the animations are not too arbitrary, but are more in line with our artistic vision. Crazy. As I am looking through the results, I am still in complete disbelief. This shouldn't be possible. Also, please don't get the impression that this is all there is to this technique. There are a lot more important details that we haven't discussed here that the more curious fellow scholars could be interested. Discrete and continuous time contact events, time warping and motion connections. There are tons of goodies like these in the paper. Please have a look. To be able to better gauge and appreciate the merits of this work, limitations apply such as the environment is constrained to be this plane that we've seen in these animations. And as always with works that are inventing something completely new, it currently takes several minutes, which is not too bad. But of course there's plenty of room to accelerate the execution times. And some bonus footage.
Blue dice are synchronized with hi hat hits and the red dice are synchronized with the base and snare.
This will be just spectacular for creating music videos, animated movies, and I'm convinced that professional artists will be able to do incredible things with such a tool. Thanks for watching and for your generous support, and I'll see you next time.
