Yuanming Hu's research makes physics simulations differentiable, letting neural networks control them directly. The paper shows a simulated agent learning to walk, fluids shaped into a Yin-Yang symbol, and ripples designed to fool a neural network. Source code is available.
Dear fellow scholars, this is two minute papers with. A few episodes ago, we discussed a new research work that performs something that they call differentiable rendering. The problem formulation is the we specify a target image that is either rendered by a computer program or even better, a photo. The input is a pitiful approximation of it. And now, because it progressively changed the input materials, textures and even the geometry of this input in a 3D modeler system, it is able to match this photo. At the end of the video, I noted that I am really looking forward for more differentiable rendering and differentiable everything papers. So fortunately, here we go. This new paper introduces differentiable programming for physical simulations. So what does that mean exactly? Let's look at a few examples and find out together. Imagine that we have this billiard game where we would like to hit the white ball with just the right amount of force and from the right direction, such that the blue ball ends up close to the black spot. Let's try it. Well, this example shows that this doesn't happen by chance, and we have to engage in a fair amount of trial and error to make this happen. What this differentiable programming system does for us is that we can specify an end state, which is the blue ball on the black dot, and it is able to compute the required forces and angles to make this happen. Very close. But the key point here is that this system is general and therefore can be applied to many, many more problems. We'll have a look at a few that are much more challenging than this example. For instance, it can also teach this GUI object to actuate itself in a way so that it would start to walk properly within only two minutes. The 3D version of this simulation learned so robustly, so that it can even withstand a few extra particles in the way. The next example is going to be obscenely powerful. I try to explain what this is to make sure that we can properly appreciate it. Many years ago, I was trying to solve a problem called fluid control where we would try to coerce a smoke plume or a piece of fluid to take a given shape, like a bunny or a logo with letters. You can see some footage of this project here. The key difficulty of this problem is that this is not what typically happens in reality. Of course, a glass of spilled water is very unlikely to suddenly take the shape of a human face. So we have to introduce changes to the simulation itself. But at the same time, it still has to look as if it could happen in nature. If you wish to know more about my work here, the full thesis and the source code is available in the video description and one of my kind students has even implemented it in Blender. So this problem is obscenely difficult and you can now guess what's next for this differentiable technique. Fluid Control. It starts out with a piece of simulated ink with a checkerboard pattern and it exerts just the appropriate forces so that it forms exactly the Yin Yang symbol shortly after. I am shocked by how such a general system can perform such something of this complexity. Having worked on this problem for a while, I can tell you that this is immensely difficult. Amazing. And hold on to your papers because it can do even more. In this example, it adds carefully crafted ripples to the water to make sure that it ends up in a state that distorts the image of the squirrel in a way that a powerful and well known neural network sees it not as a squirrel but but as a goldfish. This thing is basically a victory lap in the paper. It is so powerful it's not even funny. You can just make up some problems that sound completely impossible and it rips right through them. The full source code of this work is also available. By the way, the first author of this paper is Yuan Ming Hu. His work was showcased several times in this series. We talked about his amazing jello simulation that was implemented in so few lines of code and it almost fits on a business card. I said it in a previous episode and I will say it again. I can't wait to see more and more papers in differentiable rendering and simulations. And as this work leaves plenty of room for creativity for novel problem definitions, I'd love to hear what you think about it. What else could this be used for? Solving video games faster than other learning based techniques. Anything else, let me know in the comments below. What a time to be alive. This episode has been supported by Weights and Biases. Weights and Biases provides tools to track your experiments in your deep learning projects. It can save you a ton of time and money in these projects and is being used by OpenAI, Toyota Research, Stanford and Berkeley. It is really easy to set up, so much so that they have made an instrumentation for this exact paper we have talked about in this episode. Have a look here. Make sure to Visit them through wandb.com papers wandb.com papers or just click the link in the video description and you can get a free demo today. Our thanks to WeightsAndBiases for helping us make better videos for you. Thanks for watching and for your generous support and I'll see you next time.
