Dr. Zsolnai explains a new method for applying neural style transfer to fluid simulations using particles instead of density fields. The Lagrangian approach lets you attach different styles to individual plumes and change them over time, and it's dramatically faster: down from a day to about an hour overall, with individual frames rendering in seconds.
Dear fellow scholars, this is two minute papers with Dr. Zsolnai. Style transfer is a technique in machine learning research where we have two input images, one for content and one for style. And the output is our content image reimagined with this new style. The cool part is that the content can be a photo straight from our camera and the style can be a painting, which leads to the super fun results that you see here. An earlier paper had shown that the more sophisticated ones can sometimes make even art curators think that they are real. This previous work blew me away as it could perform style transfer for smoke simulations. I almost fell out of the chair when I have first seen these results. It could do fire textures, starry night, you name it. It seems that it is able to do anything we can think of. Now let me try to explain two things. One, why is this so difficult? And two, the results are really good. So are there any shortcomings? Doing this for smoke simulations is a big departure from 2D style transfer because that takes an image where this works in 3D and does not deal with images, but with density fields. A density field means a collection of numbers that describe how dense a smoke plume is at a given spatial position. It is a physical description of a smoke plume, if you will. So how could we possibly apply artistic style from an image to a collection of densities? The solution in this earlier paper was to first downsample the density field to a coarser version, perform the style transfer there, and upsample this density field. Again with already existing techniques, this technique was called Transport based Neural style Transfer. Tnst in short, please remember this. Now let's look at some results from this technique. This is what our simulation would look like normally. And then all we have to do is show this image to the simulator and what does it do with it? Wow, my goodness, just look at those heavenly patterns. So what does today's new follow up work offer to us that the previous one doesn't? How can this seemingly nearly perfect technique be improved? Well, this new work takes an even more brazen vantage point to this question. If style transfer on density fields is hard, then try a different representation. The title of the paper says Lagrangian Style Neural Style Transfer. So what does that mean? It means particles. This was made for particle based simulations, which comes with several advantages. One, because the styles are now attached to particles, we can choose different styles for different smoke plumes and they will remember what style they are supposed to follow. Because of this advantageous property, we can even ask the particles to change their styles over time, creating these heavenly animations. In these 2D examples you also see how the texture of the simulation evolves over time and that the elements of the style are really propagated to the surface and the style indeed follows how the fluid changes. This is true even if we mix these styles together. 2 It not only provides us these high quality results, but it is fast. And by this I mean blazing fast. You see, we talked about tnst, the transport based technique, approximately seven months ago. And in this series I always note that two more papers down the line and it will be much, much faster. So here's the two minute paper's moment of truth. What do the timing say for the previous technique? It said more than one D. What could that one D mean? Oh goodness, that thing took an entire day to compute. So what about the new one? What? Really? Just one hour? That is insanity. So how detailed of a simulation are we talking about? Let's have a look together. M F means minutes per frame. And as you see, if we have tens of thousands of particles, we have 0.05 or in other words 3 seconds per frame. And we can go up to hundreds of thousands or even millions of particles and end up around 30 seconds per frame. Loving it. Artists are going to do miracles with this technique. I am sure. The next step is likely going to be a real time algorithm which may appear as soon as one or two more works down the line. And you can bet your papers that I'll be here to cover it. So make sure to subscribe and hit the bell icon to not miss it when it appears. The speed of progress in computer graphics research is nothing short of amazing. Also, make sure to have a look at the full paper in the video description. Not only because it is a beautiful paper and also a lot of fun to read, but because you will also know what this regularization step here does exactly to the simulation. This episode has been supported by Weights and Biases. In this post they show you how to connect their system to the Hugging Face Library and how to generate tweets in the style of your favorite people. You can even try an example in an interactive notebook through the link in the video description. Weights and Biases provides tools to track your experiments in your deep learning projects. Their system is designed to save you a ton of time and money and it is actively used in projects at prestigious labs such as OpenAI, Toyota Research, GitHub and more. And the best part is that if you're an academic or have an open source project, you can you can use their tools for free. It really is as good as it gets. Make sure to Visit them through wnb.com papers or just click the link in the video description and you can get a free demo today. Our thanks to WeightsAndBiases for their long standing support and for helping us make better videos for you. Thanks for watching and for your generous support and I'll see you next time.
