Researchers figured out how to apply neural style transfer to 3D smoke simulations by working on downsampled density fields then upsampling them. The method takes 10 to 15 minutes per frame and smooths velocity changes over time so the animation stays believable, even when you add fire textures or artistic styles.
This episode has been supported by Lamda. Dear fellow scholars, this is two Minute Papers with zsolnai. I can confidently say that this is the most excited I've been for a smoke simulation paper since Wavelet Turbulence. Wavelet Turbulence is a magical algorithm from 2008 that takes a low quality fluid or smoke simulation and increases its quality by filling in the remaining details. And here we are 11 years later, the results still hold up. Insanity. This is one of the best papers ever written and has significantly contributed to my decision to pursue a research career. And this new work performs style transfer for Smoke simulations. If you haven't fallen out of your chair yet, let me try to explain why this is amazing. Style transfer is a technique in machine learning research where we have two input images, one for content and one for style, and the output is our content image reimagined with this new style. The cool part is that the content can be a photo straight from our camera and the style can be a painting, which leads to the super fun results that you see here. An earlier paper had shown that the more sophisticated ones can make even art curators think that they are real. However, doing this for smoke simulations is a big departure from 2D style transfer because that one takes an image image where this works in 3D and does not deal with images, but with density fields. A density field means a collection of numbers that describe how dense a smoke plume is at a given spatial position. It is a physical description of a smoke plume, if you will. So how could we possibly apply artistic style from an image to a collection of densities? This doesn't sound possible at all. Unfortunately, the problem gets even worse since we typically don't just want to look at a still image of a smoke plume, but enjoy a physically correct simulation. Not only the density fields, but the velocity fields and the forces that animate them over time also have to be stylized. Again, that's either impossible or almost impossible to do. You see, if we run a proper smoke simulation, we will see what would happen in reality, but that's not stylized. However, if we stylize, we get something that would not happen in Mother Nature. I have spent my master's thesis trying to solve a problem called fluid control, which would try to coerce a smoke plume or a piece of fluid to take a given shape, like a bunny or a logo with letters. You can see some footage of what I came up with here. Here, both the simulation and the controlling force field is computed in real time on the graphics card, and as you see, it can be combined with wavelet turbulence. If you wish to hear more about this work, make sure to leave a comment. But in any case, I had a wonderful time working on it. If anyone wants to pick it up, the paper and the source code and even a blender add on version are available in the video Description in any case, in a physics simulation we are trying to simulate reality and for style transfer we are trying to depart from reality. The two are fundamentally incompatible and we have to reconcile them in a way that is somehow still believable. Super challenging. However, back then when I wrote the fluid control paper, learning based algorithms were not nearly as developed. So it turns out they can help us perform style transfer for density fields and and also animate them properly. Again, the problem definition is very easy. In comes a smoke plume, we add an image for style and the style of this image is somehow applied to the density field to get these incredible effects. Just look at these marvelous results. Fire textures, starry night, you name it, it seems to be able to do anything. One of the key ideas is that even though style transfer is challenging on highly detailed density fields, it becomes much easier if we first downsample the density field to a coarser version, perform the style transfer there, and up sample this density field again with already existing techniques. Rinse and repeat. The paper also describes a smoothing technique that ensures that the changes in the velocity fields that guide our density fields change slowly over time to keep the animation believable. There are a lot more new ideas in the paper, so make sure to have a look. It also takes a while. The computation time is typically around 10 to 15 minutes per frame, but who cares? Today, with the ingenuity of research scientists and the power of machine learning algorithms, even the impossible seems possible. If it takes 15 minutes per frame, so be it. What a time to be alive. If you are a researcher or a startup looking for cheap GPU compute to run these algorithms, you can check out Lambda GPU Cloud. I've talked about Lambda's GPU workstations in other videos and am happy to tell you that they are offering GPU cloud services as well. The Lambda GPU Cloud can train ImageNet to 93% accuracy for less than $19. Lambda's web based IDE lets you easily access your instance right in your browser and finally hold onto your papers because the Lambda GPU cloud could costs less than half of AWS and Azure. Make sure to go to lambdalabs.compapers and sign up for one of their amazing GPU instances today. Thanks for watching and for your generous support and I'll see you next time.
