A machine-learning method breaks paintings into brushstrokes and relights them from a single image. The technique works on photos too, generates results in seconds, and users strongly prefer it to prior methods.
Dear fellow scholars, this is Two Minute Papers with Dr. Zsolnai. When I was a bachelor student and took on my first bigger undertaking in computer graphics in 2011, it was a research project for a feature length movie where the goal was to be able to learn the brushstrokes of an artist. You see the sample brushstroke here? And what it could do is change the silhouette of a digital 3D object to appear and as if it were drawn with this style. This way we could use an immense amount of perfectly modeled geometry and make them look as if they were drawn by an artist. The project was a combination of machine learning and computer graphics and got me hooked on this topic for life. So this was about silhouettes, but what about being able to change the lighting to address this problem? This new work promises something that sounds like science fiction. The input is a painting which is thought of as a collection of brush strokes. First, the algorithm is trying to break down the image into these individual strokes. Here on the left with A, you see the painting itself and B is the real collection of strokes that were used to create it. This is what the algorithm is trying to estimate it with. And this colorful image visualizes the difference between the two. The blue color denotes regions where these brushstrokes are estimated well, and we can find more differences as we transition into the red colored regions. So great now that we have a bunch of these brushstrokes, but what do we do with them? Well, let's add one more assumption into this system, which is that the densely packed regions are going to be more affected by the lighting effects, while the sparser regions will be less impacted. This way we can make the painting change as if we were to move our imaginary light source around, no painting skills or manual labor required. Wonderful. But some of the skeptical fellow scholars out there would immediately ask the it looks great, but how do we know if this really is good enough to be used in practice? The authors thought of that too, and asked an artist to create some of these views by hand. And what do you know? They are extremely good, very close to the real deal. And all this comes for free. Insanity. Now, we noted that the input for this algorithm is just one image. So what about a cheeky experiment where we would add not a painting but a photo and pretend that it is a painting? Can it relight it properly? Well, hold on to your papers and let's have a look. Here is the photo, the breakdown of the brush strokes. If this were a painting and wow, here are the lighting effects. It worked. And if you enjoyed these results and would like to see more? Make sure to have a look at this beautiful paper in the video description. For instance, here you see a comparison against previous works and it seems quite clear that it smokes the competition on a variety of test cases. And these papers they are comparing to are also quite recent. The pace of progress in computer graphics research is absolutely incredible. More on that in a moment. Also, just look at the information density here. This tiny diagram shows you exactly where the light source positions are. I remember looking at a paper on a similar topic that did not have this thing and it made the entirety of the work a great deal more challenging to evaluate properly. This kind of attention to detail might seem like a small thing, but it makes all the difference for a great paper which this one is. The provided user study shows that these outputs can be generated within a matter of seconds and reinforces our hunch that most people prefer the outputs of the new technique to the previous ones. So much improvement in so little time. With this we can now create digital lighting effects from a single image for paintings and even photographs in a matter of seconds. What a time to be alive. What you see here is an instrumentation of this exact paper we have talked about, which was made by Weights and Biases. I think organizing these experiments really showcases the usability of their system. Weights and Biases provides tools to track your experiments in your deep learning projects. Their system is designed to save you a ton of time and money and it is actively used in projects at prestigious labs such as OpenAI, Toyota Research, GitHub and more. And the best part is that if you're an academic or have an open source project, you can use their tools for free. It really is as good as it gets, but make sure to Visit them through wnb.com papers or just click the link in the video description and you can get a free demo today. Our thanks to Weights and Biases for their long standing support and for helping us make better videos for you. Thanks for watching and for your generous support and I'll see you next time.
