Károly Zsolnai-Fehér explains a new method for applying artistic styles to 3D objects. It's 1000 times faster than previous approaches, hitting 100 FPS at 4K, and handles animation better. Brick patterns and regular textures still cause alignment problems.
Dear fellow scholars, this is Two Minute.
Papers with Style Transfer is an interesting problem in machine learning research where we have two input images, one for content and one for style, and the output is our content image reimagined with this new style. The cool part is that the content can be a photo straight from our camera and the style can be a.
Painting, which leads to super fun and really good looking results.
We have seen plenty of papers doing.
Variations on style transfer, but can we push this concept further?
And the answer is yes.
For instance, few people know that style.
Transfer can also be done in 3D. If you look here, you see an.
Artist performing this style transfer by drawing.
On a simple sphere and get their.
Artistic style to carry over to a.
Complicated piece of 3D geometry.
We talked about this technique in 2 minute papers episode 94 and for your reference we are currently at over episode 340. Leave a Comment if you've been around back then. And this previous technique led to truly.
Amazing results, but still had two weak points.
One, it took too long.
As you see here, this method took.
Around a minute or more to produce these results. And hold on to your papers because.
This new paper is approximately 1000 times.
Faster than that, which means that it can produce 100 frames per second using.
A whopping 4K resolution. But of course none of this matters if the visual quality is not similar. And if you look closely, you see.
That the new results are indeed really.
Close to the reference results of the older method. So what was the other problem? The other problem was the lack of temporal coherence.
This means that when creating an animation.
It seems like each of the individual.
Frames of the animation were drawn separately by an artist. In this new work, this is not.
Only eliminated, as you see here, but.
The new technique even gives us the.
Opportunity to control the amount of flickering.
With these improvements, this is now a.
Proper tool to help artists perform this 3D style transfer and and create these rich virtual worlds much quicker and easier in the future.
It also opens up the possibility for.
Novices to do that, which is an amazing value proposition.
Limitations still apply.
For instance, if we have a texture with some regularity, such as this brick wall pattern here, the alignment and continuity of the bricks on the 3D model may suffer.
This can be fixed, but it is.
A little labor intensive. However, you know are saying two more papers down the line and this will likely cease to be an issue.
And what you've seen here today is.
Just one paper down the line from.
The original work and we can do.
4K resolution at 100 frames per second.
Unreal.
Thanks for watching and for your generous.
Support, and I'll see you next time.
