Dr. Károly Zsolnai-Fehér explains how differentiable rendering uses AI to reconstruct both the shape and materials of objects from a single photo. Previous methods could recover one or the other, but this technique starts with a sphere and iteratively refines both geometry and materials in less than an hour.
Dear fellow scholars, this is Two Minute Papers with Dr. Zsolnai. Today we are going to learn how to take an object in the real world, make a digital copy of it and place it into our virtual world. This can be done through something that we call differentiable rendering. What is that? Well, simple. We take a photograph and have an AI find a photorealistic material model that we can put into our light simulation program that matches it. What this means is that now we can essentially put this real material into a virtual world. This earlier work did very well with materials, but it did not capture the geometry. Later, this follow up work was looking to improve the geometry side of things. Here, let's try to reproduce a 3D model from a a bunch of triangles. Let's see. And look. This is lovely. Seeing these images gradually morph into the right solution is an absolutely beautiful sight. But of course in this case the materials are gone. And if we wish to focus more on materials and we don't care about the geometry at all, we can use one of our previous papers for that. This one uses a learning algorithm to find out about your artistic vision and recommends really cool materials. Or with this other work, you can even whip up a fake material in Photoshop and it will magically find a digital photorealistic material that matches it. But once again, no geometry materials only. But wait a minute, am I reading this right? With previous method number one, we get the materials but no geometry or previous method number two, no materials, but the geometry is so much more detailed. Or previous method number three, excellent materials, but once again, no geometry. We can't have it all, can we? So is that it? We can't really take a real object and make a virtual copy of it. Is the dream dead? Well, don't despair quite yet. This new technique might just be what we are looking for. Although I am not so sure. This problem is super challenging. Let's try this chair and see what happens. Here are a bunch of images of it. Now your turn little algorithm. And we start out from a sphere. Well, I will believe this when I see it. Good luck. And. Hmm, the shape is starting to change. And. Wait, are you seeing what I am seeing? Yes, yes, yes. The material is also slowly starting to change. We are getting not only geometry but materials too. Now, based on previous works, our expectation is that the result will be extremely coarse. Well, hold on to your papers and let's see how much detail we end up with. Oh my, that is so much better. Loving it. Actually. Let's check. This was the previous technique and this is the new one. My goodness, I have to say there is no contest here. The new one is so much more detailed, so good, and it works on a bunch of other examples too. These are not perfect by any means, but this kind of improvement, just one more paper down the line that is absolutely incredible. The pace of progress in computer graphics research is nothing short of amazing, especially when we look at all the magic scientists are doing in Wenzel Jakob's lab and the whole process took less than an hour. That is a considerable amount of time. But an artist can leave a bunch of these computations to cook overnight and by morning a bunch of already pretty good quality virtual objects will appear that we can place in our virtual worlds. How cool is that? What a time to be alive. Make sure to check out the whole paper. It has crystal clear mathematics and tons of gorgeous images. The link is available in the video description. I believe this will be an excellent tool in democratizing asset creation for games, animation, movies, and all kinds of virtual worlds. What a time to be alive. So what would you use this for? Do you have some cool ideas? Let me know in the comments below.
This episode has been supported by Cohere AI. Cohere builds large language models and makes them available through an API so businesses can add advanced language understanding to their system or app quickly. With just one line of code, you can use your own data, whether it's text from customer service requests, legal contracts, or social media posts, to create your own custom models to understand text or even generate it. For instance, it can be used to automatically determine whether your messages are about your business hours, returns, or shipping. Or it can be used to generate a list of possible sentences you can.
Use for your product descriptions.
Make sure to go to Cohere AI papers or click the link in the video description and give it a try today. It's super easy to use.
Thanks for watching and for your generous support and I'll see you next time.
