A method transfers fine surface details from one 3D model to another using metric learning. It captures statistical properties of displacement maps and synthesizes them on target models while avoiding distortions from projection. The technique helps artists and game developers add geometric detail without manual work.
Dear fellow scholars, this is two minute papers with in the world of digital 3D modeling, it often occurs that we are looking for surfaces that are not perfectly smooth, but have some sort of surface detail. Wrinkles, engravings, grain on a wooden table are excellent examples of details that we can add to our models and computer graphics. People like to collectively call these things displacement maps. Artists often encounter cases where they like the displacements on one object, but the object itself is not really interesting. However, it could be that there is a different piece of geometry these details would look great on. Consider this problem solved because in this piece of work, the input is two 3D models, one with interesting geometric details and the other is the model onto which we transfer these surface details. The output will be our 3D geometric shape. With two of these models fused together, the results look absolutely amazing. I would love to use this right away in several projects. The first key part is the usage of metric learning. Wait, technical term, what does this mean exactly? Metric learning is a classical technique in the field of machine learning where we are trying to learn distances between things where distance is mathematically ill defined. Let's make it even simpler and go with an example. For instance, we have a database of human faces and we would like to search for faces that are similar to a given input. To do this, we specify a few distances by hand. For instance, we could say that a person with a beard is a short distance from one with a mustache and a larger distance from one with no facial hair. If we hand many examples of these distances to a learning algorithm, it will be able to find people with similar beards. And and in this work, this metric learning is used to learn the relationship between objects with and without these rich surface details. This helps in the transferring process. As to creating the new displacements on the new model, there are several hurdles to overcome. One, we cannot just grab the displacements and shove them onto a different model because it can potentially look different, have different curvatures and sizes. The solution to this would be capturing the statistical properties of the surface details and use this information to synthesize new ones on the target model. Note that we cannot just perform this texture synthesis in 2D like we do for images, because as we project the result to a 3D model, it introduces severe distortions to the displacement patterns. It is a bit like putting a rubber blanket onto a complicated object. Different regions of the blanket will be distorted differently. Make sure to have a look at the paper where the authors present quite a few more results. And of course, the intricacies of this technique are also described in detail. I hope some public implementations of this method will appear soon. I would be quite excited to use this right away, and I am sure there are many artists who would love to create these wonderfully detailed models for the animated films and computer games of the future. In the meantime, we have a completely overhauled software and hardware pipeline to create these videos. We have written down our joyful and perilous story of it on Patreon. If you're interested in looking a bit behind the curtain as to how these episodes are made, make sure to have a look. It is available in the video description. Thanks for watching and for your generous support, and I'll see you next time.
