Commentary on a recent paper presenting a differentiable mesh-generation technique that reconstructs high-quality 3D geometry (and animations) from images and video. The new method reportedly produces geometry better suited for 3D printing, converges from an initial shape to desired geometry, enables animated tetrahedral meshes for physics simulation, and uses about 20% fewer triangles than traditional methods like Marching Cubes.
Questions this video answers
About 20% fewer triangles/elements.
Chapters
This paper has a ton of promise. Amazing promises. Get this. It promises to reconstruct 3D objects from a bunch of images. It promises better generative 3D modeling, even creating an animated 3D model of US moving tetrahedral mesh geometry for physics simulation for squishy things. And can also recommend geometry models that are easier to 3D print. Now wait, wait, wait. This is not new. In fact, this is old stuff because this was possible for 36 years now. What? Yes, really? The legendary computer graphics paper by the name Marching Cubes also set out to do that. It is a simple handcrafted technique and it can be used for tomography, magnetic resonance imaging and more. It gives you everything you want. Or neural dual contouring. A paper from just a year ago. So why not just use them? Well, here's why. Look, we are starting out from this sphere and would like to morph it into this object. Can we? Oof. Not quite. The previous dull contouring technique blows up immediately. The neural version is more well behaved, but it can't give us our desired shape. Now Marching Cubes comes to the rescue. Until it falters in being flexible enough to reconstruct the final shape. A good try though. So what about the new technique? Wow. That is a seemingly perfect reconstruction of the desired geometry. So what else can it do? Well, hold on to your papers, fellow scholars, because here in goes a video of a physics simulation. This is really tough because the desired geometry is not stationary anymore. And once again, it starts out from a ball and over time. Wow, look. Out comes 3D geometry that we can use in our video games and all kinds of virtual worlds. And all this from an input video. So cool. And it gets even better when combined with a text to 3D technique from just two years ago. Here we can do something similar to text to image. We write something and out comes a painting of that. But here instead we write something and out comes. Can that really be? Out comes this 3D geometry. And the difference is that finally we are now getting high quality geometry. And get this, even creating animations is also possible with this. That is a huge step forward. And because this is a differentiable mesh generation technique, it means that it starts out from something completely different and it slowly morphs into the final result over time. I absolutely love these animations. And this one shows that the new technique once again converges to the desired geometry. We have already seen that. Okay, but wait a minute. That is not only a better piece of geometry, but it also uses about 20% fewer little elements, little triangles to accomplish that. This means that these models will require less memory and and can be rendered faster. Fantastic. And there is a problem here. Do you see the problem? Well marching cubes has some more issues for instance in the nose region here. Thus this geometry is typically not 3D printable or if so not out of the box. And finally the new technique can generate pieces of geometry that don't have these problems and are now more suitable for 3D printing. Now clearly these are not yet the kinds of assets that you would directly put into those AAA quality games. Not even close. But all this is a heck of a lot of value in just one paper and just imagine what we will be capable of just two more papers down the line. We are sponsored by Weights and Biases Experiment Tracking, Model Evaluation and Production monitoring for your Deep Learning products and LLM apps. It is the best. Everyone is using it. Try it out now at WNB Me Papers or click the link in the description below.
