A Two Minute Papers YouTube episode (Dr. Zsolnai) reviewing an NVIDIA-related research system referred to as "edify3d" that synthesizes high-quality 3D scenes from text prompts or images. The presenter demonstrates generating environment maps, multiple objects, and a themed scene ("gold rush") and praises the clean quad mesh topology and asset readiness for games/animation. Technically, the system is diffusion-based, trained to infer 3D from multiple generated 2D views (quality improves with more views, e.g., 4→8), includes built-in super-resolution, and supports text-to-3D and image-to-3D. Concrete constraints mentioned: the neural network is reported as 2.7 billion parameters, textures are up to 4K resolution, and material outputs are limited to albedo (no advanced material models yet). The presenter estimates generation times on the order of two minutes (speaker notes uncertainty about per-object measurement) and situates the work among active related research (e.g., MeshGPT).
Questions this video answers
2.7 billion parameters.
Up to 4K resolution.
Approximately two minutes (speaker notes uncertainty; possibly per object).
Chapters
I was really looking forward to this paper. So let's use an AI to create a 3D virtual world. But let's do it in a way that we don't need to be a skilled 3D artist. How about just using a text prompt as an input? Nice. It is giving us a list of objects that we need. Okay, that's all well and good, but that is text. We don't need text. We need 3D geometry. Yep, that is the next step. Wow. OK, so how about an environment map? This will be used as a background and for the lighting of the scene as well. Looking great. Now putting it all together. And goodness, that is fantastic. But this still needs something. As this is just a bunch of stuff thrown together. It needs a nice overarching theme. How about a gold rush theme? There we go. Loving it. And now get this. We have a full research paper by the name edify3d that describes the secret of how all this is possible. And turns out it can do even more. Dear fellow scholars, this is Two Minute Papers with Dr. Zsolnai. They call it high quality synthesis. I would perhaps tone that down a little. However, that is still a huge improvement over previous works. So what more can it do? Well, well, as you saw, textinput works really well. But not only text can be the input. You can assemble a huge scene with just a text prompt and it can do a variety of objects and styles. Spectacular. And can that really be? Wow. Just make a photo of something and you get a 3D model of it. Hmm. This gives away a small part of the secret which we will take a closer look at in a moment. So this gives you a 3D mesh with a quad topology, normals, everything. Which in short means that I think you can use this as is in your computer games, animated movies, virtual avatars, anything. And if you look closely, these are clean, clean topologies. Something that you don't always get from previous works. Thumbs up. So how long are we waiting to get all this? A scene like this would take me hours and hours to make. And the AI does all this now? Not in hours, but hold on to your papers, fellow scholars. It does it in just two minutes. Now it's not two minutes. Like two minute papers is two minutes, which it is not. It is actual two minutes. Wow. I did not find the paper specifying it, but I think it is per object. By the way, the neural network they use here has 2.7 billion parameters. That might sound like a lot, but it is a really tiny network by today's standards. It is if you have a newer phone, you might already be running models of similar size on it without even knowing about it. So what is the secret? Well, let's pop the hood. And fellow scholars. Now you will witness something really cool. One, it is a diffusion based model. So it starts out from noise. But two, it generates not one image, but a bunch of images. Then it tries to guess what kind of 3D geometry would look like. These images. I am loving this. And in the meantime, the textures are waiting to be put onto the final model to make sure that the quality is as good as it can be. It also does its own upscaling. Super resolution if you will. The enhanced thing. Now you see that of course it can do text to 3D and image to 3D too. It is built into the system. It is how it works. And finally, here is the secret. It is also not an accident that multiple views are being generated. And the whole neural network was trained to be able to understand 3D geometry from a bunch of 2D views. The more the better. For instance, if you train it on four views, this is what it can do. But if you bump it up to eight, there is a noticeable difference. Now limitations. We have textures. They are up to 4K resolution. Goodness. OK. But still no sophisticated materia models yet. Only albedos. That is at the risk of simplifying it. Color information for each part of the geometry. There are previous works that specialize on materials and you can generate absolutely stunning virtual objects with really sophisticated material models. And I bet that this is coming just two more papers down the line. I met this team a few months ago at the Nvidia headquarters and they are super nice and brilliant scientists. They showed me an earlier version of this work and said that they are working to improve it and might be able to show it a few more months down the line. And now here we are. So cool. Note that we do not have a business relationship with Nvidia. And note that this area is subject to intense research attention. For instance, this one is called meshgpt. That is able to build new geometry right in front of your eyes and is also way better than previous works at that point. Look at how precise these objects are. So cool. And it can even suggest you possible ways of completing your own work. How cool is that? So what do you think? What would you fellow scholars use this for? Let me know in the comments below. To run your own experiments on an Nvidia gpu? Check out Lamda. I use it myself regularly for these videos. Hmm, look at that. You can generate high quality images in less than a second per image. I did a ton more of them and paid less than a dollar for all this crazy. Seriously. Try it out now@lambdalabs.com papers or click the link in the description.
