Dr. Zsolnai covers NVIDIA's latest graphics and AI breakthroughs from Siggraph. A new text-to-animation system trained on 5,000 motions can generate dance, martial arts, and physics-based movement from text prompts. One unified algorithm handles meshes, point clouds, and neural radiance fields, and was used to simulate thermal analysis on NASA's Curiosity rover.
I am currently at the Nvidia headquarters to visit their researchers and CEO Jensen Huang to learn a bit more about their research works at Siggraph, the most prestigious computer graphics conference. And lots of AI things too. And yes, before you ask, of course I made him hold on to his papers. Now I tried using a current text to image AI system to create more images of us using based on this as a reference. And I get. Well, not quite what I'm looking for. It understands that there are two people holding onto a paper, a leather jacket here and there, but the people are completely different. And this is one of the most fundamental problems in text to image AI systems. Character consistency, generating the same characters twice is. Is almost impossible. And they just published an amazing paper that in the future could help us with exactly that. Look now when you ask for the same person to be in different situations, that really is the same person. And it gets better. It also supports ControlNet or in other words, we can draw a stick figure and look at that exactly the same person will take these poses. And all this takes 10 seconds per image. Insanity. So with this we can finally create a full story and never lose a character that we generated before. So cool. So this is text to image. And in the age of AI, text to video is also possible. And get this text to 3D geometry is possible too. Too. So how about an even crazier idea? Text to animation. What? Really? Oh yes, that is exactly right. Dear fellow scholars, this is Two Minute Papers with Dr. Zsolnai. Now hold on to your papers fellow scholars, because this new paper is from just about a week ago and allows us to just write a piece of text and it synthesizes that motion of on this virtual character. It can go from simple locomotion type animations to more complex movements such as dancing and martial arts moves. It looked at and learned from approximately 5,000 different motions. And you are experienced fellow scholars, so you know that in works like this, the most important part is looking for movements that are not the simplest ones, but ones that are more complex than what is in the training data set for the AI. And I have to say they did not go easy on this little AI. These movements have complexity to them. Very impressive. Loving it. And don't forget that this is a physics based animation system. What does that mean? Well, it means good news and bad news. Good news because it is accurate. The movements are not just made up from thin air. They have to relate to reality. And the bad news is a great deal more amusing because if you make the slightest mistake you might lose your balance, which can still happen here. Or even worse, look. Ouch. It can even fall over the. Don't go too crazy with those kicks, little Scholar. Also, it is quite sensitive to the phrasing we are using in our prompts. That means two very similar prompts can lead to results that are not similar at all. But the potential is huge. And in cases like this, always invoke the first law of papers which says do not look at where we are, look at where we will be. Two more papers down the line. And to think that all this runs in real time on a consumer graphics card. Super impressive. Now note that we agreed that Nvidia would not pay us anything beyond the trip itself. And this video is not sponsored by them. And we also agreed that I would talk about the limitations of the techniques too. That is the way of the Scholar. Now, of course, no year is complete without a bunch of simulation papers. And I have to say this, this here is particularly tasty. For instance, if you have a triangle mesh, you need one technique tailored to meshes to work with them. If you have a neural radiance field, you need another method that is specialized to that. However, here is a new technique that can simulate meshes, point clouds, nerfs, Gaussian splats, anything you can possibly wish for, even tomography scans. And all this with one algorithm. Wow. Now note that a limitation here is that these are not the highest fidelity simulations. The ones that can run on Gaussian splats get the closest to the visual fidelity to a production setting, but they still need one or two more papers to get there. And check this out. Previous simulation techniques, for instance, finite element simulations, often do not work that well with highly detailed geometry. So here is their new technique. Doing something that was previously almost impossible or too costly to do, and that is doing a thermal analysis of NASA's Curiosity Mars Rover. Now you astute fellow scholars immediately recognize that these are noise patterns that clear up over time, exactly as you see in light transport simulations. Ray tracing, if you will. So, so why is that? Yes, that's right. It borrows some of the techniques used there and applies them to other simulation domains that were previously impossible or too slow and costly to do. Not anymore. This one is a real revolution and also won one of the best papers awards at Siggraph. That is an honor very few researchers achieve. Congratulations. Now this one is also not without limitations. For instance, the handling of non convex domains is possible, but more costly. Now talking about ray tracing. With ray tracing, we simulate the path of millions and millions of light rays to create beautiful photorealistic scenes like this one. Now did you notice that I keep saying rays and ray tracing, but technically light is not a ray, it is an electromagnetic wave. In most simulations we just use a ray representation because it's simpler and often good enough for most practical cases, however, not always. A full wave optical light simulation is one of the holy grail problems in this area. And I can't believe I'm saying this, but this new technique proposes a way of doing just that. So what can it do and why is it useful? Well, get this. It can compute the propagation of cellular signal coverage across a whole city. And if we do it with a simple ray representation, we get this. However, these rays of light bend and diffract and taking it into consideration makes a huge difference and yields much more realistic simulations. This involves evaluating difficult and costly diffraction integrals and thus this is still very slow. So consider this to be a proof of concept paper that made something almost impossible to be now possible. And great news, the full source code is available for this one. So yes, let the experiments begin. The quality of these papers is just stunning. Loving it. By the way, the cafe at Nvidia is something else. I ordered a pizza and it arrived approximately two minutes later. Wow. Now that's what I call two minute paper style. So what do you think? What would you fellow scholars use these techniques for? Let me know in the comments below. If you're looking for an Nvidia GPU cloud service for large scale AI projects, check out One Click Clusters by Lambda, featuring up to 512 Nvidia H100 GPUs with InfiniBand networking. And hold on to your papers because with one click clusters you can access all these GPUs on demand with reservations starting at just two weeks. Try it out at lambdalabs. Com paper or click the link in the description.
