Nvidia's NEW 'AI Perfusion' Takes the Industry By STORM! (NOW ANNOUNCED!)

Nvidia just released around 20 different research papers in which they detail the very next advancements in generative AI and how it's going to impact us. Now, in this video, we're going to cover four of the key ones that they talk about and why these ones are very, very interesting. So stay tuned because honestly, you're going to be surprised at what Nvidia has been able to do, especially with the advancements in AI. So one of Nvidia's first groundbreaking papers is essentially learning physically simulated tennis skills for from broadcast videos. So in this video and in the research paper that they've done, they essentially talk about how it's possible now for you to use broadcast videos such as, you know, perhaps the NBA, perhaps maybe just a simple tennis match, but to use this in order to simply mimic those skills, bodily movements, and map that accurately onto a 3D character. Now, you have to understand that this isn't just something where they're simply just using some fake motion capture data. This is really accurate, really precise, and really good data that is then being applied to this 3D rendered character that you can see. So we know that motion capture is very expensive to produce at scale, and we know that some companies that want to use motion capture currently cannot do so because of course, it might be outside of their budget range. Now, of course, what's also an issue as well is that motion capture data is usually quite large. So this sometimes does have its issues. So as you know right here, the. The more actually recorded footage that you do have, let's say, for example, when we look at games of athletes that are actually playing in their current sport, it's always going to be much more natural and much more better than just simply recording some motion capture footage. And essentially, what this paper talks about and what this video talks about, which we'll leave a link to in the description, is how they're able to essentially get that data from the video and then essentially translate that onto a 3D character in which they can then accurately represent exactly what was done in the video. And, and to the extent that there's a lot that these characters can actually do in terms of their ability to hit the ball into several different locations at certain times and hit it into a certain spot at a certain time. So it's something that is definitely refined. It's not something that seems quite basic, because I know that many different projects out there are quite basic, but this definitely seems very, very refined. And the system that they've developed seems very, very good when it does come to refining some of the jagged movements. But because they talk about how there are many different issues with this, but they've managed to refine those and fix those. Now some of you might be wondering what other tools are there like this out there? Now, there's one tool that recently did come to mind, which is Wonder Dynamics. They did release something that is quite similar essentially that tries to wipe out the need for necessary mocap animations and it's actually very, very interesting. There are many different demos that you can actually see online. And of course this is in beta stage where you can actually sign up to the waitlist. So this is definitely something a as well that is going to be going on in the motion capture space, another area that is being disrupted by AI. But I do have to be honest with you, this research paper by Nvidia, it does show a very fine tuned version, I think, of what Wonder Dynamics is trying to do. So I think that if this is fully released at scale, it will most certainly change the way things are captured, especially in video games and in any other industries where you do need that kind of data. Now, Nvidia, like I said once again, are coming with more stuff. Now this one, I got to be honest with you, is very, very, very impressive. Okay, this is what we call key locked ranked 1 editing for text to image personalization. That's what Nvidia have titled this. And essentially this is what they call perfusion. So it says we present Perfusion, a new text to image personalization method with only 100 kilobytes model size, trained for roughly 4 minutes. So this can creatively portray personalized objects and allows significant changes in their appearance. Now trust me when I say you're about to be shocked at how good this is and at training personalized images based on literally sometimes even just one image and then allowing yourself to customize that with a text prompt. So let's take a few look at some of the examples because this is by far one of the most game changing things that I've seen when it comes to image generation in AI. So the first example we have here is Perfusion being able to generate eight appealing images in several seconds, where essentially they transform the nature of what's going on in that image. They're able to take the actual table and then present it covered in snow. Now you can see right here that this isn't just fluff. This is actually really high quality data that we can see in this image that looks really legible and really good. Now this is what I'm saying, when I say that this is going to be more widely used, I believe, than maybe some applications like midjourney, because this personalization is essentially what really drives the utilization of these kind of models. Because if you can personalize something, then it definitely does have more use. Now of course as well, you can see that there is a large level of consistency with the zip images, which is essentially a lot of people have wanted in mid journey for quite some time. And Perfusion actually excels at this and does it very, very well. Now there's also some other incredible examples of where you're able to train the examples. You can see on the left hand side the examples of the teddy and of course the teapot. Then essentially what they've done is they've transformed the single concept into another image. And this looks very, very good. So we, we can see here that this is really cool. But what you don't see in this example, if you look closer, you can see that they've actually combined two of the trained images together. On the right hand side we can see inference, combined concepts. So a teddy sitting by the fire with a teapot and a teddy sailing on a teapot in a lake. And that's really, really, really cool because I do believe that this has many different applications. And I'm pretty sure Adobe is going to be scrambling to get this kind of software embedded into Firefly. But because this is something that I don't think I've seen anywhere apart from Dreambooth, but with this level of consistency and of course quality, we haven't seen this just yet. So this is honestly truly groundbreaking stuff from Nvidia. Now what's also cool is that you can see some of the other examples here where there's a teddy dressed in a blue suit looking at a gourmet meal. Looks very accurate. Then of course, there's a dog wearing a sombrero. Definitely another accurate output. Now one thing that many people are wondering is how does this compare against the other models that are in the space that do exactly this? So you can see right here that they actually showcase other models and exactly how they handle the same exact prompt with the same exact tasks. And I think it's clear that by far Nvidia's model is clearly just the best. I mean, although they have used several prompts to get this, the other ones just didn't seem to actually get the task. And they honestly do fail, but not spectacularly. But Nvidia's is just that step ahead. Now this right here is what I wanted to show you all, because this is what I think the future of AI image generation is going to be, where you have this one shot personalization. Now some people might just have one image of something and they want to essentially have consistency of that image and then essentially manipulate that even further. And with Nvidia's Perfusion, you can see right here that this is exactly what we have. So this is something that is truly game changing. Nvidia's Perfusion is one of the models that I think they're going to talk more about and at the event. And it's definitely something that you should be paying attention to because once it does get embedded into, I believe potentially their Nvidia Picasso, we could then be seeing it rolled out into many other different applications because Nvidia is going to be allowing many different, you know, companies and softwares to be able to use their Picasso cloud service. So definitely something to look out for. Now this is very interesting. This once again is showing where Nvidia leads the way. This is live 3D portraits from real time radiance fields for single image portrait view synthesis, or in simpler terms, they present a one shot method to infer and render photorealistic 3D representations from a single image in real time. So they can just pretty much just get an image of something and then essentially get a 3D realistic photorealistic representation of that image. Now right here you can see the inputs versus the outputs. And for those of you who haven't seen the competition's level of detail, you won't understand why. This is truly groundbreaking because it's very hard to infer 3D data from an image because there are many different complex things at stake when you're trying to depict what goes where. And it's something that you can easily get wrong. But this is something that Nvidia has mastered. And this research paper video that they talk about, it goes over in detail how they've managed to do this and all the different techniques that they've used. I mean, it's honestly quite interesting as to how they managed to do this, because this wasn't something a couple of years ago that you thought you'd be able to get with this level of accuracy. And like I said before, as AI continues to develop at this rapid pace, we're going to start seeing innovations which we weren't thought possible develop at a ridiculous level. Now this was something that I found really, really interesting. And where you can see in real time, you're able to get these different effects. You can see that with the Input compared to the output novel view. You're able to get a completely different angle on what this speaker is saying. And you can see right here that we have a driving video, then we have an input single image. And then of course we have the 2D talking head and the 3D lifting with their method. Now this can be comparable to DID in some aspect, if you don't know what that is. Essentially it's where people convert an image to something moving. But this is on a completely another level. You can see right here that you can see Nvidia's at the top left compared to the others in the other areas. And we can see clearly that Nvidia's is far more accurate with far more detail. And, and this is clearly superior to anything else that is currently being developed. So I would say hats off to Nvidia for doing this, because constantly, with every single blog, with every single research paper that I read, compared to the other research papers that I read, literally just a couple weeks before, we're seeing Nvidia push the boundaries on what's truly capable. Now what's also cool is that essentially, as we talked about previously in real time, you can see that this is a live demo of someone essentially using a phone to, to RGB video to 3D. Now of course, this is running on an RTX 490, which is essentially one of their top tier graphics cards. But this also goes to show the potential applications of this. Imagine maybe you wanted to see different angles of something, or maybe you wanted to see someone in essentially a more 3D way. This is going to present more lifelike and more realistic video calls. So I'm not exactly sure what these applications are going to be. Of course right now it's just pure speculation, but just goes to show that people being able to generate this amount of depth and detail From a simple 2D image is truly incredible. Now this one is really cool. Essentially, if you're someone who plays games, you know that hair is something that games do really struggle with because of the complex computations needed to essentially calculate how each hair strand is going to move. But Nvidia have come up with a solution. So in this new paper they essentially talk about a method that can simulate tens of thousands of hairs in, in high resolution and in real time using neurophysics, an AI technique that teaches a neural network how to predict how an object would move in the real world. So essentially what they're doing is using these neural networks to essentially teach them how these hairs would look. And remember, this is going to be in real time. So this is going to be really cool to see how it actually plays out in real game. Then, of course, we had the neural textures versus the BCX textures. So essentially what we have here is once again using neural networks to essentially compress textures that usually are quite high in file size, but at no other cost. So there's no cost in GPU size, There's no cost in essentially size. You can see that they're both around 3 megabytes in file size, but at the same time, you can see that the one on the left, Nvidia's new neural textures, actually produces up to 16 times more quality with the same actual file size, which is very, very interesting because it means essentially you're going to get higher quality at the same power level. So this is really, really game changing for Nvidia. Now, this annual conference on computer graphics is going to be happening in August, and that's where they're going to be presenting all of their ideas, especially some of those that are interconnected with AI. And it's going to be interesting to see what other stuff Nvidia has to talk about, especially expanding upon some of the topics that we talked about today.