Mind-bending AI Is Changing Art And VFX Forever

In this video I'm going to cover some of the really, really exciting AI research that we're going to get access to soon, but isn't quite ready. But it's pretty dang mind blowing.

And I know people that watch my videos, they like actionable stuff. They like to know what they can use. Now I'm going to kind of Split this video 50 50. I'm going to show you some of the really cool research that's going to blow your mind that's in the works as well as some of the stuff that you can use right now that you can that's available to play with today. So let's get into it. So the most mind blowing research paper that's come out recently that has taken the AI world by storm is this drag your GAN research. A GAN is a Generative Adversarial network. Now you've probably heard of diffusion models to generate images where an image starts with a bunch of noise and then the noise is removed to get to the image that you're trying to create. Tools like Midjourney Stable Diffusion, Dall E, they all use diffusion models. Well, Generative adversarial networks or GANs and is a different method for generating images. And I'll likely break down the differences in a future video. But it doesn't use the same diffusion method. It uses a method where it essentially tries to generate an image. The computer tries to guess whether that image is real or fake and it goes back and forth and back and forth until the computer could no longer tell whether the image is fake or not. Now that's a real oversimplification of how GANs work. But if I really try to break it down, that would probably take up this whole video. So again, I'll try to make another video in the future about the difference between fusion models and GAN models. Because while GAN models are actually an older method of generating images, they're actually becoming more and more popular lately. Anyway, I'm off on a tangent now. This drag your GAN has been blowing people's minds and you're about to see why. It's a method where you take a starting image like this lion and you can drag the nose from one point to another and it will actually tweak the image to move the nose to that new point and and it will actually be very coherent. And you can see that the body of the lion actually changes as it moves the image around. So it adjusts the whole image to make sure that it still looks like what it's supposed to look like here's another example with a cat where they move the cat's nose and the cat adjusts just so slightly, but the rest of the body adjusts so that it stays coherent with the new changes that are being made. Here's an example where it moves the dog's feet, it opens the dog's mouth, it adjusts where the dog's ears are, and as it makes these moves, it adjusts the entire dog. Or you can mask it off and say, only adjust what's inside of the mask. Here's one where it's working on a car and adjusting the size of the car or moving the position of the car or changing the way the car is pointing and it adjusts the entire image. Here's a horse where it separates the legs, or the cat where it moves the face or closes the eyes a little bit. It is just insane. Tech like this is probably going to replace Photoshop in the future. Make this girl smile, move her cheeks in, make her look her face look thinner, move her bangs up, just adjust her right eye or her left eye, make the shorts longer, move the feet, move the sleeves. There is just so much crazy stuff that you can do with this technology. You start with a single image and then just drag the image around until it gets to what you want it to look like. It's not available for us to use right now, but if we jump over to their GitHub page, we can see down here. Code will be released in June. So sometime in June we'll will have access to play with this. And if I had to guess, we'll probably see Photoshop plugins and all sorts of tools start to add this into their workflows so that you can do this kind of manipulation with imagery. Again, this research is called Drag your gan. It's at this crazy URL here, which I'll make sure is linked in the description below the video. Now here's some research out of Nvidia where you can manipulate somebody's hair all the way down to like the finest details you can see. In this example image, they've got this woman and they can manipulate almost every hair at a individual hair detail. They even can trim the hair, make the area they're messing with wider, and look at how realistic they can actually make the hair. Now I know this gives some something about Mary Vibes here, but this is actually really, really fascinating research. I mean, the average human has over 100,000 hairs on their head. And with technology like this, you can actually get in there and adjust individual hairs and get the exact detail that you're looking for on an image. This has huge implications in the gaming world as well as the visual arts and filmmaking world, where you can get so fine tuned and so granular with the images and the people that you're creating with computer graphics. Also coming out of Nvidia research recently is this Real Time Neural Appearance Models. This is actually really mind blowing as well because it creates these textures on these objects that are so realistic that if you just saw this, you wouldn't know that that wasn't a real object. You can see that it starts as just a white 3D object and then it adds this texture over it that has the reflections and just this polish to it where even when it's zoomed in, you can see all these fine details. It looks like there's specks of dust and scratches and little imperfections on it, as well as the shine and the glare of the lights around it. But this is just a 3D object that's being textured with hyper realistic textures. Here's another example of a white object and then it's given this texture to it that makes it look like this antique artifact. The level of detail that they're able to get on some of these textures is just mind blowing. Again, this one's called Real Time Neural Appearance Models. It's put out by Nvidia Research and I will link up to this research paper if you want to learn more about how this works. Now, the next one I want to show you is called Text to Performer, which is another one that's not available yet. But the demos look pretty crazy. So here's a text prompt. This female is wearing a dress with solid color pattern. It has no sleeves and it's of three quarter length. She shifts to the right and then you can see here's the girl and then she shifts to her right. But then check out this one where it actually performs a sequence of prompts. The dress the person wears has medium sleeves and is of short length. The texture is of pure color. The lady moves to the left, she is turning right from the front to the side. She is turning right from the side to the back, she turns right from the back to the side, she turns right from the side to the front, she moves to the right. And as it goes through these prompts you can see that the text to Performer actually performs these actions. And this is all AI generated here. It generated the person, it generated the dress with medium sleeves, short length and a pure colored texture. And then it actually moves her in all of the directions. It says here in each of these individual prompts. Here's another one. This person is wearing a sleeveless dress. It is of short length. Its pattern is pure color. She moves to her right. She's moving towards the center from the right to the side. She turns right from the front to the side. She turns right to the back. She turns her head to the right. She turns right from the back to the side. She turns right from the side to the front. She is swaying in place. And you can see at the end she's kind of swaying in place. One last example here. The dress the person wears has long sleeves and is of short length. And its texture is pure color. The lady moves to the right. The person is moving to the center from the right. She turns right from the front to the side. She turns right from the side to the back. And this text to Performer performs all those actions. This one again is not available yet, but it's called Text to Performer. And this is a research paper where I imagine we'll have demos and hugging face spaces where we can play around with this soon. Next up is Text to Nerf. And if you're not familiar with Nerf, I have talked about it in previous videos. It's neural radiance fields. It's basically where you take a single image from a whole bunch of angles and then all of those images are essentially stitched together to make a 3D object. Well, with Text to Nerf, you can just give a text and it will generate 3D objects. Here's some of the prompt examples. A bedroom with a desk. And it generated this 3D model of a bedroom with a desk just from that text prompt. And you can see based on the movement that it is 3D. And it created a sort of depth map with that text to image prompt. A cozy living room. A dog playing on the lawn. A bonsai on the table. A cabin in the woods. A red sports car in the park. Just from a single text prompt, it creates this neural radiance field, which is sort of a 3D image with depth to it. Here's some more. A locomotive at the train station. It generated two different versions of this. A deserted commercial street, two different versions of it. It can even do it for artistic styles and not realistic images. An oil painting style or a unicorn in a forest or a future city in a surrealism style. It could even do 360 degree scenes where it's actually doing a garden and it's rotating the camera 360 again. The code is coming soon. So this isn't one that we have access to, but everything I've shown you up until now is really awesome research which is just making our AI generation capabilities so much more realistic. Starting with drag your gans, where you can actually manipulate an image to make it look any way that you can imagine just by dragging and dropping it around. The ability to you manipulate hair to the finest details, creating textures that are just ultra realistic that you would never even know that this wasn't a real video, that this was a 3D object, because the texturing on it is just so precise and perfect. Text a performer where we can generate a person that actually looks the way we describe them and moves around in the way we ask them to move around just by using text prompts and text to Nerf, where we can use a text prompt and actually create a 3, 3D scene from a single text prompt. That is awesome, insane technology that we're going to have access to within the coming weeks or months. And from somebody who likes to make creative content like 3D videos and crazy imagery. I'm really, really excited to get my hands on this stuff. Now I want to talk about some stuff that you can actually use right now. So if you've ever used the tool did where you could upload an image and then add some text and make that image talk. Well, there's actually an open source version of that now called Sad Talker and you can actually play around with it right now over on Hugging Face, where you can upload an image, give it an audio prompt, and it will actually animate that image to go along with the audio prompt. The holidays are a time of celebrating with loved ones enjoying the festivities and muddling through Wisconsin's cold. This is completely free to use right now on Hugging Face. There's a whole bunch of examples that you can see down here. Here's another one. Un neut ilise ENS navigateur comme d' habitude el partage des information sur seis gout est habit unsommation. You can see the face moves with the voice. We can even get rid of this aud and type some text here. It's not the greatest audio in the world. You're probably better off uploading your own audio. But you get something that looks like this. Hello. Subscribe to Matt Wolf on YouTube and this is called Sad Talker. And right now it's free to use on Hugging Face or if you want to install it locally. So check it out. I will link again to it in the description. Now this next one Isn't actually AI at all, but I just found it really, really cool for video generation. I don't know how long this has been around. This may have been around for years and years. And I just now discovered it and I'm way behind, which is probably most likely the case. But you can actually create animations using Google Earth Studio. So if you jump into Google Earth Studio, there's actually a few different templates you can use where you can have it animate zooming into a specific point on a map. You can have it orbit a certain object on a map, go from one point on a map to another point, or create a spiral around a certain spot on a map, or even fly to something and then orbit that spot thing. I found this really, really cool. Again, not AI, but something really, really awesome for video generation. So I'm about to head to the awe the Augmented World Expo in Santa Clara at the Santa Clara Convention Center. You can see it zoomed in on that on the map. I'll just set my starting altitude as above Earth right there. And you can see it just made this animation where it finds the actual Santa Clara Convention center from zooming out from Earth. Now if I click check, we actually have an animation editor here where I can animation and add keyframes. Maybe when it gets to the end, I want it to be sort of more of a 3D view of the convention center. I can pivot it and draw it like this, frame it up the way I like it, set this as a new key frame here. And now as it zooms in, it's gonna zoom into that 3D version of it. I just found this really cool. I don't know if anybody else is as nerdy as me and finds this kind of stuff interesting, but you'll probably see me use this in a few of my videos. Especially when I travel. I can do a point A to point B and showing the sort of path that I'm traveling to go place, and then probably combine it with tools like Kyber to make really cool animations. Speaking of Kyber, Kyber is about to release a new feature. I did get early access to it, but it's actually coming out May 23rd. So likely by the time you're seeing this video, it's either coming out tomorrow or it's already out by now. And the new feature is called storyboarding. So if I create a new video here, I can begin with an initial image here. Let's start with this profile pic that I use everywhere. And then I can continue to a Prompt. It already kind of guessed what was in the image. A man with a beard and blue eyes in front of a purple and blue background with a neon pattern in the style of let's go cinematic. And then it automatically wrote this prompt. Photo taken on film, film grade, vintage, 8k, ultra fine, etc. Now I can move on to video settings here and I can have it make this first section be eight seconds. Let's have it zoom in during this section. Show initial image in the first frame, and let's go ahead and generate a preview here. I'll use this one because I like the curly mustache that it made. And then what I can do is I can click this plus storyboard and add another section to it. So this time let's make it a humanoid robot with circuitry and a big brain. And let's have this one do in the style of 3D rendering, go to video settings. And this time let's have it zoom out and move down. We'll go ahead and generate preview. And you can see it started with that same image as the inspiration for the first image. So we'll go ahead and use that. And then we can even add a third storyboard if we want. So let's click storyboard again. This time let's go a humanoid alien with green skin. And let's do it in photorealistic. Let's just see what happens. Go to video Settings. This time let's have it move to the left and rotate clockwise. We'll generate preview, and let's go ahead and create our video. It's got three different scenes in it now with three different prompts. And it's going to move between the scenes. It's going to change the rotation based on the settings we gave it, and it's going to create something really interesting. So let's go ahead and create video. And now it's going to craft our masterpiece. And here's what we got out of it. You can see it sort of shifts into the guy with the handlebar mustache. My hair turns blue as it evolves. Now I start to turn into a robot. And the camera starts to zoom out as I turn into a robot for about eight seconds. And now it starts to rotate as I slowly turn into a green alien like the prompt that I gave it. Really, really cool technology. You can actually add music to it as well. I didn't have any music queued up, but that is another feature inside of Kyber where you can actually add music to go along with it. And again, this is called Kyber's storyboard feature, which, as of the recording, isn't publicly available, but as of this recording, it's going to be publicly available tomorrow. So there's a good chance it's ready for you very soon, or possibly even now. And then finally, I want to talk about Blockade Labs. Now, this is a tool I have talked about in the past, but they added something that is really, really cool, which is the ability to draw in stuff with the images you're creating. Now, Blockade Labs creates these 3D landscapes where you can look around, you can generate whatever you want. But if we come down here and click Create new, we actually get this sort of grid where we can draw in whatever we want here. So if I was to take this paintbrush tool here, let's draw a little house. I'll try to kind of stick with the lines as best possible. I'm already doing a horrible job at it. Let's go ahead and draw a little house here like this, maybe even put a little chimney on it. And then let's move over here. Let's draw some circles and a square. And let's move over a little bit more. And then over in this area, I'm going to draw some lines here. Maybe even put a line across here, see what happens. And then down here, let's do modern computer animation and let's type a colorful Minecraft village and click generate. And it's actually going to use these lines that I drew as part of the prompt. Here you can see here's where those lines are that I drew. It kind of put like a little building with a line across it. Here. Here's my house that I made. You can see it, drew it in that same area. If I come around here, here's the little circle that I made. It made a little, like, orb thing. Here's my square that I originally drew. You could see if I hover over this area for these guides here, you can see where my original circles are. You can see circle to the left and the square, how it sort of followed that. If I drag over to where these lines are here, and I hover, you can see there's my original lines, where it followed those. And if I drag over to my house over here, hover over this, you can see it put the house with the chimney exactly where I drew it in, and it created us a nice, colorful Minecraft village. Following my drawings, I could change this prompt down here to something else. Let's go a futuristic cyberpunk city. Let's change this to cyberpunk. And Click Generate again. You can see where my original outline for my house is. And now in our cyberpunk city, you can see it made a sort of building here in that same shape. If I move around here, you can see here's where my original lines are. It actually put some buildings that kind of cross where those lines are. If I circle back around over here, you can see here's where my original circle was. It kind of created this big circle building and another building where I put this square. So it follows what I drew. No matter what prompt I give it, you can see that it follows my drawing pretty well. This Blockade Labs tool is just becoming cooler and cooler. You can use this for your skyboxes, for things like Unreal Engine or Unity. If you're creating games, if you're creating, you know, a 3D video, you can actually use this for your 3D world and your 3D video. This is just a really, really awesome tool that they keep on improving, they keep on shipping new features. And I'm super impressed with what these guys are doing over at Blockade Lab. So I wanted to show this off because this is something you can use right now. And as of today, as of the recording of this video, it's totally free to use. You can just get in here and play with it and start drawing some cool 3D worlds. So that's all I got for you today. Just wanted to kind of have a little nerd out fest about some of the cool tech that's coming out, some of the cool tech that you can use right now. If you love all of this AI stuff and you want to stay even more in the loop, check out FutureTools IO. This is where I add all of the cool tools that I come across. I curate only the best tools. I'm adding a handful of new tools every single day. And really I've gotten strict with the tools that I add. Out of all of the people that are submitting tools, I only approve maybe about 10% of the tools that are submitted because I don't want all of the tools that do all the same crap. You've already seen other tools do. I only want to show you the cool, novel tools that we haven't seen a hundred times already. So it's a real good curated source of all of the latest AI tools that you could use and play with right now. And I also keep this AI news page super up to date with all of the latest news in the AI space. So check it all out. It's over at FutureTools IO. If all of this is too overwhelming and you just want to stay in the loop once a week, make sure you join the free newsletter every Friday. I'll send you the TLDR of the Week of just the five coolest tools I came across, a handful of news articles, handful of YouTube videos, and one cool way to make money with AI. Again, it's just the grand overview of the signal versus the noise in AI, so to speak for the week and it goes out every Friday. You can join the newsletter over@futuretools.com and if you like this video and you want to see more videos like this in your YouTube feed, give this video a like and subscribe to the channel and I'll make sure that you stay up to date with all the latest research and news and cool tutorials in the AI space. I'm making videos at least three days a week, so if you're subscribed to this channel and you click the little bell to get notifications, it'll make sure you stay in the loop and you never miss the latest videos. Thank you so much for tuning in. I really, really appreciate you. I'll see you guys in the next video. Bye bye.

Sam.