FINALLY: Consistent Characters in AI Video! & MORE (Latest in AI)

I think one of the reasons I love the AI space, AI community and AI research so much is that you never know what you're gonna see. I mean, every day it's like a brand new thing. You're like, oh, AI can do this now. That's crazy. And it's because AI technology is genuinely so powerful that it's just applicable to pretty much any domain you can think of. There is a way to implement it and use it to build something new. Kicking off today, we've got Lucid V1. This is essentially that Minecraft demo that I showed you guys last week. Like you're playing the video game Minecraft, but it's AI generated and also feels like you're playing a dream. But this version actually has the ability to run on consumer grade hardware. Apparently they call these world models, they are kind of world generation real time models, but they're also sort of just video game models. So it's all trained on Minecraft footage, right? Like the endless hours of YouTube videos that are about just playing Minecraft. And then it makes pretty realistic Minecraft gameplay that you can control in real time. If you take a look at the demo videos, it is very reminiscent of that last demo we just saw. But here's an example of placing water in the water actually flowing, pretty much as expected from Minecraft. Obviously this is research preview stuff, but I think that even if it's a research preview, it's a very good proof of concept, right? We've got someone falling down. Obviously you're supposed to take damage in Minecraft when you fall off a high surfaces and that is exactly what occurs. And that's all they're trying to show off in this extremely blurry video. For example, we've also got water slowing the player down and not causing any damage when falling from a large height. Which is another, you know, key basic feature in Minecraft that can be tested out to show you that it truly does understand a lot of the connections, the visual connections that happen in the game. Or for example, being able to light up dark spaces with torches. Anyways, it does run on a consumer grade RTX 4090. I know that's still like a massive expensive GPU and it's nowhere even close to being more efficient than just playing Minecraft normally. But it's incredible that it can be done in real time. Already on consumer grade GPUs, it can compress two seconds of gameplay with just 600 tokens, allowing for real time inference. And it's a huge leap from the typical standard Approaches like we just saw, which require at least 10,000 tokens for similar duration. So it's just a highly, highly compressed version of this model. And what's also interesting is that it does indeed scale. So this right here is an a undred million parameter one. This is five hundred million parameters, and this is a billion parameters. So as these models get bigger and bigger and bigger, they take more time to run, but you get more and more accurate world simulation. And by the way, guys, yes, they do have a demo that you can try out yourself online, which is pretty cool. So it's running on their GPUs, not yours. But if you do have a 4090, you can run it at home if you'd like. But check it out. Yeah, this is some AI generated Minecraft. I'd say it's definitely below the quality of the last one that I tried, but. But it is working. And I would say it has less input latency than the last one I tried. Right off the bat, I will say that we can sprint, we can jump, we can still break blocks. Can we destroy some of this grass and dig down maybe? Can we? Oh, it does not want to do that. So there's already something that we're lacking in comparison to the last Minecraft demo I tried. And will I just appear in a different space every time I look up? Yep. Oh, no. It did that one pretty well. If you give it a couple seconds and then you look up, it will completely change your surroundings, which is pretty crazy. It is very dreamlike. It's very surreal to play Minecraft like this. It's not really Minecraft now, is it? But it exists and it is here. And one day this could very well be the future of video games. I think the first step that we're gonna see is very simple 3D renders that are then overlaid with AI video generation in real time to make things look realistic. And then long term, we might see fully AI generated video games, such as the one that we are playing right now, and you might not even be able to tell. There's definitely a lot of hurdles, like the fact that the environment changes every time you know, you look around. We've got, I think two suns, for example, we just saw, and basically it's super inconsistent. But it's a glimpse past the curtain, folks. It's a glimpse into the future. But yeah, if you got a 4090, try running this beast and see if you can get to the end. I think that would be pretty cool. And next up, guys, I want to talk about AI video generation updates because they just don't stop coming this year. I don't even know if I can make a strong enough point about this, but the rate of development for AI video generation this year has been totally unprecedented, like shocking. We started this year off with having very squiggly, inconsistent, gross looking video generation models. OpenAI goes, hey, we have something way better than that, but we're not going to release it to the public. And everyone scrambles to instantly make everything better. And we've had nothing short of a miracle in the video generation space this year. So let's just take a look at the latest updates the past, like, I don't know, two, three days, your mind's gonna be blown. First of all, we have an announcement from Vidoo AI. This is an AI video generator that a lot of times goes unnoticed, underappreciated. There are a lot of other bigger competitors that have better coherency in the fine details in comparison to vdoo. But VDO excels in things like its diversity, able to generate not only things that are very realistic, but also realistic physics, shockingly good animation, and above else. It's actually a fan favorite on my Discord server. Anyways, they've just announced some new capabilities that are unprecedented, never before seen in the AI video generation space. It's all a part of VDOO 1.5 and it is the world's first multimodal model to support multiple entity consistency. Basically, Voodoo 1.5 can seamlessly integrate people, objects and environments to generate stunning videos overall. And they claim that this is better than traditional single point fine tuning methods like alora. They claim that this is a huge leap forward and unified understanding and generation, setting a new standard in the multimodal AI space. So very much a research focused post here. But what they failed to basically say out loud here is that this is character consistency at its highest level, folks. Or at least the highest level I've ever seen in the video generation space. And not just character consistency, but setting consistency and outfit consistency. So check this out guys. First we have this image of a man. Take some notes about his hairstyle, facial expression, etc. Now we've got this suit of armor. Same thing, it's like this futuristic blue accented silver armor. And then we've got this street at night. Put all of these into v do 1.5 and what do you get? Well, you've got that specific character walking around the city wearing that specific suit of armor at night and it's pretty darn good. The closest resemblance here, I would say, has to either be the suit of armor or the character himself. I mean, the hairstyle is, well, a little bit blurry, very similar to this one. And as you get up closer, not only are the facial features accurate, but also I think the expression, which is so darn impressive. I mean, that is, dare I say, a consistent character. And Vito is bringing it to the table with a consistent suit of armor. Blue accents. We can even see those blue circular accents in the center of the arms. That's also visible as he's walking. And yes, some of the finer details might not be as clear. And that's pretty typical for V DO And I mean, Twitter is not doing this thing any favors in terms of video compression, but it's a consistent character walking around based on all of those separate elements. The setting, the clothing, the character. Totally mind blowing. And this is a first. I mean, has Vitu solved character consistency? I think that's what we need to ask ourselves. And in a reply to this post, by the way, guys, someone challenged Vidu said, can you create with these three reference images? First up, we've got this random character wearing a Nirvana shirt, obviously in the animated style. We've got this ancient artifact as well. And then finally, we've got maybe the place that the ancient artifact came from, obviously some old building. And Vito responds with a very impressive result. Challenge is accepted. Do you like the result? Honestly, it's even more impressive than the first one. You showed off. Right away, we can tell it's that same exact character, that animated style, the black jacket, wearing the Nirvana shirt. And he's talking or just yapping at the camera about something. And I gotta say, the movements look pretty darn fluid, for sure. We've also got that weird stone this time. It's come to life, though, and it looks a little bit worried about whatever this guy is complaining or talking about. And then that building in the background, it's done a really great job at recreating, I think. Still, it's definitely slightly different. The closest thing I would say is the character. Then this is the next closest thing. And then finally the background piece. I. It's good enough, though, that I think it can be used for decent storytelling, and that's really what's important here. I don't know, has VDOO 1.5 came out here swinging and solved character consistency in video generation? I don't know. I can't wait to get my hands on it and really try it out. But it doesn't stop there, guys. It Continues to go with the AI video generation advancements. This is Dimension X. It creates any 3D and 4D scene from a single image with controllable video diffusion. So we can, you know, zoom out in a 3D way. We can go side to side and rotate around this character. But 4D means we can also start time and it can actually move through. So while we're twisting, you know, the candles can be flickering, she can be moving and doing something. Or like in this example, we could just have time frozen. Now, they also have a project page for this which has lots of demo videos to choose from. We've got this robotic arm zooming in and out, and the consistency is pretty great as it rotates and moves things around. We've also got this house. Again, it does a pretty great job. I think the sky is maybe a little bit wonky, but it's not bad. A little bowl here we get nice and close and zoomed into it. I think it loses a little bit of the texture on the bowl as it rotates, but not too bad. Yeah, you can essentially plop any image and it will create it in 3D and you can control the camera. They even took the original Sora video and they orbited right, they orbited left, they zoomed in. So it works with video input as well. You can see on some of the more complex stuff, like these pumpkins, it struggles a little bit. This office, for example, not too, too great. I think the 4D generation is the coolest example though, because they show like the original front video. Then they do the novel view from Video one, Video two, and Video three, where they're slightly rotating and turning the camera and the AI is essentially hallucinating what should be there. This very much reminds me of, like, that zoom in and enhance stuff you used to see in the old spy movies where you can analyze things from any angle you want. It really is like magic. They also have a GitHub here with some stuff in there, but I can't find any specific model license or anything like that. So I have no idea if this is open source or closed source. Moving on. We have some very, very similar technology called Recapture. This is a Google project that, again, does something extremely similar. I think this one's only for video input, but it's able to, yes, create novel brand new angles of any video you upload, essentially hallucinating new details and based on the context of the source video, figuring out what things might look like from a different 3D angle in space. I mean, it is really shockingly Similar, at least in the results in comparison to what we just looked at. But it's good to see so much research is being done into the video generation space and especially with things like camera movement, camera control and the ability to replicate such things. This one, it doesn't seem to have any GitHub links that I'm aware of. So much, much less likely that this is going to be open source, especially with Google around. I would say though, that I think the results from this are maybe just a little bit more impressive in comparison to Dimension X. Next up we have Auto vfx, which can create realistic visual effects from just a single input image and text instructions. And what's great about this one is that it does include the code and weights here on the project page. So this one's actually open source, meaning it's going to benefit the community beyond just being a research paper. And I always love to see that. I'm a big supporter of open source AI technology because it promotes competition, which is best for everybody. But yeah, this is essentially for, as it says, creating visual effects. So what you'll see here in a minute as we rotate around this Pikachu is a fireball come through and destroy that pod in the middle. And Pikachu also subsequently disappears. And then we've got some realistic enough looking physics bouncing off of the table and then onto the ground. And it's also very consistent as it rotates around with that fireball and it continues to burn. We've also got some video footage captured of a road with clearly fake cars from this VFX model with the cars slamming into each other and maybe they're on fire, bouncing off the road, something like that. But yeah, these are visual effects. They look very, very much rudimentary. But this is free, this is open source, this is new technology. And I think my favorite was taking this asset and breaking it down into physics like that. I mean, that is just so cool. Or the ability to toss these little barrels wherever you might want. Next up, we have another one coming from Dreaming to Lupa here on Twitter, by the way, which is a great account for staying up to date in the AI space. But this is fashion vdm. You can take a original video of someone walking around and it can do high quality virtual try on video so you can see what you might look like in some clothes. Literally all it's doing here is combining an image of a specific clothing with a video of a person, but it's able to really maintain as Dreaming to Lupa says the person's identity and movement while showing off exactly what the person is supposed to be wearing. I mean, this is blurry enough where I would just not be able to tell that this is an AI generated video in the first place. Maybe this one perhaps. But yeah, it does a pretty darn good job. And this is just another way that we can see AI video generation be commercialized far beyond making stories or films. Imagine walking into a futuristic clothing store that does this in real time, where you can just select any of the clothes they have for sale and you can, you know, try it on in a virtual mirror that runs something like this in real time. It's definitely not far out. And it's something that we could probably see in the next, I don't know, couple of years, right? This one is also, by the way, by Google Research, and they do have a nice project page that shows this off pretty well. Here is the input garment, for example. I think it does a pretty great job with this. This example, also just a plain black shirt, seems to work pretty darn well. More complex garments, though, seem to have a little bit more difficulty being mapped over properly. And honestly, that is to be expected. It is able to recognize specific portions of the garment though, like this one, for example, where the input video has a very different style of shirt. This one is more low cut, so it's able to go ahead and generate that properly. It understands just from this input image how much more low cut it's supposed to be. It also has to maintain things like other garments the user might be wearing, such as these colorful pants. And I think it does a pretty great job with that. Although it did cut off the white strap or whatever that's on it. It's pretty temporally consistent, though I will say that this is a particularly, I think, impressive example, maintaining the pattern in a decent enough way. Okay, that's everything for the video generation space. Now we're going to move in to talking about some of the things that are going on with OpenAI. We're going to talk a little bit about the new Zero1 model that we're supposed to get the big full version of, not just the preview version quite soon. And also some things that Sam Altman has said quite recently. Now, guys, for the right information on this stuff, there is no better place to go other than Tibor Blaho's account. Another very, very good account that I recommend following, especially if you want to stay up to date on the actual truth surrounding OpenAI rumors, if you guys didn't know. Pretty recently, Sam Altman has Been out there talking to press. He didn't ask me anything on Reddit. So we're getting a lot of information from top people at OpenAI right now. Anyways, when asked about GPT5 and the full version of 01 on Reddit, Sam revealed OpenAI's priority shifted to 01 and its successors over basic GPT development. And I think that was pretty obvious. I mean, we saw these quality improvements, slight quality improvements to the GPT4 set of models, as well as major cost and speed improvements instead of bigger and bigger models. We never saw a GPT5, for example, and I don't know if we ever will. But Sam has cited limited computing resources for parallel launches as the reason we haven't seen a next GPT5. Basically, we're hitting a wall. The more and more compute you throw at a typical large language model like a GPT4, it will get slowly better, but degradingly so. So the model could require twice as much compute, but only be a few percentage points better in the benchmarks. However, this is no longer the case with the O1 models, because now we have another to scale upon. And that is why OpenAI has shifted to the O1 models, because a lot of the intelligence comes out of the reasoning that happens during inference, not in the training stages. OpenAI might move away fully from its GPT naming convention and they might call it, you know, Orion Q, Star, Strawberry, something like that. O2 has been thrown around, so we don't really know what their future models are going to be called, but they're probably not going to be GPTs because they're a new type of model. So yeah folks, I mean, if you needed any more confirmation, OpenAI is very much out with the old and in with the new. In terms of the new reasoning models, they really believe that those are the future, at least for now. And this is further driven home by Adam GPT who works for OpenAI. This is the truth, guys. There are now two dimensions of scaling that factor into models such as the O1 series. Basically models that can think for a period of time before they give you a response. The traditional scaling laws focus on pre training larger models for longer, which is very much still a thing that is happening at OpenAI. It just so happens that we've unlocked a new scaling law to go along with the technology. And of course this is based on a slightly different architecture to produce that chain of thought model. When you combine the overall scaling of computer, especially with the new Nvidia GPUs that are coming and this new chain of thought scaling AtomGPT claims that amazing capabilities are on the way. Even in the original paper and the original blog post they did state that the performance of O1 consistently improves with more reinforcement learning and with more time spent thinking. So what's next? Honestly, in my speculation, I think we have to unlock more and more capabilities. So not only do we train a nice big fat model up to where it makes sense with current technology and scaling laws, but we also have to train it to be a chain of thought model that can think during inference to provide you with a much better result for the same sized originally trained model. I think one of the major things that we are missing in terms of large language models and revealing that next stage of them is their ability to understand things very deeply at a high context level. To achieve a lot of the capabilities that we want to see with large language models, we're going to need models that have the ability to really know who you are, what your intentions are, and understand the context of the current situation. It's not enough just to give it chain of thought capabilities, it needs to more or less be able to hold on to memories for an extended period of time and then consider them in each piece of its decision making. Because that is very much what a human does and it's a huge part of the reason why. If you give one of these large language models access to something like a computer, it kind of still struggles. It can look at your computer screen all at once, but if it doesn't have a lot of the context of how things came to be on that computer screen in the first place, then it's going to struggle. But I'm not an AI researcher guys, so take that with a grain of salt. That's just my personal opinions. I would love to hear what you guys think down in the comments below. I love to have discussions with you guys and and certainly join the discord to talk about it as well. I think a lot of large language model research companies should be thinking less about how to catch up to OpenAI and more about how to one up them. Because as far as I can see from here, OpenAI is still very much ahead in that department. Next up, I do have some image generation news for y'. All. This is Edify image by Nvidia. It's high quality image generation with pixel space lapilation diffusion models. This is a very different kind of architecture in comparison to traditional diffusion models like the kinds that you see From Midjourney Dall E3 ideogram, etc. As Justin means points out, this is pixel space instead of latent space. They show off some pretty cool things on the research page, like some basic decent text to image, some instant fine tuning based off of an initial character, additional control based off of the actual space. So 3D space and then panoramic images as well. The model simulates a resolution varying diffusion process in the time domain by simultaneously decaying different image frequency bands at different rates. I'm not even gonna pretend to understand what that means. Well, that's why we have AI folks. Oh my God. ChatGPT just called me buddy. Alright buddy, here's the your5 explanation for all that big kid talk. It's like he's criticizing me for not understanding it. First, it makes a small blurry version of your picture. Yes, like any diffusion model and slowly adds more and more details until it looks perfect. Now the special trick here lets it work on all the rough parts and smooth parts separately, making everything super clear and pretty. So that's basically what the lapilation diffusion is doing. You can give it extra instructions like making the picture look like it's above or below, controlling how blurry or sharp it looks, and also 360 degree images. That's actually a pretty good explanation of the different capabilities that you get from this. Obviously a pretty decent contrast in comparison to traditional diffusion models. But one thing to note here is that the panorama synthesis they're doing is at 16k resolution. Look at the amount of detail that they're picking up on the tree for example, or right here in the distance. I mean, wow. It does a really, really great job at generating realistic images at a high resolution. Just something that is not easy, but is also pretty much in the funny funniest of ways the next thing that we're going to be talking about as well. Coming from a different research company, there are some issues, I will say with this model. Like the fact that this guy has two different colored eyes and the numbers on this clock, I mean they're barely even legible. So it looks like it does come with some downsides. But at least in the examples they're showing off here, it's seems to have worked very well. Anyways, next up we're talking about. Yes, that's it. More high resolution image generation this time. Faux fur AI. Another fantastic account is testing out the new Black Forest Labs high res model that we just talked about in our last video and using it to generate sets. Here is an emoji set for example of this little ghost character. And it's all very Very consistent because it's all generated in this one single image, you know, where he can be really happy, really sad, looking a little bit worried. And before Flux Ultra, we could only generate something like this in a 1024 by 1024 resolution and then maybe upscale it with something like Magnific. But now we can do it natively, so we get really, really high quality emojis here, which could be directly used on pretty much any platform you want. Right? I can already think of like custom Discord emojis that all match each other. And again, this also ties back into character consistency. We could generate something like this and have the same exact character with a bunch of different facial expressions. And then we could put that into, let's say V do, for example, if you guys remember that one, and then have this same exact consistent character with a bunch of different emotions for whatever scene we can imagine. And even for election season, we have some quite well done Trump emojis. Although I will say this one in the corner with the beard is not that accurate. Anyways, to wrap things up here, to come to a conclusion, AI video is being worked on like there is no tomorrow. The space is evolving so rapidly pretty much for the past like few weeks. I feel like at least 40% of all of the new things happening in the space are related to video generation. And I guess again that kind of goes on to the whole year. Like a good 30%, probably of all the things that happened this year were in the AI video generation space. Still though, I think it all does at the end of the day hinge on the most important, arguably AI model that we've seen so far, which is the AI text generation model. The LLMs that are now multimodal. They can also intake images and output voices and things like that. I'm excited to see how OpenAI leaves this year off. I want to see if they do a nice big model release, something really, really phenomenal. I think the chain of thought models were cool for sure, but it's not like that upgrade from GPT3 to. And that's really what OpenAI needs. They need a leap of that size, I think. So hopefully we see something like that this year. Oh, and on another note folks, for the rest of this week I want you to keep an eye out for some key things. 11 Labs is supposed to have some good announcements this week coming up, and so is Suno AI with their V4 model, which I just talked about in my last video. I'll let you in on a little secret. I got access to the V4 model, and it is dang good. So expect a video on that quite soon. Thank you so much, everyone, for watching today's video. I'll see you in the next one, and goodbye.