Finally! New AI Video ONE SHOTS Tom & Jerry Cartoons w CONSISTENT STORIES!

There is no doubt in my mind that in the last couple of years, AI video generation has gone from a pile of glitchy mush to actual coherent video that sometimes in cases looks exactly like real video. Something you wouldn't be able to tell is AI generated at first glance. While we've made all these strides in quality, there is one thing we have negated and that is, well, all these video generators generate 5, 10, 20 second clips max. What if we could generate a full story all in one go? Multiple minutes long, potentially maybe even longer, comes right out of the model. All your different shots, all your different cuts, consistent characters, that's truly magical AI video. And I believe it's going to be coming in the future. And this paper proves it. So what is this paper? Well, it's one minute video generation with test time training. So, yes, the generations are still pretty short, about a minute long. But this is the first. We're cracking open the case into a new way of doing AI generated video. This is cutting edge stuff, so don't expect perfect perfection everywhere. We'll talk about the technicals, but first I want to show off the video that this AI generated. This is one minute long. And this is just what the model spat out. No editing, no modifications whatsoever. By the way, this AI was trained on Tom and Jerry clips. So watch it tell a one minute long Tom and Jerry skit cartoon story. All right, we open up here, we've got the cars whizzing by. Yeah, the quality isn't great. Tom walks up to an elevator, presses the button to get inside the elevator, and it then cuts to him walking into the elevator. Obviously there's a lot of people. He then puts down some sort of a case that he's brought. He's working inside the building on the computer. All of a sudden, Jerry walks up. Now he's going to mess with the wire to the computer and totally screw it up. Halfway through already I am floored by this. Multiple shots, obviously consistent characters that we train this AI on, but it's telling a story, one that makes sense, one that actually has intelligence. Tom works in the. He goes to the elevator, opens up his case for work, starts working on the computer, and then Jerry comes out of the woodwork and starts messing with it. These are not clips that we separately prompted and generated and then put together. This is just what the AI is putting out. I have never seen anything like this. But let us continue. So Tom is like, what is going on with my computer? It's not working. He kind of gets off. We can see Jerry's messing with it. Tom meets Jerry. Jerry runs away. Tom is now going to try to chase him around the office. He scatters away, goes into his little Jerry hole in the side of the wall. Tom smashes into it in the classic fashion, hurts his head, checks his watch because he's got to get back to work. This is his boss, who is the dog here. So in this clip he's clearly got to get back to his meeting. And the dog's like, alright, you're finally here. This is classic Tom and Jerry storytelling. This is classic Tom and Jerry stuff. Jerry is laughing his little ass off because he knows he just screwed over Tom. And then there is the end of the clip. So it's very short, one minute little demo, but wow. It tells a coherent story where Jerry gets the best of Tom. And that's essentially what we expect from Tom and Jerry, right? A classic little cartoon. Well, this is one shot. The model just spat this out. And that is what I'm having trouble wrapping my mind around right now. How we were able to train an AI on just Tom and Jerry. And now we're able to prompt and say, just make it Tom and Jerry story about this. And I'll give you a one minute long story, fully ready to go now. I'll admit, yes, the quality of the generations themselves. Like if we look at these cars whizzing by, they look terrible. The people look like slop blobs. Yes, that is an issue here. But this is cutting edge research for longer form video content all being done at once. I still think though, like the walking up of Tom here doesn't look too bad. He's got, you know, his work briefcase. This is a consistent object that comes up later in the story. He gets in his elevator, he's still carrying. When he gets ready for work, he puts it down and opens it up. Right. Even though the walking on Tom is not too bad here, but his face looks pretty mash and horrible working on the computer. You know, he's doing typical business work, pretty normal. It's just crazy how it cuts to the computer plug. It's very subtle visual storytelling. I mean, that's what these older cartoons without actual dialogue are kind of founded on. At the end of the day is fully visual storytelling where Jerry runs up, he does a mouse thing, he nods on the wire, ruins his computer. Tom's just trying to have a good day at work, right? And I love this shot here where it cuts back to a very similar looking shot of Jerry. I mean, it's not perfect, but it's consistent enough for the storytelling to make sense. And that's what matters. This is only gonna get better from here. This is just research, folks. Tom looks pretty upset at Jerry, man. Jerry runs away. I love the classic chase scene too where Tom is clearly trying to catch up to Jerry here. I mean it's, it's just incredible. The generation quality itself, like the raw image output is not what's impressive. It's. It's really the storytelling and the coherency over a long periods of time. I thought this was just a glitch the first time I watched this around 2 where he checks his watch, but like he slams to the wall, he checks his watch, he's late for a meeting. It's actually coherent to the story. He runs over into the meeting and there he is. I don't know how to put it other than I'm really impressed. So how did they actually do this and how is it going to be the future of AI video generation? Well, they did this with test time training. They added test time training layers to a pre trained transformer model and then fine tuned it to generate one minute long Tom and Jerry cartoons with strong temporal storytelling consistency. And it's visual storytelling too. The cool part here is they have a bunch of video demos, not just that one, so we can watch a couple more. So if we look at the specific example that we saw first, we can see the full prompt is actually really long. Zooming in here, we can see it's actually prompted scene by scene. The first one with the cars whizzing by in the city, describing the center lobby, exactly what it looks like. Tom approaches the elevator doors and yeah, it goes on and on and on of course, describing exactly what goes on through the course of the story. This is a long prompt. This is a lot of information to take in at once. And it nailed it. Actually listened to the words and just spat this out. So already with this model you can pretty much prompt and generate your own one minute long little Tom and Jerry skits for whatever you want. I mean we could probably get Tom and Jerry abducted by a ufo. What if we use better models under the hood? With this test time training, the possibilities start to open up in your mind. The other thing too is while yes, this prompt is very long and would take a very long time time to actually write out for this model. Hey, we've got large language models that can essentially generate this exact thing. We can literally take this example, paste it into chat GPT and say make me another One of these. But instead this, this and this happen. Just give it some loose details and it will generate the rest for you. And you can plop it into the model. And there you go. One minute long custom Tom and Jerry stories ready to go right out of the AI. Alright, I have got to watch a couple more of these as well, since they have so many examples. Alright, Jerry's got some cheese right in the beginning of the episode here. He's trying to eat this massive cheese. Then Tom is gonna be a jerk, take the cheese away from him and kind of dangle it over his head. You can see Jerry's trying to reach it. It is absurd that this is all just out of the model right away. Wow. Okay, now Jerry's gonna plot something to get back at Tom. He's folding and doing his laundry. Alright. Oh, okay. Seems like he's leaving. He's got a suitcase full of laundry as he walks away. I mean, that's not super coherent. But here's Tom kind of missing his Jerry friend. It looks like he's a little sad, maybe that he's left. Tom picks up the original block of cheese. Again, consistent object with these consistent characters. He walks over. I think he's looking to find Jerry, find some tracks. That is clearly where Jerry probably went. Oh, there's Jerry. Just living on his suitcase, kind of all sad and bummed out. Tom walks over. Is he gonna give him the cheese? He is gonna give him the cheese. So this is a more wholesome one. He gives Jerry the cheese. Jerry is now happy and they can be friends. Once again. He pushes him home in the wheelbarrow that he was actually right next to. So again, another consistent object with the cheese. How cool is that? There Jerry goes, moving back in. So it tells this consistent story over time. Then they're sharing the cheese, prompted in the very same way as the other ones. Let's check out this one for the next example. Tom walks up with a freshly baked pie thing. Looks absolutely delicious. He's starting to eat it. He's starting to enjoy that pie. Jerry must be jealous or something because he's observing this and he's probably up to no good knowing how a Tom and Jerry cartoon goes. Goes to the front door here, and I believe, yeah, he's gonna try to knock on the door, ring the doorbell or something. Runs away. So Tom hears the door, he goes over, tries to see what's going on. Jerry is running around the corner through his little mouse hole. And now he's gonna go try and steal that delicious pie from Tom. So will he be successful? He picks it up. Wow. I love that. It actually got him picking it up. But there's Tom. He sees what Jerry's doing now. Jerry's trying to run away. A classic cat and mouse chase here. And he manages to fit it into that stupid little hole. And there Tom goes slamming into the wall again. Wow. With the classic birds over the head too. And he's just laughing with a full pie. So again, that clear, consistent storytelling. We know what's going on throughout the story. Even with the worst generation quality. I think the storytelling is really what sells animation, motion pictures, whatever you want to call it. Any kind of digital video media. It's really the storytelling at the end of the day. Right. It's just crazy how consistent it is. Like, I know the generation quality is kind of buns. I know it looks a little bit weird when he goes up and knocks on the door, tries to knock on the door and then runs away. It's not perfect, but I just can't get over that long form consistent storytelling with all these objects, these plot points, just how well it conveys it all visually. Tom and Jerry was the perfect thing to train on for something like this. I will say that. Alright, next one. See, we're branching off into some things that might not be so far into the training data. We're going to be looking at an underwater adventure. So Jerry swims over, he's got the bubbles come out clearly underwater. He gets a full map, obviously some sort of a treasure map. Swimming through the ocean. Tom is also under the ocean, looking around. He's trying to sneak up on Jerry. Jerry is not noticing it, but there he goes trying to chase it. Now Jerry, we assume, knows that he's chasing him around, going through all these reeds, going through the underwater area. I really like these scrambling underwater. I like that it's still looks like they're running above water. That's just the animation style from Tom and Jerry there. Wow, look at that. He finds a little hole to hide in as classic. Tom is trying to get in there, but he clearly can't fit. And then Jerry ends up with the treasure map. He goes through this door. Maybe the treasures in there. Yes, it is. So Jerry ends up with the treasure at the end of the day and Tom just wants some. And finally it looks like, you know, Tom trying to steal this treasure. A shark is going to come after him. Wow, look at that. See, a shark is like an extra character that the AI was able to come up with and generate. It definitely inherited some of the characteristics of Tom and Jerry characters. But overall it looks pretty great. I mean, that's something that's probably not really in the training data and it was able to infer and generate. I even love that split screen at the end where Jerry is loving all these gold coins and Tom is just running away from a crazy shark. Alright, we've got another one at the carnival. Looks like Tom and Jerry are both there. Tom has some nice cotton candy. He's hanging out with Jerry. Looks like they're friends in this episode. They're walking over maybe to play some carnival games. There's a red ball here. Oh, I see. So this is trying to show him hitting a red ball and playing like a carnival game to knock over a can. I think the AI had a little bit of a difficult time conveying something so detailed there. Like you can see his paw moves and then the ball kind of moves away and then it rolls past the can. So it's a little bit weird. Clearly he missed though. He didn't finish the carnival game. So he's got another one. Let's see how it does with this. He comes from the other side. Now he's gonna try to throw that ball again. Kind of moving in a very strange direction this time. Almost hits the cam, but doesn't knock it over. Jerry thinks that's hilarious. Obviously Tom does not think it's so funny because he's losing the carnival game. And Jerry walks up, grabs the red ball and he's probably gonna nail it first try. Let's see. He pushes, has a hard time with. With that. There goes the ball. Knocks the can over. Tom is infuriated, of course. Jerry's like, I'm just ripped. You have no idea. And he's got the first place trophy. They walk home again. Consistent storytelling. Like it had a really difficult time with that red ball. Pretty much every scene where the ball actually goes by the can isn't nearly as bad, but it's just having those main anthropomorphic characters interact with the ball object. Right. We just need better video models at the end of the day to up the coherency and the consistency of something like that. But it understands the story as a whole. It understands how the red ball and the can and the trophy and these two characters are interacting. It's making all of this, which is really the astonishing and incredible part. Just spinning it right out of the model like this. Ready to go. A little bit more on how they actually accomplish this though. The test time training layers are more aggressive. They're More expressive. Also, at the end of the day in the story, as they state in the abstract here, the hidden states themselves can be neural networks. And adding these layers into the pre trained transformer enables it to generate 1 minute long videos from just tech storyboards. They even did do some human evaluations here, which puts this thing at least compared to its other research competitor, 34 ELO points ahead in human evaluations of 100 videos per method. And like we observed, the results still contain plenty of artifacts. They blame this specifically on the limited capability of the only 5 billion parameter pre trained model they used, and that's the Tom and Jerry cartoons. And they also say that the efficiency of the implementation can be improved as well. This is super promising, hearing all of that. They also state that they've only experimented with 1 minute long videos and that's what we've seen today. But it could go much longer and it could have more complex stories. They just need more resources and more time to generate them. I also think that these side by side comparisons are pretty telling. Using the simpler local attention traditional method, it struggles with consistency in Tom's color, Jerry's mouse hole, and distorts Tom's body a little bit more over time. But the TTT layers demonstrate strong character and temporal consistency across the entire duration of the video. You can see tested against again all of these research competitors. The TTT preserves temporal consistency over scene changes and across angles. Mamba 2 distorts Tom's appearance a little bit as he growls and chases Jerry. The gated delta net lacks temporal consistency across different angles of Tom over time. And the sliding window attention alters the kitchen environment and duplicates Jerry stealing the pie more than once. So obviously, you know, we're not going to go through every single one of these comparison videos, but everything's always linked in the description below. And again, going over some limitations of the model itself, it's obviously far from perfect here. I mean, it's a 5B trained model. On Tom and Jerry we could improve this a lot. Now the best part about this is that it is indeed open source, but it also builds off of other open source projects. So on GitHub they're including the training and inference code for up to 63 seconds of video generation. But obviously I think the community will modify this thing and try to make it do much longer generation, more complex generation, the process for actually fine tuning the model. You know, specifically on Tom and Jerry, they start with the three second long videos, specifically just for the style transfer and incorporating the test time training layers Then they train in stages of video lengths starting at 9 seconds, then 18:30 and then finally 63 seconds to extend the overall context. And it worked. They do say that their architecture here adapts COG Video X. So this is an open source video generation model. 5 billion parameters. It's not fantastic. It's a diffusion transformer for text to video generation model. And they're basically putting the TTT layers on top of that or incorporating them. It really doesn't look to me like they're giving away the pre trained Tom and Jerry model. So you're gonna have to train your own model if you want to use this thing. But they do obviously provide the code and if you are very well versed in this stuff, you could definitely set it up and run it yourself. But you're gonna probably need to get your own data and fine tune everything yourself. So yeah, we'll see what people end up getting up to with this research. The full paper will also be linked down below if you want to dive a little bit deeper and check out things much more in depth under the hood. But this is incredible. I love to see stuff like this. I mean, what are the future prospects, right? First of all, take this method and apply it to better video generators or better video generation methods than just COG video. Implement it into something that's more robust, has higher coherency, has better training, and you'll end up with Tom and Jerry episodes that look a lot better in just sheer quality clarity and animation. Take that a step further and continue to train it on longer videos. I mean all the way up to 10 plus minutes. You could have full Tom and Jerry cartoons fully prompted at your whim, in your own customized style or way, probably with your own custom characters. And we are not that far away from this. So video generation is absolutely on a steady pace towards a bright future. One that has long context video generation that is minutes long, not seconds long. So yeah guys, what would you get up to with technology like this? If you had an AI video generator that could generate consistent stories up to 10 minutes in length with objects, characters, plot elements, all these different things, what would you get up to with that? And what other bright future use cases do you see for technology and research like this? Let me know down in the comments. Oh, and as always, if you want to stay the most up to date possible, I recommend that you check out my Discord server. It's a great place to not only get the latest info, but collaborate and hang out with other people that are a part of this AI. Community. And also, if you want to stay up to date, check out my Twitter because I'm always reposting stuff like this as well. Thanks so much for watching folks. I hope you enjoyed this one and I'll see you in the next video. Goodbye.