Googles New VIDEO AI 'VideoPoet' Surpises Everyone!

Okay, so Google actually just released a amazing research paper that discusses their new text video model. This is called Video Poet. Now essentially it's really incredible because of some of the features that they do show us. And in this video it's going to be a deep dive on why this is truly game changing if Google actually manages to make this into a full fledged product. Because, because oftentimes when Google always does these crazy AI breakthroughs for some reason, I honestly have no idea why they never make it available to the public or even develop it into a fully fledged product. So here you can see this is Video Poet, a large language model for zero shot video generation. And essentially the zero shot in zero shot video generation just means that it's the ability for this large language model to complete the task without having received any training examples. So you can see here, this is some of their first examples and later on I'll show you why, although the quality might not be as good as you think, it's really really good. So you can see we've got a dog listening to music with headphones, highly detailed ak, a large paint blob exploding. This one actually does look pretty good. A robot cat eating spaghetti. Now I'm going to get into each of these examples and show you why they're actually really good in terms of the accuracy. And also what I'm going to be showing you as well is how this actually does compare to some of the other popular video or slash text to video models that do exist out there because honestly they're really, really great. So essentially one of the first things that we do have is that VideoPoet can output high motion, variable length videos given a text prompt. And one of the crazy things about this is that they do have video to audio which is, I guess you could say kind of weird because it's not something that we've seen before. We only saw video to audio in I think one model that was Kodi, which was any to any, a multimodal model which essentially any form of modality could be converted to another. So audio could be converted to text, text could be converted to video, video could be converted to audio and vice versa. But anyways, and yeah, this is the paper that I'm talking about just for reference. So Google integrating maybe some things from this and is going to be really interesting. But anywho, back to Google. So let's actually test out how good this video to audio model is because although this is something new, I think the abilities are going to be surprising. So this one is a Dog eating popcorn in the cinema. And let's put that audio on. I mean audio definitely sounds a little bit raspy, like it needs a bit more. A bit, I guess you could say clarity. But I'm going to just go through these one by one and then I'm going to discuss them after. So this one is a teddy bear with a cap in leather, drums playing drums. Then we have a teddy bear in a leather jacket, baseball cap and sunglasses playing guitar in front of a waterfall. Then we have a pink cat playing piano in the forest. Then we have the Orient Express driving through a fantasy landscape on an oil canvas. Then we have a dragon breathing fire. Cinematic. So I mean the audio isn't actually that bad. And I think if this does get really good, it could present another area for, for us to access a different kind of modality where we do have these two modalities combined and it does help us produce much better content. And essentially they actually made a video with this generative model, so it says. To showcase video poets capabilities, we have produced a short movie composed of many short clips generated by the model. For the script, we asked Bard to write a series of prompts to detail a short story about a traveling raccoon who. We then generated video clips for each prompt, stitched them together, resulting all clips to produce the final YouTube short below. So take a look at this clip. But like I said, although the polynomial results might not seem that crazy, I would say watch the video until the end, till you see why this is kind of incredible.

The Raccoon in the Amazon raccoon forest lives Rookie the raccoon and his family. Rookie loves to play soccer with his friends, collect acorns from the trees, swim in the river and catch fish. But what he loves most is exploring new places and sniffing new smells. Except he's never made it out of the vast forest. Every night he dreams of seeing exotic places like Times Square and the Golden Gate Bridge. One day he sees a portal of yellow light and his curiosity makes him step inside. Suddenly he's falling through a long tunnel which takes him to Paris. I've never seen anything like it, he says. And then to the Great Barrier Reef and even the South Pole, where he makes some new friends before visiting the ISS and ultimately the planet Androida, where the locals didn't speak raccoon, which makes him miss the raccoon forest and especially his family. Rookie quickly finds his way back home. He's so happy to be back with his friends, but also grateful for his adventures. Guess he's not such A rookie anymore.

So like we said, that was an impressive demo. But let's get into some of the technical stuff so you can understand what makes this really, really impressive and why you should be paying attention to this. So it says Video Poet is a simple modeling method that can convert any autoregressive language model or large language model into high quality video generator. It contains a few simple components. A Pre trained Magvit V2 video tokenizer and a Soundstream audio tokenizer. Transform images, video and audio clips with variable lengths into a sequence of discrete codes in a unified vocabulary. Now we actually did a video that was on Magvit before, but essentially Magvit was really, really cool in terms of its ability to do some new things. Essentially this is a video from a couple of months ago, but essentially it could out paint five times, showing us how to do a panoramic video. You could also see here that was some shots that you could take. Like a single video, if taken in portrait mode, could be converted to a very long landscape mode. So you can see that there was also this smart remover feature where if there was a box in this video that was somehow cut out, the Magvit transformer could, the Magvit system could replace that box. You can see here the box is replacing some of the images with exactly what it was. And this was essentially video. Then of course we had some more examples. We also had auto flip which is uncropping any size of video. Then it also had image to animation. There was something that was really cool which was frame interpolation which is taking two images and. And then of course animating what's in between those. Now here you can see this is where they give us an overview of the video poem model. It says an overview of the Video Poet model, which is capable of multitasking on a variety of video centric inputs and outputs. The LLM can optionally take the text as input to guide generation for text to video, image to video stylization and outpainting tasks. And you can see right here it takes a text and then it can do text to video. So it's got this whole system here that works cohesively together to produce these good videos. And of course the only thing that we do have that is not as good is of course the quality. But anyways, let's move on to visual narratives, which is quite the thing. So you can see right here they have this initial video of a walking figure made out of water. But then they can extend that video and use a prompt to change that exact video. And one thing that I want you to pay attention to here is the accuracy of video poetry, because a lot of people might think that this is low quality or the FPS isn't as good, but you really do need to see how accurate these prompts are with regards to what was said. So this is a walking figure made out of water. Lightning flashes in the background, we can see that lightning flash and then purple smoke emits from the figure of water. And you can see that's exactly what happens. And when I compared this to some of the other video models, they actually aren't as accurate as this. And, and then of course, we have two raccoons on motorbikes on a mountain road surrounded by pine trees in 8K. This one's pretty decent. And then we have the extended video of two raccoons and the meteor shower falls behind them and the meteors impact the earth and explode. And you can see that the explosion actually looks pretty decent. Now I think this is the biggest thing about Google's Video Poet that most people might miss is the fact that this can actually produce very long videos. So it says by default, Video Poet outputs two second videos, but the model is also capable of long video generation by predicting 1 second of video output given an input of a 1 second video clip. This process can be repeated indefinitely to produce a video of any duration. Despite the short input context, the model shows strong object identity preservation not seen in prior works as demonstrated in these longer clips. So you can see here these clips are around, I think around 10 seconds long or so. When I open them in a new tab, they're around 10 seconds long or so. And, and essentially what they're talking about is the fact that the object does remain in the image. Then of course we have some of these examples. And depending on what you think, I think this one and the Teddy's walking are the best. A drone footage of a very sharp elven city of stone in the jungle with a brilliant blue river waterfall and large steep vertical cliff faces. Teddy bears holding hands walking down rainy fifth Avenue. And an SPV drone footage entering a cyberpunk city at night with many neon lights and reflective surfaces. And then this one of an ancient city in autumn actually looks pretty cool, although it is quite trippy. Now something that I also did find cool from this video poet from Google was the interactive video editing. It says interactive video editing is also possible. Extending input videos a short duration and selecting from a list of examples by selecting the best video from a list of candidates. For we can finally control the types of desired motion from a larger generated video. Here we generate three samples without text conditioning and the final one with text conditioning. So you can see that they first text so you can see that they first prompted the model and then this is what they got out. And then based on these three samples, they decided to add this text here. And you can see that it shows that when you add the text prompts, it actually becomes much more accurate. And this one here does look very, very accurate in terms of what we do see. They also do talk about controllable video editing, which is essentially the Video Poet model can ed subject to follow different motions, such as dance styles. And here we have a raccoon doing very different dance styles. I don't think these dance styles are that accurate because of the video model not being that accurate. But I do think the fact that Video Poet does actually have longer video generation where you can generate clips of any duration, this definitely does have some deeper applications. In addition, what was also cool that many people may have missed is the stylization aspect of VideoPoet, which is where they said videopoet is also capable of stylizing input videos guided by a text prompt. And it demonstrates stylistically pleasing prompt adherence. So you can see we have the inputs, and then the stylized version of these inputs are very, very incredible. This is very similar to what we got from Runways Gen 2, in which you could put a style onto the said video. Although I do think that somehow this does seem to be more accurate. I don't know about you guys, but this input of a lion and then transforming it into a metal lion, it does look really cool. And then this one here of a wombat, and then you change it to a wombat wearing sunglasses holding a beach ball looks so cool. And of course, this one right here, a magical snow forest covered in dense pine trees. So I think this stylization is very effective. And then of course, we have applying visual styles and effects. Styles and effects can easily be composed in text to video generation. And we start with a base prompt and append a style to it. The prompt that they have is an astronaut riding a horse in a lush forest. And then you can see the different styles that they have. And so far out of the prompts that they did have, these ones here are the ones that look the most effective. The photorealism one definitely looks really cool. If it had more FPS, Then the digital art style of the astronaut sitting on a horse, then the pencil art style and ink wash style, then the Double exposure style looks absolutely incredible and then the small world style looks even more incredible. So it's absolutely insane the visual styles and effects that you can use for Video Poet. But like I said before, will Google actually give us this to use and do things with? Now, of course, previously, like we did discuss with the Magvit video that we did in the past, they do actually have inpainting and outpainting. So Video Poet can add detail in masked out proportions of the video, optionally guided by text. So you can see here that the outpainted video is done, the outpainted view here is done. And you can see also it does have inpainting like we discussed before. So you can mask a piece of an image and then of course you could essentially replace that with anything. Now this is what I'm saying about the applications of this. Think about if you, you know, for example, have a video of someone doing something, then put a mask in and then you could literally put anything in there. For example, we have this person riding a surfboard in the ocean, then put a mask on it and then you instantly put a shark on there. So I think that the applications for this, once this does get really good, are absolutely outstanding. Now in addition, one of the last things I do want to discuss is of course the comparisons to other software. So we do have image to video. And I did actually test this with some of them. And it's interesting because Google's research, Video Poet does actually seem like it might be a bit better in some areas than the other software. And the one I'm going to be comparing it to is Pika Labs. But it goes to show that they clearly have different architectures. So what we're showing here is image to vineyard generation. And of course you could test this for yourselves by going on the Video Poet page and just taking the images and then inputting them in. Now the image to video does look pretty more decent because I think the image qualities are just that much better. But for example, you know, this image of us, you know, a ship navigating the rough seas with several passengers on board, thunderstorm lightning, animated or canvas looks really good. And a green man riding on a green horse with the wind blowing also looks pretty good. And then this one also looks really cool because of the way how the flag is being waved. And of course I did actually try some of these. So for example, right here you can see white milk splashing in a ring, drop above, the ring falls down making a splash. I actually did try this Here and you can see that it didn't seem to get exactly what I said on the first try. So I did actually try and reprompt this again. But you can see it's white milk splashing in a ring. A drop above the ring falls down making a splash and. And it does have some kind of splash here, but it doesn't recognize exactly what this is. And then make that move. And then this was the second try that I did do with Pica Labs and you can see that nothing really happens. Now this isn't to go ahead and say that Pica Labs is terrible and Pika Labs is bad. It's just to go ahead and say that these different architectures will result in different outputs. And I think it's like how with Claude is more mainly used for really long form, document and creative writing where we have different large language models for different purposes. We're definitely going to have different video models for different purposes too. Now one of the last things that was really cool from VideoPoet was the Zero Shot Controllable Camera motion. So one emergent property of video poets pre training is that a large degree of high quality camera motion customization is possible by specifying the type of camera shot in the text prompt. So the prompt is adventure game concept art of a sunrise over a snowy mountain by a crystal clear river. And you can see that from this single prompt they were able to generate different kind of camera angles. So we had the zoom out, we had the dolly zoom, we had the pan left, we had the arc shot, we also had the crane shot, we also had the FPV drone shot. So I think this kind of thing goes to show that their video model is clearly really diverse and is able to do a lot of stuff. The only thing that I would like to see now is number one, Google finally for some reason releasing this and then of course them being able to fine tune this with a higher quality in terms of pixel density and resolution. And then of course more frames per second. I know that it is not easy to do, but of course these are some of the steps that we need to take if we are going to get text to video that is actually usable.