Googles New VIDEO AI 'Magvit' Continues To To SHAKE UP The Industry

So Google Research have done it again. And in this video, we're going to be taking a look at a research paper that was just released which shows a various number of uses for their new generative video transformer. So, as you may know, large language models have taken over the artificial intelligence landscape, but video is one that is particularly difficult, but apparently not so for Google. So in this paper, they call it magvit, a masked generative video transformer. And, and if I'm being honest with you, I've looked at tons and tons of different research papers and so far we haven't actually seen anything like this before. So this is going to be very interesting because the range of applications that we do have here are particularly new. And I do think that once this is released in the future, perhaps next year, it will be a game changer in a variety of fields. So Google's research paper macvid does get a bit confusing. So we're not going to gloss over the research paper too much. It basically goes into the very fine details as to how they built this video too. What you're likely going to want to see is, of course, the results, and they do show that clearly on another cool page. So that's what we're going to show you because the results are truly incredible, considering we haven't seen anything like this before in the video space. So when you load up macvid, you're going to be presented with this page, and you can see right here that this shows you pretty much everything you need to know. Now, one thing that you do find on magvit is that there are simply a huge number of separate applications you can see on this page. There are currently around eight different separate applications that you can use for magvit. So what we're going to do is we're going to break down every single application for magvit, because honestly, a lot of these have wider applications than you might actually think, and a lot of it is in video. So. So this is actually something that is quite early on in its stages of development. So video AI is definitely something that is still relatively novel. So that means that over time we're likely to get increases and further increases in the level of quality of this kind of software. So essentially what we can see here is that this is panoramic video. If you don't know what a panoramic video is, it's essentially a type of video that is very long in its frame. Maybe you have an iPhone, maybe you have an Android, maybe, but likely on your iPhone, there is likely A setting in which you can use where you can take a panoramic photo. And that just means you're taking a long photo of a bunch of images pushed together. Now looking at Magvit, you can see that it states that given a small vertical shot, Magvit can turn it into a panoramic video by applying video outpainting multiple times on both sides. So what we can see here is the first initial shot of the hot air balloon. Then of course we can see that Magvit actually has the ability to be able to out paint 10 times on each side, providing us with a panoramic video. Essentially what that means is, is let's say for example, someone took a video on their phone. And of course, as you know, many people don't take videos horizontally because of how the phone is designed. You could then transform that video using this AI to get it out out painted to get a full scale panoramic video. And of course right now the quality isn't that impressive. But this is the first type of outpainting that we are seeing on videos. You see, outpainting is essentially where you use the inner edges to predict what the outer edges are going to be. And that is something that recently happened with Adobe's Photoshop where you were able to out paint certain images and users were very satisfied with the quality. And this is essentially a video version. So what we're seeing here is one right here and another right here. It definitely looks very, very interesting because this kind of content is going to be super useful, provided in the future. I mean, take a look at what we actually do see here. This is all we are seeing in terms of the initial footage and then this is what we get. I've got to be honest with you guys, that is absolutely incredible, even if it is still in low resolution, because as you know, artificial intelligence does move pretty quickly. Then of course we have the further shots and you can see right here that this is where we get out and painting on five times. So rather than just out painting once and using a huge long shot, this is where we do it five times. And the results are quite surprising. Now whilst it might not be completely accurate, it's definitely really good for merging videos together. You can see right here that essentially what it does is it uses the video and merges it into one larger frame. And this is going to be really effective to for those kinds of videos where you did wish you did shoot on landscape. So in order to out paint like this, the AI has to use the initial frames on the inside and predict what's going on the outside. And from these examples that we've seen, although they are low resolution, the outpainting on five times does look like where you have something that's cropped in 4, 3 aspect ratio, and then essentially you're out paint. Then of course, we have Smart Remover. So this is one of the most interesting things as well. So this says, given a masked video, magvik can generate coherent inpaintings to remove unwanted contents. So remember how in Photoshop sometimes you might want to inpaint something? Maybe there's a hat, maybe there's something on the ground that you want to simply remove and you use a brush to get rid of it. This is basically like that for video. So I'm essentially figuring out that what Magvit probably does is figures out what area it needs to remove and then somehow manages to to use that AI to seamlessly integrate the images together. And what we can see here is a pretty seamless response. Now, of course, as you know, AI video is of course quite still low resolution, but the applications for this are pretty interesting. Here we can see a bunch of eggs on a table. And then of course, we have this black square going across the image, which interrupts what you would otherwise see. And then of course, we have this final image right here, which looks pretty incredible. I mean, I wouldn't be able to tell that there was actually a black square over this. So it just goes to show how. How great Magvit actually is. Then, of course, we have another example here with all of these lines. And it's very interesting in how it actually manages to collect these lines perfectly without any real distortion. So you can see as well right here, with this black line going across this badge, it's able to coherently manage to get this as well. Now, take a look at this because I think you'll find that this part is particularly interesting. So you see where this square is on this person's tie and shirt, do you? You can see that even though Magvit inpaints this, we can also see that there is actually a shadow right here. And this actually looks really, really well done. I mean, when inpainting, it doesn't just use the inbuilt bits, it doesn't just use the outside to generate the inside. I'm guessing that it must understand the context of certain images in relation to one another to be able to generate this, because, honestly, without this first video image, there's no way that the AI is going to know exactly what to put here. So. So I'm guessing that they probably used a large amount of data to train the AIs on what goes where. Then of course we have this one right here with a croissant and of course a cup of coffee. And we can see that using this black box, this is exactly what we see here. And to be honest with you guys, this does look absolutely incredible. So having a black box in the way of something doesn't mean that the video is completely ruined. From magvit. We can now see that you're going to be able to have a smart remover which, where you can simply remove objects in a video in real time. And this is something that we've really wanted and it will be really interesting to see if future applications like Adobe do add this. So here we have something from Magvit called Auto Flip. This is where Magvit can uncrop a video on any sides for an easy switch of aspect ratio. The way this works is essentially you have the original video with the black bars, and then of course, the black bars are eventually removed to present an actual video moving like this. So what we can see here is that magvit easily allows users and individuals to use their original software and then of course, outcrop it. Now what I do find interesting here is, is this technology going to be added to tools like runways, Gen 2 or other video editing services like Adobe Premiere Pro? As we do know that AI is rapidly moving in the direction of being integrated into pretty much every application that there is for increased efficiency. And with this, imagine you have something and it's shot in the wrong size. I know many times filmmakers shoot things in the wrong aspect ratios. Being able to change it very, very quickly just with a click of a button would be incredible. Then of course, we have something called Image to Animation. So it says given a single image, Magvit can turn it into an animation by frame prediction, optionally with action conditions. So essentially what they mean here is that when you use a certain image, Magvit potentially understands what that image is. And because it understands exactly what that image is, it's going to perform the correct animation. Now you have to understand that this means that this is not just a click of a button thing. This AI really understands what's inside the video. So we know that this is a neon type of sign. And as humans, as you know, this kind of sign doesn't blink, it glows. Which is why we can see the difference in the opacity increase and decrease. And I think that that key example shows that this kind of AI software is much smarter than we do think. For example, with this example right here, we know as well that this is of course a Christmasy winter style themed image. If we wanted to animate this, we know that this would be snow and this snow would be falling. And of course the AI would have to understand this as well, where it would have to understand that this tree isn't moving, but the snow is moving. And the snow is not moving up, sideways or left, it is only moving down. Then of course, once again, we have the image of a globe here. And of course, with the image of a globe, we know that this is going to be to rotate. And honestly, I'm not even sure how it manages to do this, because interpolation of 3D objects onto 2D like this is absolutely incredible. So this is truly interesting being able to do this. I mean, honestly, it's quite surprising. And then of course we do have the waves here, and it does do a very good job at the waves here. The only problem that I would say Magvit suffers from is of course quality. If there is going to be an artificial intelligence tool that many programs do use, it would be great if there was one that could increase quickly any low resolution video into a high quality one. That way when these AI models output these videos, we can get them into a higher quality pretty quick. Now, this example wasn't as good, but it does show that the future applications are going to be interesting. So it says, given two images, Mavic can turn it into a stop motion animation. But by frame interpolation, that's essentially where you have two images and it's easily represented by this one right here. We have the brown pencil over here, then we have the brown pencil move to the right. And of course, once we have these two images, Magvit is then going to combine these two images and then of course move them in a specific direction. So using what it knows about two images, it can then combine these two images to create a specific video animation, which would be very, very interesting. I mean, many different applications imagine a video is corrupted, or imagine you just got a few images of something happening. It would be very interesting if you could get that synthesized into a direct video. So right here you can see someone putting a pen inside of a cup, which does look not great, but still it's the application of these that we are looking to work. And of course, this one was my favorite, where it's someone pouring water into a cup and then of course removing that from a cup. And, and I do find that the animation of the water was actually pretty decent. Now this was one of the best things that I did see from this, and this is arguably what we should have placed at the start of the video. But I think this will be the main talking point for Magvit, because given a single image, Magvit can turn it into an animation by frame prediction, optionally with action conditions. So essentially given a single image, it can make a video from that. And what we see here is that this is largely in driving dashcam videos. So of course it's going to be very hard to predict other videos because absolutely anything can happen. But certain styles of videos do have a certain set of preconditions that usually do follow. For example, when a car comes to a junction, it usually slows. And the AI knows that a car usually comes across. And right here, this is usually where you are driving and it continues to move down the road. The same goes for this one and the same goes for this one right here. So it's really interesting as well, because even when we look at this one of the car turning, you can see that the car manages to turn onto the straight road and then generate what appears to be a new kind of road. So that goes to show that future prediction. I mean, you have to understand that right now it's very, very early. And for those of you who think, okay, this is early, this is terrible, what is the point of even covering this? Understand that right now this is how good AI is. Remember, GPT2 was released in 2019 and since then we've made huge, huge, huge gains in terms of artificial intelligence. I mean, in 2022, ChatGPT was released and you all saw how good that was. So within the next five years, imagine this kind of technology is absolutely perfect. Let's say it's on mid journey level. Future prediction, from images, image to animation, auto flipping, changing your perspective, being able to remove objects really quickly, being able to add a panoramic video from a single iPhone shot. This kind of stuff is going to be absolutely incredible. And I think this is really, really cool. And since this is by Google, one of the things that might not be as cool, but is definitely good for us is that video compression. So what you're seeing is two different videos with various different compression rates. And what they're basically saying is that one of them is 600 times smaller than the other, but you don't actually notice a visual difference. So this will be interesting because I do know that one of Google's largest expenses is video hosting. So I think that they'll be expediting this as quickly as possible, because if they do manage to get this working as quickly as possible, then they can seriously make a ton of money.