Metas 'MovieGen' AI Just SHOCKED The AI World (Text To Video AI)

So Meta is introducing Meta's moviegen. Take a look at this short video and then I will dive into all the specific details that might surprise you. Sa, Now that you've seen that short.

Video, I'm sure you can understand that.

Meta are clearly one of the underrated leaders when it comes to AI. Meta's moviegen actually comes to a surprise to many in the AI space. So now that you've all seen that short video for Meta's moviegen, you can largely understand with as to why many are surprised. In the short video it actually was compared to other state of the art systems like Runway Gen 3 Alpha, and we can see it outperforms it on a variety of different tasks. This was something that is rather surprising to many individuals in the AI space like myself, because we didn't really expect Meta to come out with a foundation model for videos that was going to be surpassing other companies that are solely specifically focused on those videos. So this was something that took us off guard because whilst we thought that Meta was busy with Llama 3, it seems that they were also busy with generating videos in really high quality. We can see that there are many examples of this software even being compared to Sora and in many cases currently excelling and surpassing what the model is able to do. It seems that Meta has innovated in its efforts to do this effectively and and somehow they've managed to come out on top. This is one of the most surprising updates from AI, and I think this is one of those examples where it showcases that this technology is advancing once again more efficiently than we could have predicted. Many of these examples are simply outstanding. But let's dive into the four key features that Meta's moviegen has to offer quickly. I'm going to show you guys the model sizes so you don't get confused. So, and then we're going to dive into all of the features. So at first we have a 30 billion parameter model, which is the main MovieGen model. Then we have MovieGen Audio, a 13 billion parameter model which generates audio clips. Then we've got the personalized moviegen video, which is a post training extension. And then we've got Moviegen Edit, which is where you can edit the videos. And this is another post training extension.

So it's two models, and one of.

Those two models has two extensions that allows you to customize your videos. Now you can see right here that we're going to be taking a look at some of these video examples. And during these videos, what I want you to do is to Pay attention to the things that you might not. For example, pay attention to the lighting, pay attention to the physics, because these are things that video generators particularly do struggle with. For example, in this video, I think that the physics and overall lighting do look really, really good. It is quite hard to get lighting really done well in AI video generators because lighting is something that affects many different things. And this is something that is quite hard to get realistically when you've got many different surfaces, many different reflections. And in this video right here, we can see that the lighting around his body, the lighting on the ground, actually managed to match up with exactly what we are seeing in front of him. So those are the things that I look at when I'm analyzing video to see if it is actually effective. And this is something that I think is definitely up there. The text input for this summary was the camera is behind a man. The man is shirtless, wearing a green cloth around his waist. He is barefoot with a fiery object in his hand. And he creates wide circular motions, a calm seas in the background. And the atmosphere is mesmerizing with the fire dance. So you can see at the prompt adherence, it is actually rather really high. We have another example here, and if I'm being honest, out of all the examples showcased on the meta website, this is by far my favorite. We just have a sloth that is drinking some tropical drink and seems to be as relaxed as one can be. The text input is a sloth with pink sunglasses lays on a donut float in a pool. The sloth is holding a tropical drink. The world is tropical. The sunlight casts a shadow. And I think this one is really nice because most people might not pay attention to the background area right here on the left hand side, but we can see that the water reflections on the left hand side and the right hand side look remarkably effective, which is like I said before, one of the key details that I do pay attention to. So also what I am paying attention to as well is the shadows. Now I didn't notice this at first, but upon further analysis, something that I found that was really cool was that as this sloth is moving from right to left up the water, you can see that the shadows actually contrast on its face. Like we can see that the shadow and the sun rays coming in and out and it manages to dynamically change the lighting based on it being underneath a tree. And we can see that these shadows actually look effective as they are tracked underneath this donut. And that was something that I thought adds to the realism, which is why I think that this model is really effective. Sometimes models may struggle with these small things and we might not notice them. But when we have everything working in our favor, these are the kinds of things that make a system like this look super realistic. We also have some more wild animals. And this is a prompt that says, a red faced monkey with white fur is bathing in a natural hot spring. The monkey is playing in the water with a miniature sailship in front of it, made of wood, with a white sail and a small rudder. The hot spring is surrounded by lush greenery with rocks and trees. Another really effective demonstration of what we can see here. There is some very, very minor morphing, but this isn't something that I haven't seen from other models. It honestly is one of the most surprising videos that I've seen in quite some time, considering how hard this would be to get done correctly. For example, if we actually look at, number one, the reflections of the monkey, we can see that they are quite accurate. And of course, the reflection of the boat, we can also see the ripples as it manages to move forward, which is really nice as well. And we can see that the legs of the monkey are also there. And we also do see the hand just underneath the surface of the monkey. So this is something that is really, really effective. I mean, it's incredible that this technology is going to be available in the near future. And other companies are also working on other iterations of this. But like I said before, when we really take a look at the details and realize that what we are watching isn't real, it's something that should not be overlooked at all. There's one last clip here of a girl running across a beach holding a kite. She's wearing jean shorts and a yellow T shirt. The sun is shining down. I think this one is once again really effective. And I think what this clip wanted to showcase was two things. First off, they wanted to showcase the fact that it has really good dynamic physics. One of the things that I've seen from video editors constantly struggle with is the ability to actually have legs that run rather effectively. Sometimes in video clips that are longer than two seconds, we will see the legs morph. Sometimes we will see them look like they're speeding up and slowing down. And this issue is just not present in this video at all. And I think that was something that they wanted to showcase as it is quite hard to get right. There was also something else that I picked up on that I think you may have missed. One of the things that you can see is that as the girl manages to walk in the sand, we can see that there is the footsteps that are picking up the sand. And we're seeing that these footsteps are being left as those feet are being placed in the sand. So another example of really accurate physics detection, which is really nice. Now, I do want to say quickly, before we move on to the video, this video demonstration, it didn't actually showcase any areas that might have failed. So if there are demos out there, for example, when we do get access to this, maybe we're going to see the areas where it does fail. But right here, this is a really nice example. Now, I think Meta's movie Gen is severely underrated because being able to have these kinds of abilities immediately is something that is incredible considering the consistency of of this kind of editing. So edit with video text is here.

And we can see just how effective this looks. We can see that you're able to have the original video in the top left.

And there are three other iterations of the video. We can see adding fire sparklers to his hands. And I think, like I said before, once again, let's take a look at the lighting. As his hands come down. As his hands come down, if we pay attention to his face, we can see that his face gets a little bit more orange in order to account for the lighting coming in. And then as they move away, you can see that the lighting isn't there as much and it's correctly shaded in the sense that we have the light down here and we can see all of the areas of his face that are lit correctly, which is really, really nice. For example, here as well, we can see that the background maintains its perspective, which is really, really effective. We can see the background also isn't in focus, which adds to this kind of realism effect that we get when looking at this video. And I think it's really, really nice with how that managed to work. And of course, you can see right here, the last one, change the sky to the northern light, which is really nice. Now, I will say in this one, it does say change the background to an outdoor stadium, but it doesn't say change his shirt to a blue shirt.

So I'm not sure if the model.

Was meant to do that, but nonetheless, it's still looks great. There's another example of this as well, and I think this one is definitely one of the most effective. We can see that the original video looks really nice of a man running in the desert. And then we have text input, add blue pom poms to his hands. Not only do we have the pom poms added to his hands, but we do have decently accurate physics of how these pom poms would be looking if this person was running with them. We can also see you're able to turn it into a cactus desert. You're also able to replace the running clothes with an inflatable dinosaur costume. And the best thing about this is that, like I said before, it doesn't just adapt the video to the other video. It actually makes it look exactly like how it would be if that person was wearing that outfit. Because if you've ever seen an inflatable dinosaur costume before, you'll know that its head kind of bobs in that exact way, which is really funny to see it happening. And then, of course, for the pom poms one to see him running like this, it's pretty cool to see how each video differs with subtle changes.

Now, this is where we start to.

Get into the area where you can start to see how this may have.

Applications for future uses, such as generative video effects. For example, we can see you're able to add pouring rain right here. This does look somewhat realistic, I will say, in this example right here, we don't get the added physics of, you know, the clothes getting wet and all those minor details. Those things are going to be pretty difficult to add. So it's likely that we will get that in the near future. But the other details that we do get, such as changing the background to a carnival, it looks really nice in terms of the effects that we have in the back, the blur that we get, and of course, making them wear 3D glasses. I think this one deserves more credit because the reason this one is so incredible is because if you've ever tried to 3D track something onto an object, you'll know that the software tries to get all of the data points on that object. So in traditional tracking, when you try to track something, what will happen is you will try to track multiple different points, and then you'll pin a 3D object to that. And with AI, it seems that we're going to be able to get software that's just going to be able to automatically do that, which is just incredible because it's going to save people a lot of time. We also have one last example here of the text to prompt area, and we can see again at just how effective this is. Now, I do think that this is something that is remarkably effective. I think it's going to have so many different use cases, and I think in the future. This is going to be how video editing is done. Surely when you're in Premiere Pro, maybe you're in your editing software, whether it be Capcut, whether it be DaVinci Resolve, you're going to be having buttons where you're just going to be able to say, okay, let me just change this, let me just change that. And I think this is definitely going to change how certain movies are made. It might be a few years before we get it completely polished, but this is most certainly a time saver, especially right now, for those who are indie filmmakers. Another example that we do have that is quite fascinating is the personalized videos. Many different AI systems allow us to personalize things with personalized images. So it's no surprise that Meta has come out of the gate with personalized videos. We can see that it's able to simply use one image here, and it's able to accurately resemble what this man looks like. I think the fact that it's able to just use one image is something that's really, really cool, because usually you'll have to have many different images in order to get this looking remotely effective. We can also see some more examples of this personalization and effect, which is really nice. I'm sure certain content creators are going to be really enjoying this. I'll definitely be testing this with a few people's faces. Maybe we're going to see some funny videos of Mark Zuckerberg riding horses or doing something that he probably otherwise might not be doing.

So next we're going to take a.

Look at one of the best things to actually come out of this. And that was actually something that caught me off guard, which is the video to audio feature in moviegen. Now, I've seen a few examples of.

Software like this before, but this was something that was rather fascinating. So basically what we have here is.

A system that was trained on a massive data set of videos with matching soundtracks.

And the model has learned how to.

Match up different sounds to different specific videos. And I'm going to show you guys a few videos and then I'm going.

To talk about in more detail how this works. But I first want to show you.

Guys the different sounds that was generated.

By the model when looking at video as the input prompt. It.

So hopefully, guys have seen that you can see that the audio was really good. But how this works is that the model takes in the video frame and then processes them to predict sounds that should accompany those visuals. For example, if a video shows a car driving, the model will generate sound effects like the car engine, the tire squealing or ambient city noise. And it does this by understanding what actions are happening in the video and then generating the appropriate sound effects. Now, the system is capable of generating different types of audio. So we've got diagetic sounds, and these are sounds that come from directly what's.

Happening in the scene.

And then of course we've got non diegetic sounds. And these are sounds like background music or mood setting sounds, like intense music during a car chase scene that fits the tone, but it isn't actually happening in the video. Now the movie gen model can generate sound at high quality, specifically 48 Hz, which is the standard for cinematic sound. This means that the audio generated by this model is clear and of professional quality, making it suitable for use in films, games or other media content. The model not only matches the sound with the video, but also produces long coherent audio tracks for videos that are made up to several minutes long. And it's designed to create sounds that naturally extend over time so the audio feels continuous and realistic. The audio model was trained on millions of hours of video and sound data. So during the training it actually learned learned the physical relationship between what's happening on screen and of course the sounds that should be produced. Of course, it also actually learned the psychological aspect of audio in video, such as how specific sounds can heighten emotions, create tension, or establish the mood of a scene. The system manages to generate really effective sounds, like a splash when someone jumps into water, or other things that are really effective. And in order to get this to actually be as effective as it is to where you can hear like the firework exploding as it explodes. After the initial training, the model was actually fine tuned with a smaller set of high quality video and audio data. And this fine tuning step helps improve the overall quality of the generated sounds, making them feel polished and more cinematic, similar to what you'd hear in high end production. So overall I think this is a remarkable step because we have something that could pretty much automatically do soundtracks and background music, and it's something that I just didn't expect with this level of quality.