Runways Text To Video "GEN 3 ALPHA"" Actually STUNNED The Entire Industry!

So Runway actually just introduced their Gen 3 Alpha. It's a new frontier, high fidelity, controllable video generation model. It is truly, truly impressive. And trust me when I say there are a few details that we're going to go over that will show you why this is so impressive and of course, what sets them apart from the other video models. So you can see here that they state that Gen 3 Alpha is the first of an upcoming series of models trained by Runway, a new infrastructure built for large scale multimodal training. It is a major improvement in fidelity, consistency and motion over Generation two, and a step to building general world models. We'll talk about general world models later, but I truly do find this intriguing because some of the key examples that Runway are going to show you, they show us a lot of different things that are really, really impressive in terms of what the model is able to do and. And a few things that I literally haven't seen any other model do. So let's get into things. So one example that we can see here is, of course, one example that we can see here is an astronaut running through an alley in Rio de Janeiro. I think this one's pretty good. And as someone who, like, pays attention to how videos are produced, what I.

Always want to do is focus on.

The subtleties that are apparent.

Like, for example, you know how, of.

Course we can see that the hand is morphing a bit right there. I'm not sure if you guys can see that, but if you do focus on the hand, you can actually see a few mistakes in terms of the morphing. That's not the kind of things that I want to focus on. What I do like to look at is things like the reflection and of course, the background in terms of how it is moving with the times, because not with the times, but like with the actual motion. So those are the things that I.

Do want to focus on.

Okay, and what we can see here, flying through an underground neighborhood, this. This is something that looks really, really nice. Like this one right here is of course, generative in the sense that, like, you know, you don't really get these kinds of footages within the training data, and combining them is going to be really, really difficult. Now, this is arguably one of the most impressive ones, and I think this one is really impressive for a few reasons. So, like I said, I always like to focus on the subtleties of the model. But one of the things that we can see here about this clip that makes it just so impressive is the fact that it uses dynamic lighting so you can see here that at the start, there's not that much lighting in the background. And so the character's face, like, I.

Truly do, like, if I saw this.

I wouldn't believe this was AI if no one can tell me. But you can see that it is dark. And we can see that the lighting from here, it's not on the character's face. Like, these areas right here, we don't see that kind of lighting that you would expect, but there to be if there was a light. But as soon as we get, like, you know, the darkness right here, you can see that things go completely dark. We can see the shadows, and everything looks pretty accurate, which is really nice. And then, of course, as things start to brighten up, we can see that the lighting changes to accurately represent what we're seeing. So you can see that the yellow, you know, hues, the yellow, you know, which kind of looks like a sunset kind of thing. We can see that it's, you know, going onto all these kinds of features of the person's face, and it's just really, really accurately mapping how the light reflections are. Of course, we can see as well at the back. Something that I find really impressive is, of course, how the light breaks through the hair right here. We can see that the light looks really effective. And then as well on the skin, we can see that the lighting is also pretty impressive. And around the edges of the skin, we can see the impressive lighting features. I'm not sure on the exact fluent.

Technical terms of them.

You know, I could get some of them wrong. But what I'm saying here is that this dynamic lighting, where we're seeing the face in real time, be able to, I guess you could say, adapt to the lighting shown. I just think this is truly, truly impressive as all of the different backgrounds come into play. So this is definitely one of the most impressive scenes. This is something that not only it has the background moving, moving, you know, really, really nicely, and the character in the middle, they stay extremely, extremely consistent. And the lighting behind the scene, it really does update it. Now, this does kind of remind me of that one Sora clip, but I would argue that this is just a little bit more impressive because we're seeing at how much the lighting is accurately done here. So this is something that is really, really impressive.

And honestly, a very, very good example.

Of what makes this video model incredible.

Now, now, they initially speak about how.

Trained jointly on videos and images, Gen 3 Alpha will power runways text to video, image to video, and text to image tools and Existing control modes such as Motion Brush and Advanced camera controls and of course Director mode, as well as upcoming tools for more fine grained tool for more fine grained control over structure, style and motion. And this is one of the things I like about Runway is because they.

Actually allow you to, I guess you.

Could say, focus on what is relatively important. Runway. If you don't know they have Motion Brush and they have Advanced camera controls as well as Director modes. And these modes actually allow you to do a lot more things than a traditional text to image generator. That's probably why they haven't released this just yet, is because the Motion Brush feature, they probably haven't updated it and integrated it into the Gen 3 alpha. Of course, with the Advanced camera controls, that is going to be something that they'll need to work on as well. But I do personally think that Runway is probably going to become the one stop shop for everything text to video because it's very, very impressive on certain things that it's been able to do the existing control tools that people will actually want. I do believe that, you know, this is going to be something that allows more controllability. If you're someone that uses generative AI, then the problem with a lot of these clips is that there's not a lot of control with as to what happens in the clip. You enter a text prompt, you can describe it as best as you want. But the problem is that having real control over where the actors may go, over where the lighting it goes. Like for example, if you wanted to just change out the yellow light for a blue light.

Those kinds of things are really hard.

To do in video models.

Now I know that Sora did have.

A demo which where they showcase that, you know, you could switch a car and do stuff like that. But I do think that Runway is positioned better because their thing has been focusing on that from the beginning. So they're likely building these models with that in mind. So you can also see here some of the more impressive ones. There is also one that I really want to show you that is more impressive. You can see here that this is a train going around a track. And the temporal consistency is relatively impressive. This one is one of my favorite examples. This is a close up of a living flame.

And it looks really, really effective.

Like we can see all of the different lights going on here and we can see the background. The characters do look pretty consistent in terms of their motion. And I mean something like this is relatively hard for things to do. And what's very impressive about this is.

That this was only released just after, you know, and this is just the demo. This was only released just after Luma Labs video. And it really does show us that.

We are advancing across the board here, because this is far more impressive than Luma's AI video. Of course, right now, Luma does give us access, but I think Runway's presentation here is truly, truly impressive. This is also something that I thought was pretty incredible. You can see that we start in this abandoned area, and then this thing pops up for it to be covered in vines, and then literally, it's like a whole new portal into another dimension, which is pretty, pretty cool. So overall, I think that in terms of the actual video model that Runway has here, it's truly, truly remarkable and truly, truly impressive. And one of the things that I think we can see from Runway is that they definitely wanted to work on this model a lot, because we know that they've been working on video models for quite some time. And although other companies may have beat them to the punch, and initially, I do think that Runway will offer a more comprehensive system in terms of what you'll be able to do for this AI area of text video. Now, this is where we get into the fine grained temporal control. And this is why I say I really, really like how Runway has managed to do their system. So right here they say that Gen 3 Alpha has been trained with highly descriptive, temporally dense captions, enabling imaginative transitions and precise keyframing of elements in the screen. So, which is really, really nice, which is what I talked about. So they says an extreme close up of an ant emerging from a nest. And the camera pulls back, revealing a neighborhood beyond the anthill. Now, this is one of my favorite clips because it shows the ability to go from something extraordinarily close up and then reveal the background in the entire scene, which shows us just how incredible it is. I mean, it was truly, truly incredible to see this scene back here, just going entirely from this initial scene right here. So. So this kind of consistency that we do see here is something that is really, really hard to do because you're trying to maintain, like, the entire scene without actually having, you know, access to it at first. You can see that right here. Of course, you just have, you know, this, and then, of course, you have to smoothly transition, you know, and what's even crazier is that you can see this even comes out of focus and. And then slowly goes into focus, and we can see this entire horizon up here, and it still maintains its entire consistency, which is Truly, truly impressive on a scale that I didn't think was even possible. Now, of course, here I think one of the very impressive things that we can have and that you can see is of course, the water simulations. Now, simulating water, if you've ever done cgi, you'll know that it takes so long to get water simulations to do. Like, if you're trying to do them on your personal computer, if you don't.

Do it, you know, if you don't.

Hand it off to like a render farm, it's going to take a really, really long time. And I think for the most part, these water simulations do look pretty, pretty impressive considering how hard it is to get something like this to be right. And the prompt for this was a tsunami coming through an alley in Barcelona. And I think that it does look pretty, pretty cool because considering the fact that this is really, really hard to do. So things like this are definitely going to present a lot of, you know, use in the future. Because like I said, one of the practical applications of this is that, like.

I said, water simulations take a lot of time, like a very, very long period of time. And in the future, when we can get to a period where you can have an image or you can have like, for example, an environment, and then.

You can just simulate the water using.

Generative AI, I think it might be.

Industry practice to do that because these.

Water simulations and a lot of these physics simulations do take an extended period of time. So that's going to be something fascinating that I do want to see in the future with as to how it works. And there were some other video demos that showed the water simulations and how well they looked too. And this one was the one that they actually showcased on their Twitter account. It was. It showcased opening a door into a waterfall in this kind of room. And it is really, really comprehensive. Like, when I saw this, I was like, there is no way that this is absolutely text to video. But you can see here that one of the key things that Runway have focused on is they focused on making sure that when they were doing this model, they ensured that the transitions were actually really accurate and effective. And we can see that because we start with a scene where the doors are basically entirely sh. And then as they open, you can see that the water, it ripples out. The lighting is reflecting off this wooden panel here, which looks really nice. And then you can see that the water is, you know, it's flowing. It looks really nice. And we can see that all of these water physics at the bottom here, like around this area, it looks really, really reflective. You know, we can see the reflections on the plants that look really nice as well. We. We can even see in the background.

I'm not sure, you know, if it's.

Like some sort of weird house or whatever, but we can see that the lights from there, and it's just really, really effective, like, in terms of how everything is looking. So I think this example shows you where Runway's focus was in terms of the overall model, because they're trying to focus on making sure that their transitions were also very effective. And whatever, you know, training mechanism they used for the water as well definitely looks really, really effective, because as the door opens as well, you can see that the water kind of, you know, underneath the door, it's kind of hard to see. You can see that the water kind of gets pushed to the side. So it's definitely something that I think, you know, they deserve props for. There was also this one of an FPV drone flying through a cliff. And we can see that the ability to enter these environments with remarkable accuracy is really, really cool.

And one of the things that most.

People actually might not notice about this is, is that, you know, from here, everything looks pretty normal. But as you can see, as we start to enter this area, you can see, like, the little fish islands coming into effect. It is pretty hard to see. But that is something that is notable on these drones. And I find that to be really, really cool. You know, that kind of gives us that realism of it actually being a drone. And then, of course, we do have the flames flickering inside and. And it still maintains its consistency, which looks so effective, because what it has to be able to do as well is it has to be able to do the lighting in a very, very effective manner. So that is something that being done here. It shows us just how effective this model is now. One of the areas where they actually speak about, and this is why I was stating before that I wouldn't have known if it was AI if they didn't say, is that they excel at generating expressive human characters with a wide range of actions, gestures and emotions, unlocking new storytelling opportunities. So basically, one of the things that they wanted to focus on was photorealistic humans. And this is something that is really, really hard to do. Like, if you've seen the game industries and the trailers of when they're actually focusing on these kinds of things and trying to make humans as photorealistic as possible, you'll know that this is not easy to do at all. Even with game Engines. And we can see right here that as this human manages to move his eyes around and even the blinking, it looks extraordinarily realistic. Like, this does look a lot higher quality than a lot of these other clips that we've seen. Like, if you compare this clip right now, which looks extraordinarily realistic, like, if someone showed me this, I would say.

Maybe it is a weird clip, but.

If we go back to the top and if we compare it to, you know, this clip, for example, we can see that there's, you know, on the background, it sometimes morphs a bit. You could see right there. Like, I don't know if you saw that. Like, just watch out for this. You can see that there's a bit of morphing here. But this is something. You know, those are minor things. Those will get, like, etched out in the next update. But the point is, is that they managed to actually focus on photorealistic humans, which makes this model really, really, really impressive in terms of it being able to stand out in certain areas.

And I think this is going to be a key area for Runway.

Like, for example, this one right here is truly, truly impressive. Like, this genuinely looks like a stock footage where you do see someone. Like, this one is. I don't know if it's better than Sora quality. I would argue it's a little bit better because it looks photorealistic. And photorealism is hard to achieve because there are certain details that, like, video generators just don't capture. But whatever runways is, like, if you genuinely compare this clip to any one of these Sora ones, they don't look photorealistic. Like, they don't look as photorealistic as this is. And I know you guys might think that this is just, like me, exaggerating, but trust me, I've analyzed Sora clips, but this one is definitely better in terms of overall humans. Like, I would say this is by far whatever this is right now in terms of generating photorealistic humans. And this is truly, truly impressive because they've managed to capture the nuance in the details of the skin of the eyes. I think there they did get the sun a bit wrong. Like, the sun, it comes out before a little bit there on the background. But you can see that the video model is still pretty effective. Like, for example, if we take a look at this piece of skin right here, we can see that as the thing rotates, we can see that the shadows will from the hair are effectively, you know, planted on the skin. It's just really, really, really nice. And something that we also do see as well is that we don't see the hair meshing about and looking weird. And as for the skin, the tattoos look remarkably intact. Like, they actually look very, very, very effective. Like, there are no weirdness about, you know, how these humans look. So the photorealism, I would say, is a 10 out of 10 for. For this model. Like, I'm not sure what they could truly improve at this point, because it just looks absolutely impressive. You can also see this example right here, which is another one that looks remarkably impressive. The emotions of the human look really, really impressive. Although there's not much emotions here, it just doesn't look AI generated.

There is nothing about this clip that.

Tells me that this is AI generated. And I've been looking at this stuff. All of the weirdness that you get. I mean, even if you look at the fine, fine hairs on her forehead, like, right here, like you would expect, because it's an AI generated video model for there to be, you know, small changes like interweaving and stuff like that with the other textures, but we don't get that at all. It's remarkably, remarkably consistent, which is pretty, pretty impressive. And then, of course, we get this clip of this old man. And you can see as well that everything here looks remarkably impressive and extremely photorealistic. So I. I would say the biggest thing that I've seen from Runway when I was trying to make this video and put certain things together and look at things beforehand, is that the human aspect is definitely there. You can see as well the wrinkles of the face as this character manages to not smile, but move up his cheeks a little bit. And it's stuff like that that is really, really hard to get, because simulating human skin and all of the folds and all of that stuff is very, very hard. So this right here is super impressive. I'm guessing that their training data, maybe they focused on just, you know, having a lot of examples of photorealistic humans. We know that they've been working on this for quite some time, like, probably longer than OpenAI, which is why this looks so, like, actually photorealistic. And, guys, remember, there is a difference between photorealism and. And something that looks high quality. Like, there is a complete difference. Like, you get Photoshop pictures. Like, if you've seen a Photoshop picture and it looks high quality, but it doesn't look photorealistic, there is a complete difference. So photorealism is something that looks actually real. Like, you can't tell whether or not. And stuff that likes is high quality, is completely different. But stuff like this, this is where you genuinely won't be able to tell if it's AI generated or not because of the photorealism. And that's exactly kind of the things that we've seen with Mid Journey, and where the photorealism aspect makes it that it looks like it was genuinely taken by a real person. And that's the point I'm trying to drive home here, is that these photorealistic humans are just super impressive. Like, even this example right here, you would expect there to be a lot of morphing around the lips. You would expect there to be a lot of morphing on the face, but there just isn't. It just looks really, really good. Like, this is a surprising level of detail and accuracy from Runway that I can truly get behind. And then this was the most impressive example that I did see before because it was so impressive with a lot of factors. So one of the factors that made this so impressive was the fact that you have someone who's doing this emotion. You can see the skin is all wrinkled. And that is traditionally hard for AI systems to do, because differentiating between what is the eye, what is the mouth, what is the cheekbone, what are these folds here, all of these things just make it harder for an AI system to kind of really figure out what's going on. And then what we had here was not only that, we had him change the motion, which looks really effective. He doesn't look weird. Okay. And then he was able to get a wig dropped on him. Okay. And remember, the wig is, you know, made up of loads of different hair particles, or whatever you want to call it. And we can see that the wig bounces perfectly on his head. And then we can see that as the glasses come down, there are initial reflections on those glasses, and we can see his eyes behind those glasses. So this, by far, I'm not going to lie to you guys, this is, you know, just insane. Like, this was something that I thought, wow, that is truly, truly impressive, because you're not only taking, you know, a human, you're actually adding something into the environment that is not easy. Like, hair would be something that you would think, you know, messes up pretty easily. But just looking around at the edges and the little light reflections that you do have, that looks just way too impressive. Now, Gen 3 Alpha actually has some very, very cool stuff. You can see that here. They wanted to on, you know, creative characters, so they made this one, which is like a giant, you know, strange creature there's also these ones right here where, you know, you're going through a neon, you know, neon like forest. Which is. I think, you know, why this is so important is because when you have a system that doesn't have a lot of training examples, the examples that you do get when you generate, you know, images or videos, usually they come out to be pretty bad. But what we're seeing of this system is that the data set that they probably use is, is definitely evenly balanced as these things don't look bad at all. Which means that they've covered a wide range of diverse topics. Like this right here. A cyclone, cyclone of broken glass in an urban environment. These kind of things are just absolutely incredible, you know, to look at. And then of course we've got a man standing in front of a burning building giving a thumbs up. Like, this is pretty, pretty incredible stuff. We've also got this ultra wide shot of a hand on a mountain made out of stone. Like, this is really, really cool. I can imagine the kind of movies and films we'll be getting soon. And then of course, this was one of the things that they showcased, which was a creature walking through the city. And like I said before, the photorealism here is incredible. I mean, looking at this character, you know, many people on Twitter would always talk about how, oh, this, you know, video has, you know, morphing. It has. This has had that. I truly want to know what is wrong with this clip because other than it being a little bit slow and maybe, you know, not simulating the physics entirely correctly, I think that one thing that it does get is, is the lighting from this lamp. We can see that there's also this right here, the beams of light coming through. If you can see that. And then what we do have is the lighting radiating onto this character's face on this side right here. So it's truly impressive because it's also able to generate the hairs. We can see that the hairs also move stick. They don't like morph a lot. The only thing I think that's bad about this is the slow motion, which is not too effective because of course this character wouldn't be speeding through, but it also wouldn't be moving that slow as well. If it was to jump, it would move a little bit faster. But that's just the least other problems because this is also, you know, being done through a window. So it's also pretty hard to kind of look at things there. But I think this is definitely super, super impressive. So overall, I think this model is genuinely super impressive because I think what Runway have showed us is that their research and the foundations and the things that they've actually, you know, spoken about, such as their general world models. And basically Runway actually said, and I'm going to scroll back to this, I might have some things going up on screen, but basically they said that they were mainly focusing on, if you go to this, is that they were saying that they were focusing on the major advancement in AI is going to be coming from systems that understand the visual world and its dynamics, which is why we're starting a new long term research effort which is around what we called general world models. So basically they're stating that they wanted to have this AI system that builds an internal visual representation of an environment and uses that to simulate future events with that environment. And of course they've been pretty controlled, but essentially they're going to be able to simulate a lot of, you know, wide range of interactions and situations and I'm guessing.