Dr. Zsolnai reviews OpenAI's Sora text-to-video model, showing examples from a house tour to a horse on roller skates. The model nails material realism and physics, but struggles with limbs and geometry. He expects rapid improvements as compute limits lift.
OpenAI's text to video AI Sora took the world by storm just a few weeks ago. We all know these results. But good news, there are some new ones that I found absolutely stunning and I learned a lot. Stay tuned because here you will see beauty and imagination and you will see horror. I was not ready to see the horror. Dear fellow scholars, this is Two Minute Papers with Dr. Zsolnai part one. Let's start with the beauty. Here is a house tour. I absolutely love how well made this is. But what is perhaps even more impressive is that it really has a model of the house itself. It does not feel like it is just making things up as it goes. And as I am a light transport researcher by trade, the that is ray tracing if you will. When I look at these beautiful reflections and refractions everywhere, this makes me really really happy. And glossy material models too. They are not perfect, but leaps and bounds beyond any AI based text to video I have seen so far. And also look, the resolution of some of the geometry is quite interesting. For instance, the bed sheets. This almost feels like being in a video game where the detail has been toned down a little. Perhaps it also looked at materials coming from video game engines too. I don't know for sure, just speculation. However, when we look at this colorful paper flower blooming, I see high resolution textures, lots of tiny folds, imperfections everywhere. It is quite interesting how it combines low and high resolution materials. Now how about a little more action? Oh yes, here. I love how it understands the physics of the race car. And the movement of the wheels is also incredible. Not perfect, but incredibly good. Also a tiny little detail that only appears in the last few frames, but is super important. Look. Oh my. The interaction of the wheels and the dust. Absolutely lovely. Part 2 Creativity. Oh boy, this is going to be amazing. It can reimagine Niagara Falls, but with colorful paint instead of water. Once again it puts up a clinic with its understanding of physics. As an undergrad student, it took me months to understand the underlying hydrodynamics. And. And this one, after training can create these in a matter of seconds. I am not jealous. Ok, just a little bit. But it gets better here in an abstract animation where drops of ink form into lifelike creatures. This, dear fellow scholars, is not just fluid simulation, but also fluid control. Once upon a time I wrote a paper on how to bend the laws of physics to control fluids from for them to take different shapes that we are looking for. And it was extremely difficult, almost impossible. Yet once again the AI can pull it off just fine. I Am completely stunned by this. Once again, not jealous, just a little. But the creativity goes crazier, much crazier. If you wish to see a horse on roller skates, not a problem. Once again, not perfect. I keep saying that there is a reason for it. I'll get to that in just a moment. Or an elephant made out of leaves. Not a problem. So much so that they even move properly. And the specular reflections off of it are absolutely incredible. Perhaps a bit too bright, but still love it. And now, may I interest you in a cosmic Tea fellow scholarship? Interestingly, it is made out of more viscous material, a bit more like honey or oil, not just regular tea. I am loving how not only the theme, but the movement of the fluid also has some creativity. And so far you have seen prompts, but you haven't seen what other things it can do. For instance, it can even mix the content of two videos together. It is able to use this city for most of the content and place this little winter wonderland in it. However, to do that, it has to be able to draw it from different viewpoints. And it not only does that, but it does that consistently. We talk a lot about neural radiance fields and other papers that researchers had to write to do even a rudimentary version of this. It is hard, hard work. And once again, the AI just does it. I can't believe it. I hope you appreciate this additional context from other papers. Now, part three. The horror in this previous unassuming robot video. This one is doing pretty well until. Are you ready for this? Are you sure? Okay. Ouch. The legs are attached incorrectly to both people. And when we switch camera angles, they have grown some new ones. So I keep saying that these are not perfect. So how much of a problem is that? How is it possible that it can create beautiful, often almost impeccable artistic videos? Fantastic camera work, consistency, the whole package. And yet it almost feels like it sometimes imagines people as IKEA furniture and attaches some of the parts completely incorrectly. I have a guess. Once again, this is just speculation, but look here. When we don't have a great deal of compute, we get something like this. As we add more computational power, the whole concept really comes to life. And our current compute capacity might very well be the before version that pales in comparison to what we will have just a few months or maybe a year from now. We need just one more paper. And as the technology gets cheaper and cheaper, everyone will be able to become a movie director. You will be able to get 10,000 variants for the same prompt in a matter of seconds soon. What a time to be alive. I am also visiting the OpenAI lab in a few weeks and I really hope that I'll be able to run some prompts to show you in a video. You know, videos of scholars holding onto their papers need to be done, but in any case all this will likely be available later this year. I cannot wait. Wow. If you're looking for inexpensive cloud GPUs for AI, Lambda now offers the best prices in the world for GPU cloud compute. No commitments or negotiation required. Just sign up and launch an instance and hold on to your papers. Because with the Lambda GPU cloud you can now get on demand H100 instances and they are one of the first cloud providers to offer publicly available on demand H100 access. Did I mention they also offer persistent storage? So join researchers at organizations like Apple, MIT and Caltech in using Lambda Cloud instances, workstations or servers. Make sure to go to lambdalabs.compapers to sign up for one of their amazing GPU instances today.
