LARRUZ explores OpenAI Sora, an AI that generates video from text prompts, creating everything from hybrid animals to futuristic cities. The tool could democratize filmmaking for musicians and designers, but struggles with keeping characters consistent across frames. The "Airhead" project shows how creators are already working around these limits.
OpenAI Sora is a brilliant new text to video AI where you write something and out comes a video of it. With this, everyone will be able to become a movie director. And I was really curious to see what brilliant creative people can do with it. Luckily, that is exactly what happened and the results are stunning to say the least. Buckle up because we are going to talk about the virtues and and the limitations of the current system. Dear fellow scholars, this is two minute papers with Dr. What I absolutely love about Sora is that it can do things that you could normally not do. For instance, make up new animal species. Now hold on to your papers, fellow scholars, and enjoy the giraffe flamingo in the savannah, its natural habitat. Probably looking for its next prey to be mercilessly licked to death. Or the mighty flying pig elegantly gliding across the skies with the finesse of a belly dancer. Or how about the bunny armadillo, a true warrior protected by an unbreakable shell armor. Unless you strike the unprotected areas, which is most of the bunny, of course. Or the brilliant fox crow, a cunning assassin lurking on the trees with an undefined number of limbs and any combination of two different species that are guaranteed to not make any sense. You can try them all. This would otherwise be impossible or at the very least prohibitively expensive to produce. Not anymore. This was incredible. I love this one. And now we will get to see what an amazing sculptor can do with Sora. This showcases the work of an artist who starts out with ideas that were generated by an AI and then getting his hands dirty and bringing these sculptures into the world by actually creating them. So cool. Musicians will absolutely love this technology because they will be able to rapidly create visuals for their soundscapes and see how they complement each other. It will also be a great deal of help with rapid product design. I love how well the text came out on this one. Or you can design a futuristic city as well and have a peek at what that might look like. This is going to be an incredible tool to supercharge the creativity of all the brilliant people around the world. And all this will be possible very soon, likely this year. And don't forget, over time the costs are going to decrease. So much so that you will be able to do all of this instantly and do it even better than this right with the phone in your pocket for nearly free. What a time to be alive. Now limitations so far this tool is excellent if you wish to create not necessarily a fully consistent story, but but essentially a montage of visual footage. Now I hear you asking? OK Caroj, but why is that? Well, check out this one called Airhead. This is a project about carefully trying to overcome one of the main current limitations of Sora and that is character consistency. This means that when we create a new character, it is very challenging to get exactly the same character back in in the next scene. Telling the story of someone, but without being able to portray that someone would be a necessity. But that's impossible. Or at least very challenging. So what is the solution? Well, Airhead, you see, this character has so few features that it is relatively easy to reproduce. And what does a brilliant artist do with that? Of course, embrace that limitation and turn it into a really cool story. Loving it. And interestingly, this may be the first AI based movie where I find myself enjoying the storytelling and not looking at the technology. That is an incredible breakthrough. Why is that? You see, I remember Ed Catmull, co founder of Pixar saying that he created his first animated movie and was delighted by the fact that that the people were criticizing the story. He said that is fantastic. No one is talking about the technology, they are talking about the story. This means that he has succeeded and we might have the same for Sora today. Now I cannot get enough of Sora and I hope you are excited too. I have another video coming on it which I've learned a lot from. It is about beauty, creativity and horror. Trust me, you are not ready for the horror part. Subscribe and hit the bell icon if you are interested in watching it. It is coming very soon. Also, I will fly to San Francisco for the first time ever and I will speak at the fully connected event. There are still tickets available and I am super excited to meet you fellow scholars there. Last time we did it, this happened. The link is in the video description. See you there. Experiment tracking, model evaluation and production monitoring for your deep learning projects and LLM apps. This is what weights and biases does and it is the best. Everyone is using it. Try it out now at WNB Me Papers or click the link in the description below.
