DeepMind's new AI can generate realistic audio from video, synthesizing sounds like drums, guitars, and pens. The system also controls physics simulations from text, creating cloth and fluid movements on command. It marks a major step toward AI that understands both sight and sound together.
Today I see more and more text to video AI techniques everywhere. You know, you write a piece of text and out comes a video. We will talk about this incredible technique that is out there right now in a paid product in the second half of this video too. But first things first. These techniques are getting better and better almost every week. But there is some something still missing in nearly all of them and that is sound. Oh yes, we cannot hear them at all. So what can we do about that? Dear fellow scholars, this is two minute papers with Dr. Zsolnai. You see, there are previous research papers in computer graphics where you could give it a virtual scene with this happening and it would synthesize crumpling sounds or for fluids with bubbles. As you hear. They are amazing, but there are many problems with them. They are super complex and they are also super specialized for one particular task. They cannot do anything else. And most of all, they need to see lots of simulation data like the velocities of objects. Just the video is not nearly enough. That's a bummer. However, this new AI technique is capable of looking at a video like we humans would do and synthesize what it could sound like. That sounds almost impossible. But now hold on to your papers fellow scholars and let's check this out to together. Wow. Yes, this is a. I was especially interested in seeing what it does with footage where it really has to understand the timing of the movement to even have a fighting chance. Let's see. Oh yes, the drums really check out. And adding. Loving it. The guitar was a bit less crisp, but it appears to understand when a string is being picked and roughly what sound it should make. Same with the movement of this pen. So cool. To the best of my knowledge, it will be available as part of Veo Google DeepMind's text to video technique. There's a link to it in the video description. By the way, I really like how they are using a diffusion based approach to generate audio. For earlier techniques, this meant that it started out from noise and over many steps this noise is reorganized into an image or video or in this case even sound. It is incredible how useful this approach is to a variety of tasks. Now on to video synthesis. Here is a new tool called Gen3. Once again you type a piece of text and you get this and its results are absolutely stunning. This is the first time I am seeing results that are closing in on OpenAI Sora, perhaps the best video AI out there. However, Gen 3 has a very appealing property that Sora doesn't have. Ah yes, this one is available right now, although we have to pay for it. And as you see, the new Gen 3 is particularly good at generating photorealistic humans, but it has some more surprises up the sleeve Simulations, the two minute Papers special and be it cloth or fluid or fire simulations, it seems super capable and frankly super fun to use. With a carefully chosen prompt. It can also create videos that are not only funny, but the quality is out of this world. I mean just look at that. Whoa. Just think about it. About a year ago this was the best text to video AI around. We were stunned by how advanced it was and now just a bit more than a year later we get this and it will soon get significantly cheaper. And even if the improvements stopped, right now we are only scratching the surface of what these tools are capable of. For instance, a fellow scholar just tried creating green screen videos with it and yup, it can do that too. And I was also stunned to find out that it can also do this. Well, not because of the text itself that you see here, but because yes, this is controlling smoke simulations, but the text. Come on. Alright, now we're talking. I wrote a paper on fluid and smoke control long ago and it is incredibly difficult and today an AI system can learn it as a completely unintentional byproduct. Just think about it. I am completely baffled by this. And of course the other two minute Paper special scholars holding onto their magical papers. So good. Love it. So now we can create videos from scratch. And not only that, but we will also be able to synthesize sounds for it. We aren't there yet but we will all be able to become movie directors for free or for very little. How cool is that? What a time to be alive. So what would you make with these? Let me know in the comments below. Also, after a long long time we we might be able to onboard a new sponsor for the series. If you are interested, there's a link for you in the video description. We need new tools for the era of LLMs and weights and biases. Now has weave a lightweight toolkit to confidently iterate on LLM applications, use traces to debug how data flows through each step of your app and use evaluations to measure your progress. It is the best. Try it out now at WNB Me Papers or click the link in the description below.
