Microsoft trained an AI on 100 million YouTube videos to build a system that generates images, video, and audio together from text, images, or audio inputs. Dr. Zsolnai walks through example prompts and outputs, comparing it to earlier systems like Google's MusicLM.
Dear fellow scholars, this is Two Minute Papers with Dr. Zsolnai. Yes, that's right. Microsoft's new AI watched about 100 million clips from YouTube and today we are going to ask one simple question. What did it learn? I'll tell you in a moment. You see, today, text to image AIs are all their age. By just doing as little as entering a text prompt, we can create all kinds of images in and even videos. A true miracle of science, really. And I often hear people saying that they can't wait to be able to express their creativity by creating movies with them. However, not so fast. We can already do that with Google's and Runaway's technique and there are even more out there. But for some reason I rarely hear people talking about sound. Well, this new paper solves exactly that. Now wait a minute. We are going to invoke the second law of papers here which says that whatever you are thinking about, there is already a two Minute Papers episode. On that, for instance, we talked about Google's music LM an incredible AI based technique that takes a text prompt and synthesizes incredible music. Listen. This is fantastic. Also, our seasoned fellow scholars remember an earlier episode where we talked about a brilliant handcrafted graphics paper that did something that was previously thought to be almost impossible. So what is that? Well, hold on to your papers because it took an already existing video clip or computer animation and synthesized sounds for it and the results were outstanding. Yes, through the power of computer graphics research, this was possible approximately 10 years ago. Yes, you heard it right. This paper was published 10 years ago at the Siggraph conference. That is an incredible place with similarly high quality papers. Now with all that said, what does this new AI paper add to the mix? And I would say everything. Now I hear you asking doctor, what do you mean by this? Well, exactly what I said. This is an any to any generation technique that also includes audio. Here are some of my favorite examples. One, if we enter a piece of text like fireworks in the sky, it synthesizes video and audio at the same time. Or we can write dive in a coral reef and we get this. Two, we can also go the other way around. If we have a text description of a painterly style and look, no mention of shapes at all. And we also add this sound sample. As a result, we get this image. So cool. But wait, it gets even better. Three, we can even enter a text prompt plus a piece of audio and out comes not just an image, but oh my, even a piece of video. As requested, the camera indeed goes forward Ever so slightly. Good job little AI. And we can also perform the usual suspects. Text to image, a piece of audio to an image, or if we have a cracking photo, we can even ask it to create a video with a somewhat similar vibe. Wow. It feels like this can do absolutely everything. Now every single one of you fellow scholars see that the quality of these results is not the greatest, however. Remember two years ago we had Dall? E 1 for image generation, a year later Dall? E 2. And now a year later, midjourney can do this. And I would not be surprised if today we would be at a Dall? E one moment for full film or in other words, video plus audio generation. Just imagine what we will be capable of just two more papers down the line. My goodness, what a time to be alive. And if you don't mind, I just can't resist quickly showing a really cool computer graphics paper to you. This is a new technique that is able to simulate situations with a brutally large number of collisions. These simulations contain objects that are built up from millions and millions of tetrahedra, little pyramids if you will. For instance, when squishing this ball, it can simulate what happens next. Incredible. It can also simulate Davy Jones and his friends falling into a glass box and other complex nutty situations. All this in a matter of seconds. Absolutely incredible graphics research. Sorry, I just couldn't resist showing this to you. I hope you fellow scholars don't mind. Let me know in the comments below.
Weights and Biases provides tools to track your experiments in your deep learning projects.
What you see here is their Tables.
Feature and the best part about it is is that it is not only able to handle pretty much any kind of data you can throw at it, but it also presents your experiments to you in a way that is easy to understand. It is used by many prestigious labs including OpenAI, Toyota Research, GitHub and more. And the best part is that Weights and Biases is free for all individuals, academics and open source projects. Make sure to Visit them through wnb.com papers or just click the link in the video description and you can get a free demo today. Our thanks to Weights and Biases for their long standing support and for helping us make better videos for you.
Thanks for watching and for your generous support and I'll see you next time.
