Károly Zsolnai-Fehér demos Google's new text-to-music AI, which generates everything from orchestral soundtracks to 8-bit arcade music. The model can produce coherent pieces up to 3-5 minutes long and outperformed prior techniques in human testing, though not across every metric.
Dear fellow scholars, this is Two Minute Papers with Dr. Zsolnai. Today I am going to show you a paper and I am almost out of words. It is so good. You know what, let's jump right in. This new AI is very much like Dall E2 which is a text to image AI. Our text goes in and an image comes out. And this new paper is not text to image, but music. Yes, a piece of text goes in and some hopefully amazing music comes out. Now let's see what it can do. Let's ask for an epic soundtrack with orchestral instruments. And do we let the AI get away with such a simple prompt? Of course not. Not around here. The piece should also build tension and create a sense of urgency. And. And you know what? Add an acapella chorus on top of it. There is no way that an AI could generate something like that. Right? Well, hold on to your papers fellow scholars and listen. That was incredible. We will see in a moment how incredible exactly. Now let's give this AI an even harder time. Oh yes. We wish for a bass and drums led reggae song with electric guitar. Very difficult, but it gets even worse. Vocals, expressive vocals with a laid back style. There is actual absolutely no way that this is happening. But just in case, let's listen. I can't believe it. Electric guitars. Check Mark. I love the intonation and the personality. And the vocals were also something else. Wow. Now let's ask for something completely different. We are real scholars so we are going to ask for real 8 bit arcade music. Holy mother of papers. Now that's a song that has momentum, intensity and a great guitar riff. I am truly stunned. The prompt also asked for unexpected sounds. I have to admit I am not so sure about that part. But the rest was incredible. But not as incredible as these four other things that it can also do. One, we already know that we can ask for some piano playing, but it gets better. We can ask it to be a beginner piano player or a real pro. Let's see. Now that is super cool. Loving it. Two, these were only 30 second sequences. Most songs are significantly lower than that. And there is a problem with this. What is the problem? The problem is that these AI based techniques have trouble with long term coherence. That means that a couple minutes into the song they don't quite remember what they played earlier. And that is indeed a problem because writing music is about building up a melody and we have to remember the melody and embellish it over time. So what can this new paper offer in this area? What is this you are kidding, right? A five minute long song? Well, let's give it a shot. We will listen to small snippets of this jazz song from different parts. Now, this was truly something. For about three minutes it felt like one coherent song. And after that it did not morph into another one. But I feel that it lost some coherence. However, it was not too far. Let's say that this was one and a half songs. That is an insane result because many previous techniques could barely stay coherent for even a few seconds. A few minutes is science fiction region. And all this just one more paper down the line. My goodness. 3. It can even imagine what paintings would sound like. I have to admit that I did not find these samples too convincing. However, the one for Starry Night was pretty good. This is a tranquil, soothing song with a hint of mysteriousness. It's not perfect, but I like it a great deal. And I think the capability is there for all of these paintings. It is truly up to us humans writing a good prompt to make them come alive. But wait. 4. What happens if we write a prompt and we don't quite like the results? Well, it can generate several variants. In fact, as many as you wish. So I promised that I would tell you how good it exactly is. So how do we know that? Well, through a user study. Get some humans in the room, show them some real music and some AI music from this technique and from previous works and see which one they like best. And the results are truly something else. Music CAPS is real music. So that should win most of the time. That is not surprising. However, what is really surprising is that it does not win all the time. And. And it gets even better. Look at that. The new Technique Music LM is rapidly closing in. And remember, Dall? E1 was just an OK text to Image AI and just one more paper down the line. This is what Dall? E2 was capable of. And with all this, I am convinced that we are at a Dall? E 2 moment for music generation. And it only gets better from here on out. And just imagine what we will be able to do. Two more papers down the line. What a time to be alive. So what do you think? What prompts would you like to try in the future? And what paintings would you like to come to life? Let me know in the comments below.
Weights and Biases provides tools to track your experiments in your deep learning projects. Using their system, you can create beautiful reports like this one to explain your findings to your colleagues better. It is used by many prestigious labs, including OpenAI, Toyota Research, GitHub and more. And the best part is that Weights and Biases is free for all individuals, academics and open source projects. Make sure to Visit them through wnb.com papers or just click the link in the video description and you can get a free demo today. Our thanks to Weights and Biases for their long standing support and for helping us make better videos for you.
Thanks for watching and for your generous support and I'll see you next time.
