OpenAI's Jukebox generates original songs from genre, artist, and lyrics by combining score-based and waveform-based AI methods. It works mainly on Western music and English lyrics, and takes about 9 hours to produce one minute of audio.
June 9, 2020.
Approximately nine hours per one minute of music.
Mainly trained on Western music.
English.
2017 and 2018.
Dear fellow scholars, this is Two Minute Papers with Dr. Zsolnai. Today I will attempt to tell you a glorious tale about AI based music generation. You see, there is no shortage of neural network based methods that can perform physics simulations, style transfer, deepfakes, and a lot more applications where the training data is typically images or video. If the training data for a neural network is in pure text, it can learn about that. If the training data is waveforms and music, it can learn that too. Wait, really? Yes. In fact, let's look at two examples and then dive into today's amazing paper. In this earlier work by the name look, listen and learn, two scientists at DeepMind set out to look at a large number of videos with sound. You see here that there's a neural network for processing the vision and one for the audio information. That sounds great, but what are these heatmaps? These were created by this learning algorithm and they show us which part of the image is responsible for the sounds that we hear in the video. The hotter the color, the more sounds are expected from a given region. It was truly amazing that it didn't automatically look for humans and and colored them red in the heatmap. There are cases where the humans are expected to be the source of the noise, for instance in concerts, where in other cases they don't emit any noise at all. It could successfully identify these cases. This still feels like science fiction to me. And we covered this paper in 2017, approximately 250 episodes ago. You will see that we have come a long, long way since we often say that these neural networks should try to embody general learning concepts. That's an excellent and in this case testable statement. So let's go ahead and have a look under the hood of these vision and audio processing neural networks. And yes, they are almost identical. Some parameters are not the same because they have been adapted to the length and dimensionality of the incoming data. But the key algorithm that we run for the learning is the same. Later in 2018, DeepMind published a follow up work that looks at performances on the piano from the masters of the past and learns to play in their style. A key differentiating factor here was that it did not do what most previous techniques do, which was looking at the score of the performance. These older techniques knew what to play, but not how to play these notes. And these are the nuances that truly make music come alive. This method learned from raw audio waveforms and thus could capture much, much more of the artistic style. Let's listen to it. And in the meantime, you can look at the composers it has learned from to produce these work. However, in 2019, OpenAI recognized that text based music synthesizers can not only look at a piece of score but can also continue it, thereby composing a new piece of music. And what's more, they could even create really cool blends between genres. Listen as their AI starts out from the first six notes of a Chopin piece and transitions into a pop style with a bunch of different instruments entering a few seconds in. Very cool. The score based techniques are a little lacking in nuance but can do magical genre mixing and more, whereas the waveform based techniques are more limited but can create much more sophisticated music. Are you thinking what I am thinking? Yes, you have guessed right. Hold on to your papers because in OpenAI's new work they tried to fuse the two concepts together. Or in other words, take a genre, an artist and even lyrics as an input and it would create a song for us. Let's marvel at a few curated samples together. The genre, artist and lyrics informations will always be on the screen. Way I'm before you. Time Christmas night you know what got you? It's Octopus. Wow. I am speechless. Love the AI based lyrics too. This has the nuance of waveform based techniques with the versatility of score based methods. Glorious. If you look in the video description you will find a selection of uncurated music samples as well. It does what it does by compressing the raw audio waveform into a compact representation. In this space it is much easier to synthesize new patterns after which we can decompress it to get the output waveforms. It has also learned to group up and cluster a selection of artists which reflects how the AI thinks about them. There is so much cool stuff in here that it would be worthy of a video of its own. Note that it currently takes nine hours to generate one minute of music and the network was mainly trained on western music and only speaks English. But you know, as we always say around here, two more papers down the line and it will be improved significantly. I cannot wait to report on them should any follow up in these experiments really showcases the usability of their system. Weights and Biases provides tools to track your experiments in your deep learning projects. Their system is designed to save you a ton of time and money and it is actively used in projects at prestigious labs such as OpenAI, Toyota Research, GitHub and more. And the best part is that if you're an academic or have an open source project, you can use their tools for free. It really is as good as it gets. Make sure to Visit them through wnb.com papers or just click the link in the video description and you can get a free demo today. Our thanks to Weights and Biases for their long standing support and for helping us make better videos for you. Thanks for watching and for your generous support and I'll see you next time.
