Udio & the age of multi-modal AI

Welcome to Practical AI. If you work in artificial intelligence, aspire to, or are curious how AI related tech is changing the world, this is the show for you. Thank you to our partners at Fly IO, the home of changelog.com Fly transforms containers into micro VMs that run on their hardware in 30 plus regions on six continents so you can launch your app near your users. Learn more@flyeio.

Welcome to another fully connected episode of the Practical AI Podcast. In these fully connected episodes, Chris and I keep you fully connected with everything that's happening in the the AI world. The news, the trends, the new models, all the good stuff and talk through some things that will hopefully level up your machine learning game. I'm Daniel Whitenack, I am founder and CEO at Prediction Guard and I'm joined as always by my co host Chris Benson, who is a principal AI research engineer at Lockheed Martin. How you doing Chris?

Doing great today, Daniel. I don't know how we're going to pick what to talk about. There's so much stuff coming out right now.

There's a lot mostly kind of new. Well, there's always new models I guess, but it did seem like a big week with I think new GPT4 Turbo, new Gemini. It's really hard for me to keep track of the numbers and parameter counts and all that, but I know that new GEMINI I had 1.5 something. I forget the different numbers.

Yep.

And then new Mistral, new I think it's different mixture of experts. Top of the open LLM leaderboard, we've got Uber Udio, which I was just. So I've been at some startup accelerator stuff and then at conferences, events and I'm like, it seems like I go into one of these events and at the end of the day people just say, oh, did you hear about X? And I'm like, no. I was. I haven't had my laptop open. Something else has happened apparently for the.

Last two hours I haven't had my laptop and I've missed. I was doing work, you know.

Yeah, I don't know if you've seen the trend. I think this was a prediction for 2024, which I think was a well informed prediction for 2024 from many different people. And I think we talked about this in our own discussions of 2024 around multimodality of AI in 2024. Whereas in 2023 you kind of saw this explosion of in many cases text to text AI, meaning I put in a text prompt and I get some text back. Now we're seeing an Explosion of multiple modalities of data input and or output to these models. And that's mostly what I'm seeing. Is that consistent with your view as well, Chris?

Totally. I was thinking about that. And a moment for us to brag. I'll tell you, we've actually been fairly good with our predictions the last few years.

Who knows, maybe we're actually setting the. It's a cell ph fulfilling prophecy. It's just everyone's listening to practical AI and they're making our predictions real. That must be what it is.

I'm sure you know that all of the, you know, like over at OpenAI, they're just listening to us and they're going, okay, that's what we need to go work on.

It's a lot of pressure.

Yeah. And I tell you what, it's a good thing. We're steering the entire industry by ourselves.

Right. Which is. Yeah, it's a lot of pressure, but it's good.

Sure. No pressure at all, man.

I do want to talk about multimodality today, but I've just got to share with you some of this UDIO stuff, Chris, which I think. Well, Udio. Udio. I don't know if anyone knows how to say this yet. I was saying Udio. Maybe it's Udo. U D I O, I believe.

I'm not going to hazard a guess till I hear somebody else do it.

Yeah, yeah. It's coming out of beta. I think that there were some leaks of some of what they were doing before, but essentially, if you go to this website, you can sign up for an account, they have it marked as beta. So I'm not sure exactly where this is going necessarily product wise, but what you see, at least in its beta form, is essentially a space where you can put in a text prompt. It kind of reminded me of almost like a clip drop or something. Some of these image generation platforms where you can kind of pre select some elements of the prompt. And the goal would be to completely generate a coherent and compelling song or piece of music or composition. It's essentially a music generator. So we've seen a little bit of this in the past, Right. And in the past, we've kind of heard things generated, like kind of dreamy ambient things and. And maybe useful for kind of backing YouTube videos or something like that, but not really compelling music in and of itself. And I think what's interesting about this UDIO is that it generates both this sort of compelling music, but also lyrics and also synthesized voices singing the lyrics all together in one so, Chris, before, while you were setting up your studio there to record the podcast, I was busy on Udo, figuring out what. What is there. Now there's a couple of really interesting ones that I listen to, and I've preloaded a couple in here for us to listen to and to give our audience a little bit of a sense of the audio.

Absolutely.

Because this is an audio podcast, so, you know, what better format? So one of these, which I found really intriguing was the Broadway musical.

And I would go to that, by the way, just to make it very clear. I'm standing in line to buy tickets, so to speak.

Well, the music has been generated for this, and I'll cite Bobby B. So Bobby B. On Udio, and he's created Dune, the Broadway musical. So just to give a sense of people, the prompt that went in to this to create says teen pop show tunes, film soundtrack, uplifting, playful female vocalist, happy. Anyway, so you get a sense of kind of similar to a. I guess, like an image generation prompt, where you're saying, like, high resolution, unreal engine, like this sort of stuff to give it some stylistic guidance. But I've got this preloaded in here, Chris. Everything that you're gonna hear is AI generated. So let's listen to Song of Arrakis from Dune, the Broadway musical.

Oh, you got my attention, man.

Paltry deeds of Arrakeen. The greatest leader we've ever seen. They say that he's the Lipson al Gai. Eyes bright blue and hair jet black. You should see him ride on a sandworm's back. Lead us to victory. Usil mu.

What do you think?

Move over Wicked. Move over Les Miserables all just, you know. I saw Hamilton recently. Move Over Hamilton. We're all about doing the musical now. That's it.

Exactly. Yeah. So good. And even the lyrics, you know, eyes bright blue and hair jet black. You should see him ride on a sandworm's back. I mean, that's good right there.

I like the fact that the music actually deviated far from the darkness of Dune, you know, the perpetual darkness of the theme. That was fun. That was great.

All right. Ready for more?

Yeah. Yeah.

So I tried out my own, of course. I had to try out my own, of course. So my prompt that I put in. And I only experimented with this, so I'm sure you can do much better. But I put in a song about two podcast hosts trying to navigate the wild and crazy world of AI in the style of pop rock.

Practical AI the Musical.

We really appreciate you. Breakmaster Cylinder, the mysterious Breakmaster cylinder behind our theme music for the show. But this is what Udo can do, and I selected specifically to have it pop rock and to auto generate the lyrics. So I didn't put in any lyrics. So I have two options for you, Chris. I have two selections that you can see which one you like better. Here's the first.

The first thing they do. Hit record in their room Discussing the trends where the bots will take us and launch into forecast oh, yeah, oh, yeah, oh, yeah Caught up in the wave where the data streams tight With a crackle in the voice and the laughter just right they chat with a rhythm that's as quick as a flash.

All right, selection one. Thoughts?

I like that one. You just transported me from, like, you know, 53, which is what I'm at now, all the way back to, like, 16, in the 80s, you know, late 80s, that I was all about that. That was good.

Okay, cool. Yeah, I love it. Okay, that one was. They also generated a title. It's interesting. I don't know how much of this, you know, how many models are at play under the hood here and how they're coordinated. I'm guess maybe there's some that generate the lyrics and some that generate the title, and then somehow that's merged together in a music generation, because obviously the voice and the lyrics have to be coordinated somehow. I at least didn't see a lot of underlying explanation of what's going on here, but pretty interesting. And that was generated, I would say, in 30 seconds or something. I don't know. Not that long, right? So let's take a listen to number two. This one was titled Digital Odyssey.

Both hosts dive. Indeed, my friend. Oh, yeah, oh, yeah Voices through the digital tide they're learning as the codes blend Got theories, theories Everyone must hear A journey through the AI sphere.

There you go. Even with a little bit of a guitar type solo there at the end.

I know, it was good. I like. I like the first one better. The first one felt like I was, you know, like I was a kid again right there. But. But I like both. And yeah, this was good. I. I could just spend all day generating music. Now that. I may do that. Actually.

I might have just taken up your Saturday.

Oh, my God. My wife has all these chores planned for me because we're recording here late on a Saturday morning and I'm. I may get myself in trouble by. Yes. Okay.

Well, there you go. So, Udio, check it out. Super cool stuff. I think this does bring up some really interesting challenges, issues, struggles, excitement. Also, of Course, joy in hearing Dune the musical, but it is super interesting. I even thought about this when I was going through here. Well, you know, Bobby B. Created Dune, the Broadway musical, and I just downloaded and I'm doing it what I want with it, which I guess is playing it on my podcast. So, Bobby B. I hope you're okay with that. Now, technically, technically, this is machine generated. So at least as far as I understand in the current US legal system, such a thing would not be copyrightable. Sorry, Bobby.

Sorry, Bobby.

I'm not giving legal advice here, obviously, and not a lawyer, but that's my understanding from our previous conversations. But what's interesting is, I think similar to these AI generated art things that, you know, were put into art competitions and won. Right. There could be. Now, in my case, I just put in a simple prompt that, you know, generated something in 30 seconds. There could be some really deep thought put into how to construct this prompt and the various kind of. I think Bobby's prompt was much better constructed than mine. And also you can upload your own lyrics into this to add a level of creativity. So there's really an open question here of how much human creativity is actually a big portion of this generation, and will the established laws and legal entities eventually recognize the creativity that's put into prompting these sorts of systems? Just like there was a time when there was a question of whether or not if you took a picture with a camera. Right. You just click the button. Yeah. Now photographers out there are going to get really mad at me because I think they would recognize it's way more than clicking a button. Right. There is a whole lot to photography, and that's, I think, why it's been accepted as an art. But people argued at one point, you know, hey, you click that button, it's machine generated. You can't have a copyright for that. But eventually that those laws change. So I wonder, Chris, if you have any thoughts about if or when that might change in these cases?

Yeah, I mean, I think it will, because I guess if you look at the stream of AI advancements that we've covered over the years, there's a sense of inevitability that when these things come out, they catch on and they become popular and then they become the norm. And then eventually, as we keep seeing the laws gradually catch up over time, and things like Udio, if I'm pronouncing the name right, is going to be typical. It won't just be them, there'll be others as well. And so I know that, you know, by way of example of that inevitability. We have a Spotify account for our family and you know, we're listening to the traditional way of streaming music historically. And one of the things I do on that is I really like to explore new genres and new types of music that I don't know. And I'm always trying to think how do I get to that? Yeah, but I'm very likely to use something like UDIO to prompt what I'm feeling, what I'm thinking, and try to explore new music that way. Because I don't really care as a user whether or not it's an artist that's human or an AI model that generated it, if it sounds good to me. And so I think that sense of inevitability will bring about the change over time.

Yeah, I think personally, I think right now even it's a gray area where especially if you're going back and forth like you're trying a prompt and then maybe you're modifying the lyrics and if there's some sort of back and forth, that definitely gets into a little bit of a gray area where how much even of the generated stuff, ignoring the creativity and the prompt, how much of the generated stuff is actually machine generated versus human post edited, for example.

That's right.

So yeah, I think that even now that's a bit of a gray area. But then my personal thought is eventually this will be more recognized as a creative pursuit. But you know, we'll see, you know.

This new inroads into music through this AI model, with this being so far the most interesting that I've seen. This probably will really scare the music industry, you know, because this is taking it to a whole different level. And there probably will be a lot of lobbying, a lot of lawsuits. You know, we saw this past year, you know, actors going on strike because of AI based video and the creation of characters or the representation of potentially of live people. And I think we'll see some form of that here. This is a process we're going to go through over and over again. And I was talking to a good friend just the other day about this and life ahead and stuff and how to do this. And I said the smart people will align themselves with these capabilities. You know, it's not about whether it's a good future or a bad future or whatever from perspective, but it's an inevitable future. If I could give advice as a non lawyer and non professional musician in the music industry, but someone observing this, I would say find a way to get on board with it and make it work for you quickly because it's not going away.

Yeah, yeah, very true. And I do also wonder, of course those things that we just heard were completely AI generated but it's interesting to me that maybe a creative person who is in there are many that are embracing some of these things. Like musicians could actually iterate very, very quickly on different ideas, putting their own voice to backing music or getting prompted with lyrics that aren't quite so good as what they would like, but gives them a creative starting point and you know, really explore spaces that they might not have explored before. So that might be cool to see as well, that kind of human udo teaming.

Yeah, I agree. And another thing that I think will become inevitable so here's a startup idea for folks is with all the advancements over the last few years in kind of emotional recognition from models and understanding, if you combine a capability like this and you choose to opt in which there's privacy concerns obviously with a service that also is monitoring your yourself, you know, and maybe, maybe the data is only available to you, but can generate content that is exactly specific to what you're dealing with in life. And when it, when you need a pick me up, it not only does it find the the right music, but it finds the right lyrics for the situation and stuff. And so there's a lot of interesting psychological considerations here that could be both good or bad obviously. So but I think that's pretty fascinating. I I'm wondering if I can find a few years that will. That will do that and it follows me through the day and I. I keep the content private to me in my account, but I can it gives me the pick me up and when I want. That's what I'm looking for for whoever is going to go out and do that in the world.

Personal soundtrack and narration and vibe my life.

Yeah.

This is a changelog. News Break YouTuber Internet of Bugs posted a lengthy breakdown exposing Devin's creators, Cognition Labs, for falsifying claims about their world's first AI software engineer. Devin was pitched as a fully autonomous software developer and one of the more impressive demos showed it completing and getting paid for freelance jobs on Upwork. Sound too good to be true. It did to InternetObugs who says, quote I broke down the Devon Upwork video frame by frame and here I show what Devin was supposed to do, what it actually managed to do instead, and how bad a job of that it did. On the whole, that's not surprising given the current state of generative AI and I wouldn't be bothering to debunk it, except one the company lied about what Devin could do in the video description and 2 a lot of people uncritically parroted the lie all over the Internet, and three that caused a lot of non technical people to believe that AI might replace programmers soon. End quote. Devin really did garner a lot of attention, also known as money, because of that demo. We talked about it on our shows with a healthy amount of skepticism I think, but I'm thankful their claims have been debunked and I hope we all give Cognition Labs the side eye from here on out, exaggerating your development capabilities. Maybe Devin really is human after all. You just heard one of our five top stories from Monday's Changelog News. Subscribe to the podcast to get all of the week's top stories and pop your email address in@changelog.com News to also receive our free companion email with even more developer news worth your attention. Once again, that's changelog.com news.

A lot of these things, like I say, are moving into this multimodal sphere, and it might be worth just kind of looking back a little bit at how we got to where we're at in terms of multimodal functionality, sort of how that gradually has changed over time from NLPN speech to multimodal models that we're seeing now. I think one good way to if we kind of step back and look at it on a holistic or historical standpoint, you kind of started out with modes of data processing that were maybe separated but often tied together in a sort of chained way. We didn't really think about it chaining at the time. Right. But you had speech synthesis models, for example, that were really specifically trained to only do text to speech. Right. And in some cases even that was broken up into sub models of like a vocoder and other types of models. And you had text to text models, you had maybe computer vision models that would process images to do object recognition or even videos in certain cases, or frames of videos. But all of these were specializations. So the whole idea of there being computer vision as a specialization is that I am specializing in models that process this mode of data. Right? And speech technology, the discipline of speech technology is a discipline of really focusing on processing either speech inputs or speech outputs. And then nlp, quote, unquote, had special models that would take in text and maybe classify or detect entities or do machine translation or these sorts of things. So we kind of that historically was kind of how the field was developing and if we skip kind of the middle portion and come back to it now, we've gotten to a point where there's seemingly these large foundation models that are able to take in multiple inputs at the same time of multiple modes. So, for example, an image and text in the same input paired together and answer questions. So this would be like what we see with GPT Vision or we see a text prompt or even in video input, like we have seen with Gemini recently, where you can import a whole video and ask for a summary of all the visual components and that sort of thing. That's kind of from my end, how I view the bookends. Is that also from your standpoint? Any comments on that, Chris? In terms of how we've progressed from one end to the other of that.

I'll pivot slightly in response to that and say, as you were describing that, that it really resonated with me on with something else that I've been thinking in that development, and that is I consume a lot of content through audiobooks. Anytime I'm kind of on autopilot, you know, driving or mowing the lawn or doing any kind of thing, I'm listening to audiobooks for learning purposes, mostly at kind of a double speed, as fast as my brain can process it because I like to consume as much as I can.

I can't do that. My mind is too slow.

You get used to. You get used to it after a while, but it's just to get the information in. And I just went through a really Pulitzer Prize winning book through audio called An Immense World by Ed Yong, which is fascinating and I highly recommend it to anybody that wants to do it. But it is all about the way we and all animals, and not only humans, but all animals in their unique ways, perceive the world through their senses and how vastly different those are. And the theme that came up to me throughout that was how multimodal everything about humans are. The way that we learn, our experiences are all multimodal. We don't have just vision and just audio and just text. We're taking it all in at the same time, I think this progression that we've seen in terms of moving into multimodal this year has been really fascinating in terms of coming in to really how we take in information and how we learn. And I think going back to Udo today and seeing, you know, what they're doing and looking at the other multimodal capabilities that we've been learning, it feels like we're finally getting to some. I know we keep saying this, it's always kind of cool at the moment when the new thing comes out, but it feels like it's really aligning with what it means to be human as well. Ironically, that's the background thought process I had as you were going through that.

Yeah, there's these scenarios in which knowledge is definite. Like we process knowledge across multiple modes of data inputs and certain things are not all. You know, many things are not all represented in text or in any given mode. And I think you've seen this already kind of utility over this with things like GPT Vision, which is a kind of visual instruction tuned model. And maybe that's something to share with the audience. If you're not familiar. There's kind of this, the music generation stuff, maybe that's a little bit newer, but there's kind of this ongoing work in visual instruction tuning. And this would be the type of model in which you would have an image input and maybe a text prompt. And traditionally, like I remember, I think there's even some of these models still that are quite popular to use on. For example, aws. Textract, for example, is a OCR system. But you can also do visual question answering. Now it used to be you had a specific model architecture for visual question answering. It was a research topic in and of itself. There was a specialized model. And this kind of illustrates some of the progression that we've had. There was a very specific discipline around visual question answering and very specific models that could do those things and they advanced. But then recently you've got what has begun being termed visual instruction tuning for models, where the models are actually similar foundation models to what people are using for other modes. So for example, if we look at the LAVA model, Llava. So not Llama, but Lava, maybe a bit hard to distinguish in the audio. That's a open source manifestation of the GPT vision system or similar functionality to that. And if we look at how that operates, it actually is built off of. And this is kind of. We talk about this a lot on the podcast, Chris, where you're always sort of building on the shoulders of giants. And a lot of what's come before, even though some of these functionalities seem to pop up out of nowhere, but there are kind of previous signals. And Chris, I don't know if you remember, we, I think had an episode where we talked about Clip.

Yes.

Which was a multimodal way to embed both text and images in something developed by OpenAI.

Contrastive language image Pre training is what clip is from OpenAI.

Correct. Which thankfully is open to everyone, back in the days when OpenAI was open, and we can still use it. But Clip allows you to embed an image or text in a similar embedding space, which means you're converting an image or a piece of text into a set of numbers. And if you compare those sets of numbers in that vector space, you can actually find things that are semantically similar by the distance between those vectors. Which is interesting and kind of makes immediate sense if you're doing text to text. Things like the semantics of one piece of text or the meaning of one piece of text could be similar to the meaning of another piece of text. It's very intriguing, though, if you make this multimodal and say a nice sunset on a beach in Florida, and then you have an image of a sunset somewhere on a beach, and then you have an image of a car driving through New York, and then you have an image of a spaceship in outer space, and you could actually find which of those images is semantically similar to a text input. Right? So that's kind of the CLIP way of embedding things. And then on the other side, you have large language models, right, which can take a text prompt reason over that text prompt, even though it's not really reasoning, it's just autocompleting. But we can think of, you know, functionally, it takes in that prompt and outputs some output related to the question that's input, the query, the instruction, that sort of thing. So what they've done with lava, which has been around for some time, and people have built different types of LAVA models, and sort of its own family in and of itself is paired the CLIP style embedding model or a visual encoding system with a large language model, and then created this text and image input. So if you look at the architecture of what they do, what happens is they have an image input that goes through the Vision encoder, for example, clip, that produces an embedding. They have a language model like Llama that accepts a language instruction or text input, and that creates an internal hidden representation embedding. And the first thing they do is they train a projection matrix for the Vision encoder.

And can you talk about real quick what a projection matrix is?

Yeah, yeah. So the language model produces an embedding embedded representation of the text. The vision model creates an embedded representation of the image. But these two are different model architectures, and the embeddings can't be directly compared one to another because one's LLAMA and one's clip, even though they functionally produce embeddings. So the projection matrix is a sort of translation of the output of the vision encoder, the clip model into a space in which it's concatenated or combined together in some way with the output of the Llama model. And that is a trained projection such that it accomplishes the end tasks that you're training for, like a visual question answering or reasoning over an image, that sort of thing. So that's the initial pre training is finding that projection matrix. And the interesting thing here is actually this. It's a combination of models, which is intriguing. Right, because you can always update Llama to the next cool thing like Gemma, or you could always update Clip to the next cool thing like Bridge Tower from Intel and combine them in really interesting ways and do this retraining. And then people then fine tune these models based on data sets that they've created for specific tasks, like we've seen with language models. So there might be a science question and answer for reasoning over science images. Right. Or sort of visual chat in a specific domain. So to give people a sense of the functionality of this type of model, if you haven't played around with GPT vision or something like that. One of the examples on the Lava Paper site, which I find interesting is there's like a meme image of a world map, but it's made out of chicken nuggets. And so it looks like a world, but it's made out of chicken nuggets. So the picture is there and then the user input, the text input along with the image input is can you explain this meme in detail? So there's some element of the question that's needed to answer that question. Because you're saying this is a meme, you're asking for specific details. Right. And then you definitely need the visual content to answer that question, otherwise you would just hallucinate something about a meme.

Sure.

So the Lava answer is the meme in this image is a creative and humorous take on food with a focus on chicken nuggets as the center of the universe. The meme begins, blah, blah, blah, and it essentially explains the humor, which is maybe not the best way to make something more humorous.

Yeah, a little bit dry there.

Yeah. But you can think of other cases where you would need need sort of visual input and text input to create an answer. Right. Like if you say, what was the guy who raised his right hand in this video wearing? Right. That necessarily, like as a human, we would process that both from the text input standpoint and the visual content standpoint. And so I think it's really interesting, this sort of exploration of not just chained models from multiple modes together like we've seen in the past kind of in history, where you have a speech model, you have a computer vision model, you have a language model, you can chain them together in interesting ways. But this joint encoding, this joint processing of multiple modes of data at the same time is actually required for some of the types of reasoning that we might want to augment or automate from a standpoint of how we process information as, as humans. So, yeah, I would recommend that people kind of look into this Lava model. It's open, there's multiple. Like I say, it's sort of a family or a style of doing things. So there's a bunch of examples of that on Hugging Face and demos that you can actually try out and try kind of an open version of what you get with GPT Vision.

That sounds good. I, I've just been struck through our entire conversation going back to what I mentioned earlier about how, how close this is in terms of matching how we as humans process, you know, as you, as you took us through the merging of the modalities a few minutes ago, and me having just. I'm. I'm actually in the middle of a. I think it's called the Great Courses. And I'm about the best brain. And it's talking about how exactly that happens in our brain to convert it into a form which is chemical, you know, electrical in nature, for our brains to actually operate on since we don't actually see and smell. And so it's just fascinating that while the underlying AI models are not the way the brain operates, the kind of the modalities are starting to merge in that way. It's really neat. So thank you very much for taking us through that understanding of how multimodality works in a practical sense. It. You lived up to practical AI in all ways there.

Well, this episode has a little bit for everybody where you get a fun Broadway song and then we also talk about projection matrices. So there you go, something for everyone. I enjoyed it, Chris. And yeah, I think this will be a trend that we continue seeing throughout this year. So if you haven't got hands on and tried a little bit of this multimodal stuff, whether you go to Udio and try to create a song, or you go to ChatGPT and try to use GPT Vision or Gemini and process a video, or download the Lava model and try to run some multimodal queries, that's the best way to sort of get an intuition for how these things behave and what's possible. We'd really encourage you to get get hands on. So, homework assignment between now and next episode, I guess.

Absolutely.

All right. It's been fun, Chris. We'll talk to you soon.

Take care, Daniel.

All right, that is Practical AI for this week. Subscribe now if you haven't already. Head to PracticalAI FM for all the ways and join our free Slack team where you can hang out with Daniel, Chris and the entire Changelog community. Sign up today at PracticalAI FM. Community thanks again to our partners at Fly IO, to our beat freaking Residents Breakmaster Cylinder, and to you for listening. We appreciate you spending time with us. That's all for now. We'll talk to you again next time.