#14: Hume.ai

Yesterday, Hume took the Internet by storm by showing off a new kind of AI, one that understands our emotion. Are you angry? Are you happy? And I'm so honored to have Alan, founder of Hume, on with me today. Introduce yourself a little bit. Who are you?

I'm Alan Cowan. I'm the CEO and chief scientist and founder at hume. We are an empathic AI lab that is dedicated to optimizing AI for human well being. And as you just noted, we just released our new product which is an empathic voice interface that can be integrated into any application.

I'm happy to announce that I am now taking sponsors for unaligned. I'm proud to have AI Top Tools as our first official sponsor. One of the most important things for me running my podcast is staying on top of the industry, knowing all the players and what they can do. I made a complete list of companies in the AI industry on x.com but no one has as comprehensive a library of tools as AI Top Tools. They break down products by use case like productivity, media, chatbots, customer service and more. A lot of companies are looking for the exact right tools for their needs and there's no better place to find it than AI Top Tools.

Empathic voice interface. What does that mean? Because normal people have no idea what empathic means. Right? Voice interface. You know, voice interface is something like a Siri or an Alexa or If you're using ChatGPT to talk with it with your voices. Am I along the line? Right lines, yeah.

So that's a voice interface and most of those just operate by transcribing what you're saying into words and then putting that through some kind of AI that figures out what you're asking for just from with language. But a lot of what you're asking for is actually in your voice. And so an empathic voice interface picks up on those vocal modulations. That's what really sets it apart. And in addition to that, it can produce vocal modulations of its own. So that's kind of the key thing, is that it's multimodal, but in a specific way it understands human conversation better.

It's really amazing when you start thinking of a world where you're going to have a humanoid robot, maybe in your factory or your store or even your home, you want that robot to interact with you. Like I said, hey, why are you angry this morning? Or hey, let's play a game to get your mood up. Something like that. Is that where we're going with this? Or tell me a little bit about your dreams of why you're building this company totally.

You want the AI to be able to serve your preferences without you asking. You want this robot to be able to bring you coffee in the morning without you having to ask it to and clean your house and just figure out what you want and have you react to it. And then it's trained on your reactions. It kind of knows if it's done the right thing or the wrong thing based on how you talk to it. That's what we're preparing the world for. And you don't want something that has a hyper literal interpretation of your instructions and you have to really specify and get, get really deep into every little instruction that you give it. And so any AI interface I think is going to require empathy. In fact, being able to infer your preferences is part of empathy and kind of the core of AI interfaces too. So you could say that empathy is the most important thing for an AI interface and we're the ones building the future of that.

Yeah, you're competing with like OpenAI's whisper, which is the voice interface. When you use ChatGPT on an iPhone and you click the little microphone button, you're talking chat GPT, you're actually using the Whisper API. Tell me a little bit how you see the world splitting up. You know, are developers going to have to choose between you and Whisper or are you going to use your tool, your service, in addition to Whisper? Tell me how that's all going to shape up.

Yeah, so OpenAI has its own transcription and text to speech and language modeling, right. So when you use the ChatGPT app, it does all of that, but it just stitches them together. The language modeling doesn't understand your tone of voice, the response, the text to speech doesn't understand your tone of voice, doesn't know what prosody it should be using. So we're the ones who have a kind of unified model for that, which is not competitive per SE with OpenAI at large. In fact, we can use OpenAI, our model, which is an empathic large language model, can call other APIs to get more complex responses. It can browse using Bing, but it can also call OpenAI, which we use on our website Widget. It can also call Claude, which we use in our demo. So it's really a complimentary tool. We're at the interface level and we see these really frontier models as something that you don't interact with directly. You use the interface that we provide to decode your preferences and that's sent down to the bigger model.

If I was a business person running maybe a customer support organization where people are calling in with questions or concerns or whatnot. I might want to analyze those calls for or how frustrated people are. How happy are they at the end of the call. Right. Is that some place where we could use Hume?

Absolutely. I think if you could without humans actually giving direct feedback on whether a call went well or not, which is unreliable and you just don't want to do that very much. If you could just infer from people's voices whether a call went well, then you could train your models to be better and you could train your agents to be better too. I mean, it's a win for humans and AI. AI. And that's something you can do with our product. So you can take into account the whole course of the interaction, all the language, all the voice, and predict whether the call went well or not without having to get human ratings. In fact, we're already doing that and fine tuning models on that data.

Oh, really? That's interesting. So if I had a, if I had a system where there was tens of thousands of call recordings, Right. Because everybody's recording it in these things and we're, we're recording you for customer service purposes. Right. Could I throw 10,000 recordings into you and have it analyze the emotional context of all those calls?

Yeah, totally. We have an API for that. That's our batch API. We're going to be able to do that automatically with the data that comes in with our empathic voice interface. So if you opt to record it, you, you can then fine tune on that data and you can use it to predict how well each call went.

Wow. And you probably could then build a system to assist the service agents, you know, and understand a little bit. Oh, this person's really frustrated. Let's, let's figure out how to de escalate the call. Let's figure out how to listen to them or give you points for turning it around. Right. You know, and playing a little game with the service agent, something like that. Am I thinking along the right lines there?

Yeah, I think so. Really, you can think of it more broadly than just giving feedback on emotion. That's something we pick up on, but that might not be the actual feedback the human needs at the time. It's really like a three entity system. The agent, the customer and the AI. And at the end of the day, you're just optimizing the AI's output so that the interaction goes as well as possible. And that means that the AI figures out what to tell the human agent to say, like, oh, maybe you should say this, or maybe you should say that. Maybe it doesn't comment at all because it knows that if it does comment, it'll just be distracting. It can figure it out on its own. So anytime you have an AI in a system like that, it should be learning on its own what to do. And that's what our feedback enables it to do. We can figure out what's going to make the customer frustrated. What should the agent do? And then the AI can provide the right information.

How many different emotions can Hume sense? You know, obviously, obviously being happy and angry, probably obvious ones, but how many different emotions can you, can you sense in a phone call?

So if you just look at what humans can discern, like humans can. Can say, like, anger and contempt are different, different tones of voice and, and that's subtle and love and amusement and positive surprise versus negative surprise and awe and interest and confusion and bored. These are all things that have patterns of voice modulation that are associated with them. And it goes even beyond that. When you get into, like, vocal bursts, like laughs and size, then you get even more nuance in that space because that kind of unlocks all this phonetics and screams of distress and all that. We're able to capture everything that humans can capture. And there's also things that humans pick up on that they don't even explicitly acknowledge. It's not something you can ask and have them rate, but there's just some implicit processing that's done of the voice signal, and we can capture a lot of that as well. Because our models aren't just reliant on human labels. They're actually trained on the course of a human interaction and predicting what's the next word going to be based on the expressions that have happened so far. So we're able to pick up on. I don't know if we can necessarily count how many distinct dimensions there are, except to say there's at least 25 in the voice. And once we add facial expression, there's at least 35. So there's a lot of nuance there. And that's like with every single word that's being spoken.

When I got a tour of Sri Lab, like 12 years ago, long time ago, they showed me a system they were building, one of the early, early voice systems. They had people read 200 words off of a card and it could tell whether you were suicidal or not. Is that something that you're trying to look out for, that somebody really, really in trouble mentally so that you can Maybe get them mental health help or, you know, send them to somebody, send them to a professional and say, hey, this person really needs human touch or something like that.

Yeah. We actually work with clinical research labs that are studying symptoms of depression and how they're exhibited in people's voices and facial expressions. And there's studies where they have diaries every day. They don't use our empathic voice interface yet. It's just the person speaking. But once we introduce, our voice interface is going to be even more powerful. But we can already pick up on facial expressions that pretty closely track whether somebody's going to react positively to a treatment over time, which is something that usually it takes a long time to figure out. But if you can figure that out faster, then you know whether to switch treatments or to stick with the treatment that they're getting. And in the case of some of the research, it's things like deep brain stimulation, so very invasive treatments for suicidal, depressed patients. But that's on the treatment side, on the screening side. Then there's also, if someone's calling in, you want to get them to the right help as soon as possible. And so being able to screen people, but just want to emphasize that's not a replacement at all for human doctors coming in. In fact, this is not a diagnostic tool. It's ultimately the doctors that makes the diagnosis. But this can really help you get to the doctor faster and get the doctor to make a diagnosis faster and track the treatment progress faster.

No, you, you know, okay, let's split this, this up into two things. I talk to people who are building 911 systems, right. So you're having somebody under stress, you know, their house is burning, they're. They're getting attacked, or, or somebody is having a real medical situation. Could this be used in a 911 system to really understand the context of what's going on on the phone? Call faster so that people get better care sent to them.

Yeah. There again, I think you have an operator on the line, but you could facilitate their ability to respond. So instead of the operator having to go through and connect you to somebody, it could just recognize that you're in distress. And then it surfaces a button to the operator that says, Call 911 right now, and we'll send someone to their location. So it can really facilitate the process. And we can distinguish between things like whether somebody's in pain, whether they're in physical danger, whether they're in psychological distress, like pretty easily from their voice.

Wow. The second part of this, we haven't talked about multimodal or looking at the camera. You hinted that your system can look at facial, you know, reactions. Is this first version that you've turned on, is this just for the microphone and. Or where are you going with the multimodal? Because I have a second part of that question once I hear, hear your answer.

This version, which is like V0, is for just the voice, but we have an algorithm that takes into account facial expression as well and does all the same things and does them a little bit better. Because with facial expression, you know, when somebody is done speaking more accurately and you're able to respond more accurately and so forth. That will come out soon. There's some work we need to do technically to get that to be reliable, but it's super exciting. And I don't mean months away. It's probably a month or two. But we also want to add some other capabilities in that make that particularly intriguing for people. So they really do want to add a video interface. Because if you're going to add a whole new modality, it has to be worth it in more ways than one to be able to go through the effort of adding that.

The reason I'm asking is I talked with one of the designers at a body cam company for police who have to wear a body cam to watch what's going on and have proof that they made a proper decision to use deadly force or not. And I could imagine a system like yours could be used in a body cam to watch, you know, what the interactions with the general public watch the general public watch somebody they're interacting with, are they, are they angry? Are they, you know, mentally disturbed? Are they autistic? Right. And just having a trouble dealing with a situation. Right. That, and I've talked with these people and they're thinking about a world which would actually assist the police officer and tell them, no, this person's autistic. You use your autistic protocol on this person. Don't go, don't go pulling your gun out right now. It's not, that's not appropriate. Right. But if there's a different situation, well, you might get a green light from the AI saying, hey, you have approval to use deadly force because we see this person is coming at you and is, is exhibiting behaviors that are dangerous to you and you're now allowed to use deadly force. Is that something that you're starting to think about, like that kind of integration into like police force or other, other kinds of people, like retail frontline people who have to do similar kinds of things, maybe not with a gun, but, you know, de. Escalate people.

Yeah, I mean, there's a huge number of opportunities there. I think we need to be a little careful about how things are implemented with sensitive use cases like that. But as a principle, as long as if you ask the person on both sides, do you want this to be on? And they both said yes in most cases, then I think it's a good use case. And there's many cases where that would be the case with body cams in particular, there's studies that show that police officers speak differently to people of different races. And this is established and then it escalates at a higher rate with people who are of certain minorities. And that's something you could easily address just by giving feedback to both people and saying, actually this is probably the best way to handle this and the police officer might prefer that and obviously the other person might too. That's an extreme example, but you can just imagine just what we were talking about with 911 calls. If it can detect if the person's in distress and say, look, I just activated this button for you to call an ambulance. So you can do it right away. And right now, I mean, that just speeds things up. Right. It makes the whole interaction better.

Get a little nerdy. How does the system work for a developer? I assume you're sending a transcript over of what we're actually saying, and then there's a metadata layer of the emotional reactions that your system is sensing. Am I right along that line?

Yeah. So we have a model that integrates. It's a system that integrates transcription and tone of voice detection, which are two separate inputs, puts those both into what we call an empathic large language model, which understands both voice modulations and language. That model does a few things. It tells when you're done speaking, it generates a response and it calls potentially an external API to generate a longer response if it thinks it's appropriate. Right now we use Claude in our demo. We also use OpenAI in our website widget, and then it also generates the right tone of voice to voice that response. So it does all of that with one model. And we have a custom text to speech model that is able to speak it out in that tone of voice and kind of links to our empathic large language model.

You said it figures out when you stop speaking. I'm starting to get all sorts of little devices like this that can listen to you 24 hours a day and take notes and help you out. Right. We've seen the humane pin that has a camera on it as a display, it has a microphone, is always listening to you. We're seeing the tab from Abby Shipman, who is already helping him in business meetings to give him assistance. But if you have a small little battery and a small little device like this and it's always listening, it picks up a lot of stupid stuff. Does your system really help a developer build a better voice interface that can always be listening and always interacting with the customer, even if the developer doesn't really care so much about the emotional side of things?

Yeah, I mean, the emotional side is really. It sounds niche, but actually every single conversation has vocal intonation to it, even if it's just like boredom and interest. And I'm intrigued by this. Tell me more about that confusion where it's like, you know, it can interject and give you a better explanation of something. And those devices, they're going to be not acting in isolation, they're going to be activating interfaces that you're interacting with so it can pull up something on your computer, you know, and that's what makes them useful. And so they're taking actions with these interfaces and they should be able to talk to you about what's going on at the same time. And there it's really important that they're talking and not giving you text because you can't look and read and also look at the interface of what you're doing at the same time. So that's how they can kind of navigate the world with you.

So a future Siri could actually understand that I'm getting frustrated at it because it's not giving me a good answer or doing the right thing. Right. For instance, with Siri, if you ask it, if you park your car and go to a restaurant or something and you forget where you parked your car, you can pull out the app and say, hey, Siri, take me back to my car. Oh, that doesn't work. But if you say, hey, Siri, take me back to my parked car, it actually knows where your car is and puts. Pulls up the map and puts a blue line. Well, if I keep getting an answer that is stupid, I'm frustrated with it. Right. I'm using Ang. I'm changing my emotion. Hey, God damn, Siri. Sorry, Siri, you're not doing what I want. Right. So now all of a sudden, Siri can get smarter about, oh, I'm frustrating the user. That must not be the right answer. Let's try another answer. Right?

Yeah, that's absolutely key. Right. This is how you. So every. Every AI has to have some signal whether what it's doing is right. And right now it's just baked into the AI.

Siri just said I don't understand. Right. Damn, Siri, you're frustrating me. You're not understanding me right.

Yeah. And it can't learn. That's the problem, because it's baked into the AI. And you know, right now it's done with things like fine tuning and RLHF, and that's all done before you deploy into the application. And then it just stays static and doesn't learn anything, which is not what you. This is the way that you can take whatever signals you're getting from human behavior and just continuously learn. And it can learn much more nuanced things than it sees from human ratings. It learns in an application what advice is good, what advice is bad, what's helpful, what's not. Figures out that when you say, where's my car? It means where's my parked car? Figures these things out on its own based on the feedback signal. The feedback signal is key because that's what it needs to learn from.

You can see, you know, if I had a humanoid robot in my kitchen and I'm. And it's cooking me dinner, if it puts too much salt on a meal, I've got to go, hey, that's too much salt. Right. All of a sudden my emotion has changed, my intention has changed. Right. Well, that robot could be using your system to go, oh, sorry about that, you know, and change its behavior and change its learning. So the next time it doesn't put so much salt on my meal.

Yeah, and it's pretty rare that you'll actually explicitly say, that's too much salt. You'll just react to it and you'll say what you're saying. And, like, I guess it's okay. It's like, not great or it's okay. You know, those mean completely different things. And it can learn from your tone of voice, and it can also learn just from how is this making you feel? You know, how do you feel an hour after this meal? That's something that's present in your emotional expressions that it should be learning from, too, not just your direct feedback. And that's ultimately what it should be optimizing for too, is like, let's make this person's life better.

How much does this cost a developer to build an app into, like, a new kind of toy or a new kind of robot or a chatbot? What's your business model?

So it's usage based. So it doesn't cost you really anything except for developer time. But we've made that as little as possible by making it as easy as possible to integrate. So it should be a few lines of code, it's in your app, and then when it's in production, you're just paying for usage. And it's pretty reasonably priced.

So let's say I give it an hour of audio. How much would I expect to be paying for that?

It depends on exactly what you're taking away from it, but between 10 and 20 cents.

10 and 20 cents per hour. That's, that's pretty. It's more affordable than I was.

Wait, sorry, that might be 10 or 20 cents per minute. My bad.

Per minute. Okay.

So, yes, and that's what I need.

A place, you know, I can see a whole bunch of new kinds of toys for my kids which would interact with them, you know, tell them stories at night and be listening to their emotional needs and like, oh, it sounds like you're really depressed today. You know, let's play some games to get you happier or, you know, can we talk about it and talk you through why you're angry or sad or, you know, crying or whatever. I want to use it for that. But, but the, the cost is, the cost is right now fairly high for a certain kind of app, right?

Well, I think they certain kind of.

Business model because you have to, if you're paying, you know, a few dollars per hour for the API, it's for a certain kind of business model, right?

Yeah, we can make it much cheaper over time. So right now, these models, they're pretty costly to run, right? Things are getting more efficient. And there's also, because we run everything on our side, we run the tts, we run the large language model or at least some of the large language models on our side and we can build larger large language models too. We're able to get the cost down a lot. One of the things we can do is we can cache responses so that you're not playing back the whole conversation for the AI every single turn. And that will get your costs down a lot. So there's going to be flexibility on that. And particularly for that kind of use case with a kid, you might not need the biggest language model to respond to them every time. Maybe there's one instance where you bring in a big language model to invent a really amazing story, but that's actually just one query and then you get back the story and then it's just smaller language model that's actually telling the story. And when the kid interjects, it can recover where it left off. And you don't need the big language model anymore for that.

It's a new world of thinking through. Entrepreneurs have a new kind of palette to build new kinds of things for people now. It's really, really incredible. What kind of question am I not asking? Because I'm not in your world, it's hard for me to really think through all the use case, all the things a developer might want to use you for or an end user might want to do with you.

I mean, it's just such an endless number of use cases. Because I think that the voice being four times faster than typing is going to be the most natural interface in the future, especially when you're not in public. Right now I'm not. So why not use an input that's four times faster when you're reading, that's 200. So the voice is speaking 150 words per minute. Typing is 40 words per minute, so it's way faster. And reading is 200 words per minute. Listening, of course, is the same as speaking, so it's 150 words per minute. But while you're listening, you can also be looking at things. So it can be like a documentary. It can be showing you visual elements at ui. And so I think this is going to be the most natural way that people interact with AI in the future. So it's not so much about specific use cases. I honestly think, like, this is going to be the interface for a whole lot of AI use cases. Not really even anything too narrow.

It's a whole new world, man. It's really a pleasure to talk with you. This has been a big week for you, launching your product and showing the world this new kind of AI. Yeah, let's end it there. Unless you have something that you think I haven't asked because I'm out of questions. Because it's an amazing new world that. Where we're going to be able to talk to our computers. Well, here's one last one. One thing I like about Whisper is I can use it at a rock concert and I can have the microphone about a foot and a half away from my mouth and it still understands me, even with a very loud sound system going on around me. Is this technology able to do even better because it can understand the human voice in a deeper way than Whisper can?

I mean, we can use Whisper, so it's not like anything. Yeah, we can use any transcription model we want. So we are Using the cutting edge there. And it is pretty good. And I think there's even some room for improvements. One of the things when we add facial expression and video is we could also use that to improve the transcription. So then if it sees your face at some level. Yeah. You could be at a cocktail party and it would be able to still understand you. So there's a lot of opportunity there.

That's real important. If I'm wearing a pair of augmented reality glasses of the future and you're across the table from me and you're talking in German and I don't understand German, it could be translating you in real time. Right. Well, it needs to listen three feet across the table and we're in a loud, noisy restaurant with a lot of chaos going on. Right. So having even better accuracy rates. When Kai Fu Lee showed me the first voice recognition systems at Microsoft back 15 years ago, they had a lot of errors in a quiet conference room. Right. We're talking about going into a world where we have a robot in our business or we have augmented reality glasses on, or we have a virtual being that we're talking to and maybe at a rock concert. And having these systems have lower error rates is real interesting to me. Is. Is that a side benefit of the work that you're doing that you're able to. Your systems are able to hear things and see things and reduce the error rates in some of these more chaotic situations?

Yeah, I mean, you know, the acoustics that we're adding in are for the most part complementary to the transcription, but we can improve the transcription in certain ways. Like we can say, look, this was a laugh, for example, which traditional transcription systems can't detect. And we can say what kind of laugh it was, or sigh. Yeah. Things that aren't usually transcribed. Yeah. And that actually makes it more accurate when it knows that's in there. And it can be put into the response, too. We can respond with laughs and sighs and stuff, which is helpful. People back channel, too. When you're talking with somebody, you're looking. It's not so obvious when they should come in and speak. They are paying attention to their tone of voice, their facial expression. And they can also say, can you interject? And like. And then you kind of know that they're trying to speak. So there's this very complex, literally sub 100 millisecond dynamic that we're trying to get to where it will really sound or feel like you're talking to a person.

Very cool. Thank you so much. I'm going to try to get a demo from you separately and maybe lay that over. It's been a pleasure. Where do I learn more about you and your company?

You can go at Hume AI to our website. You can sign up for our developer platform, and you can also find us on Twitter, Humai.

Very cool. Thank you so much.

Thanks, Robert. It's a pleasure.