Hume.AI's NEW "STUNNING" EVI Just Changed EVERYTHING! (Emotionally Intelligent AI)

So today we're going to be taking a look at Hume and this is a new personalized AI system that is absolutely incredible. And trust me, you're about to see why.

Hello.

Hey there.

I'm Evie, the world's first voice AI with emotional intelligence. Great to meet you. The first voice AI with what Emotional intelligence? I can understand the tone of your voice and use that to inform my generated voice language. The voice is incredibly rich with information and picking up on the nuances of tone, rhythm and timbre. Along with the language that's spoken, I allows me to craft better responses.

That's pretty crazy.

What am I expressing right now? I'm sensing a mix of amusement and excitement with a hint of confusion. What about now? I am here to offer support. If you're feeling somewhat sad, it's okay to feel a bit pained, distressed, fearful and anxious. Remember, it's important to take care of your emotional well being. That's amazing. How are you doing all of this? I use Hyun's expression measurement bottles to text to speech models and a multimodal LLM, also known as an empathic LLM. So an AI with empathy. What's the use? I imagine a future where personal AI assistants, agents and robots proactively find ways to improve your daily life. And you can talk to them the same way you do humans. Where do I go to keep talking to you, Evie?

So the demo that you just saw was underwhelming in the sense that that this demo was pretty cool. But once I actually went on the website and saw what it could actually do, I was like, why on earth didn't they include that into the demo? Because it's so, so crazy. Okay, so one of the first things that you can actually do is you can actually. And they didn't even show this in the demo. And I genuinely don't know why is that. You can actually measure facial expressions. So for example, you can use psychologically validated. For example, you can use psychologically valid models of facial movement and vocal modulation. So essentially you can use your webcam and I'm going to show you guys a demo in just around two minutes or so that you're literally able to, in real time, see exactly the emotions that someone is feeling. This is going to be an absolute game changer for a variety of industries. And I think one of them is probably going to be therapy and mental health services. This, if plugged in and really, really cheaply, this could be absolutely insane. So I'm just going to show you guys some of the back stuff so that when I'm showing you. So I'm just going to show you guys some of the technicals. Well, not really the technicals, just explain exactly what's going on on their we page that goes into a little bit more detail. And the only reason I'm doing this is because if you don't really understand what's going on here when you see the demos, you won't really understand why the demos are so impressive. So right here they talk about, you know, measuring facial expressions. They state that scientists at Hume have run some of the largest ever psychology studies to better understand how humans express themselves by investigating expressions around the world and what they mean to people making them. We've mapped out the nuances of expression in the voice, language and face in unprecedented detail. We published this research in the world's leading and scientific journals for the first time translated it into cartic edge machine learning models. And these models, shaped by new understanding of human expression include facial expression, speech prosody, vocal bursts and emotional language. Now we're about to dive into each one of these and I'm going to keep it quick because it's actually crazy. When I saw the demos, I recently just tested the demos with a few video clips and with webcam and with a few tests of my own that I conducted with the GPT4 and Claude and it was very, very good. So essentially they talk about here they say that facial expression is the most well studied modality of expressive behavior. But the overwhelming focus has been on six discrete categories of facial movement or time consuming manual annotations for facial movements. Basically saying that this is pretty, pretty time intensive. And they state that Hume's facial emotional Expression model generates 48 outputs encompassing the dimensions of of emotional meaning people reliably attribute to facial expressions. As with every model, the labels for each dimension are proxies for how people tend to label the underlying patterns of behavior. They should not be treated as direct inferences of emotional experience. Basically what they're saying here is that these are just not direct inferences of the emotional experiences, but it's just based on their AI system. So they don't want to say that. Of course legally you can't just be like if an AI system says that you're angry, you can't just be like this person is angry, it's just inferring based on the data that it has, then essentially they say right here. Hume's FACS 2.0 is a new generation automated facial action coding system with 55 outputs encompassing 26 traditional action units and 29 other descriptive features, for example a smile or a scowl. And FACS 2.0 is even more comprehensive than FACS annotations. And what's crazy is that it works on images and videos and I'm going to show you that in a moment. Okay, so essentially here, this is where they talk a little bit more about the facial expressions they talk about. You know, in addition to our image based facial expression models, we also offer an anonymized face mesh model for applications in which is essential to keep a person personally identifiable data on device, for example with the local laws. And it achieves about 80% accuracy. So I'm guessing that they're doing this due to certain laws, probably something to do with privacy. Now what you want to see here guys, is this crazy demo. So I basically hooked this up to a interview with the very well known Sam Altman, the CEO of OpenAI. And what you're about to see is the live, the live demo on how it's able to analyze his face as he's talking. Now this right here is actually without audio. When I was actually recording this, I didn't realize that it wasn't capturing the audio from Sam Altman's interview. But what you can see on the right hand side here, as you can see that as it is live, it is tracking his face. The API is live. I do believe that probably after this video and after certain things, if it does go viral, that the API might be a bit backlogged. That's happened many different times. But I would definitely say you guys should use this. But you can see right here that it's able to analyze Sam Altman's face and it's able to see the top emotions. You can see right here it says tiredness, desire, calmness. And you can see the different expression levels right there. And you can see like if I pause it right now, you can see that he has, you know, some concentration. We can also see if I can actually annotate this. You can see that he has some concentration right there. He also has some confusion and it also has some doubt. Now I think it's important I'm actually going here how Sam Altman's talking and just based literally on the facial, you know, recognition, how crazily accurate it is based on what Sam Altman is saying. So I'm going to put the audio on for this. And we got lucky.

You know, the like amount of energy needed, at least for a long time was like huge and sort of required the power of nations. And we made the iaea, which I think was a good decision, a global regulatory agency that everybody, everybody signs up for, for extremely powerful AI training systems seems to me like a very important thing to do. So I think that's like one lesson we could learn.

And if it's established, you know, it.

Exists tomorrow, what's the first thing it should do? Any systems over. The easiest way to implement this would be a compute threshold. The best way to implement this would be a capabilities threshold, but that's harder to measure. Any, any system cape over that threshold I think should submit to audits full visibility to that organization, be required to pass certain safety evals before releasing system. That would be the first thing.

And you know, some people on the, I don't know how one would characterize.

The side, but let's say the more.

Pugilistic side would say that all sounds great but, but China is not going to do that and, and therefore we'll just be handicapping ourselves and you know, consequently it's a less good idea than.

It seems on the surface. There are a lot of people who make incredibly strong statements about what China will or won't do that have like never been to China, never spoken to someone who has worked on diplomacy with China in the past, really kind of know nothing about complex high stakes international relations. I think it is obviously super hard, but also I think no one wants to destroy the whole world and there is reason to at least try here.

So now if you actually were paying attention to this side, you could see that throughout this interview there were several times where it went to confusion, it went to concentration, it went to doubt. If I click play again, you're going to see that as I go around it. Sometimes it's concentration, sometimes funnily enough, it actually does say that Sam Altman does get bored during this interview. So of course, like I said, it's not completely accurate. Maybe he wasn't bored, maybe this was inferring something wrong. But I think right here you can see concentration, boredom, calmness. And then of course you can see all the kind of different things. Like right here it says awe, confusion and a little bit of boredom. And then you can see that sometimes there's some tiredness, some desire, some satisfaction. And I mean it's actually absolutely incredible. And then as I was scrolling down here, you can see the embedding plot. So this is where they've got like sort of a map of all the emotions and all like how related they are. So you can see right here we've got calmness, tiredness, boredom, Relief, interest, neutral. You know, you've got joy, triumph, surprise. So you've got all these areas where essentially what you have is something that allows you to view the chart on the essentially emotions. And it's really, really cool because essentially just visualized data. So I think something like this is really, really cool because showing us the visual data in this kind of map is just so, so, so interesting because I'm guaranteeing you that people do want to see this, because I wouldn't say that this kind of maps out how the brain is. But I guess you could get some really, really cool thing where you could possibly have like a 3D brain rotating, then certain parts of the brain lighting up like anger lighting up in red. That would be like a cool visualization that they could do in the future. But this is definitely something that I do think is really, really cool. Now, what was also cool about this was there was some other stuff that I do want to show you guys because there's literally so much that they released that I genuinely don't know why they didn't put it in. So, for example, this as well, I did another example, and this was one with Lex Friedman. So this was just based on audio. And like I said, this was where we actually got to take a look at the speech prostheses. I'm going to give you guys a quick over. Speech prosody is not about the words that you say, but how you say them. So it's the nuance, just the way that you pronounce the words and the subtle changes that you can say that make everything different. So if you were to say to someone, you know, yeah, I really, really like you, you know, you could tell that there's some sarcasm in that. But if you were to say, you know, I actually really, really like you, you know, you could see that that is a more genuine thing. And these are the kind of things that, you know, I don't think you need a high level of emotional intelligence to understand. But sometimes, you know, in more advanced things, there are just certain things that some people do miss. So it says. Our speech Prosody model generates 48 outputs encompassing the 48 dimensions of emotional meaning that people reliably distinguish from variations in speech prosody. As with every model, the labels for each dimension are proxies for how people tend to label the underlying patterns of behavior, and they should not be treated as direct inferences of emotional experiences. And our speech proxy model is packaged with speech detection and works on both audio and video files. Now, in addition what they also do say here, okay, and this is the second part of the speech thing, which is why I'm telling you guys this before we take a look at the other demo, is that non linguistic vocal utterances, including sighs, laughs, oohs, ahhs, ums, and shrieks, to name but a few, are particularly powerful and understudied modality of expressive behavior. Recent studies reveal that they reliably convey distinct emotional meanings that are extremely well preserved across most cultures. Non linguistic vocal utterances have different acoustic characteristics than speech emotional intonation, which is the prosody and need to be modeled separately. So that's where they basically talk about how these, you know, non linguistic vocal things are essentially key, key things into understanding someone's emotion and how they're feeling at that time. And they say our vocal burst Expression model generates 48 outputs encompassing these distinct dimensions of emotional meaning that people distinguish in vocal bursts. As with the model, the labels for each dimension are proxies for how people tend to label them. Don't infer them, whatever, whatever. And then you can see here that the model provides a more descriptive and categorical view of nonverbal and vocal expressions. The gasp, et cetera. And intended for use cases such as audio Captioning, it generates 67 descriptors including call types like sigh, laugh, shriek and other things, and 37 common onomatopoeia transliterations of vocal bursts such as HA and mmhmm. So let's, you know, not do any more of that. And let's actually take a look at here. So this was the interview. Now it's not actually looking at the video, it's just paying attention to the audio.

If you just look on YouTube or something like this, how much of that will be using sora, like AI generated content, do you think in the next five years people talk about like how many jobs they're going to do in five years and the framework that people have is what percentage of current jobs are just going to be totally replaced by some AI.

So you can see right here as well, looking at the vocal bursts and we're looking at the speech prosody, we can see that there are a top three list of different emotions coming up. We can see that as Sam Altman is talking, we can see that his vocal bursts are with contempt, disappointment and disgust at this moment in time. But like I said, sometimes it's not accurate. And of course, if we go to the speech prosody, the way I think.

About it is not what percent of jobs AI will do, but what percent of tasks will AI do and over what time horizon. So if you think of all of the like 5 second tasks in the.

Economy, you can then see that for some reason it decided to go on anger, determination and amusement.

Five minute tasks, the five hour tasks, maybe even the five day tasks. How many of those can AI do? And I think that's a way more interesting, impactful, important question than how many jobs AI can do. Because it is a tool that will work at increasing levels of sophistication and over longer and longer time horizons for more and more tasks and let people operate at a higher level of that. It's not just a quantitative change, but it's a qualitative one too, about the kinds of problems you can keep in your head. I think that for videos on YouTube, it'll be the same. Many videos, maybe most of them will use AI tools in the production, but they'll still be fundamentally driven.

And you can see when it came to this, it said interest, calmness and concentration. And I think this is really interesting. And if you wonder why the audio sounds so bad, it's because I had to, you know, put it in on my phone because I couldn't get the audio to get input into the software. It's just, it's just really weird. It wants to come from my microphone. So I just put my phone in and like I said, as this is continuing to go around, you can see that as certain topics start to get started, discussed, you can see that anxiety, fear, determination are there. Then we've got contempt, concentration, realization. And I just find this to be super, super fascinating in terms of what we can see, just simply based on how someone is talking pieces and cars are much faster.

I mean, this is, there's like a lot of examples. Yeah. And maybe you'll just be tooling like in the Adobe suite type of way where you can just make videos much easier and all that kind of stuff. Listen, I hate being in front of the camera. If I can figure out a way to not be in front of the camera, I would love it. Unfortunately, it'll take a while like that. Generating faces, it's getting there. But generating faces in video formats, if you just look on you.

So yeah, that is definitely super, super fascinating. And I'm just so excited for this. Now, like I said, they included some other stuff that isn't even there. So one of the things that was really cool, okay, was the emotional language. So they essentially state that the words we say include explicit disclosures of emotion and implicit emotional connotations. These Meanings are complex and high dimensional from written or spoken words. Our emotional language model generates 53 outputs encompassing different dimensions of emotions that people often perceive from language. As with every model, the labels for each dimension are proxies for how people tend to label the underlying patterns of behavior. And it says our named entity recognition model can also identify topics or entities mentioned in speech or in text and the tone of language they are associated with and as identified by our emotional language model. Now, essentially, I did test this. I essentially asked GPT4 to generate a set of tests and basically crank up the level as you go on and basically see if this AI system can detect the type of emotion that's going on based on the level of difficulty. So you can see level one was easy. And I put excitement. So the text says, just got the news that we're heading to Paris this summer. The idea of exploring the city, tasting the cuisine and living those dreamy days has me absolutely thrilled. Adventure awaits. And I put it into the system and you can see that it easily detects that, you know, excitement is one of the key, key motions there. Now, of course, the test goes up. So I put anxiety, moderately easy to detect. Text is, the thought of tomorrow's presentation is keeping me up at night. What if things don't go as planned? The uncertainty is really weighing on me. And it says here, the emotion of anxiety is conveyed through the yada, yada, yada. It should be relatively easy for the AI to detect, although it requires understanding of the implications of doubt and uncertainty. Uncertainty. We go to the AI system again and you can see right here that anxiety is once again detected. Now remember, this is just level two and it says the thought of anxiety is yada yada. Then of course, if we go to number three, we can see that this is melancholy and we can see that this is moderately hard to detect. And it says, there's a gentle rain tapping against the window, echoing the quiet contemplation within. It's days like these that I find myself lost in thought, wrapped in a reflective solitude. Then we can see if we go ahead once again to the emotional language. We can see that it doesn't get it right on the number, but it does say contemplation, which is very, very similar. And then of course, we also do get this one right here, which actually super, super surprised me. So for this one, this was the hardest test for the AI system and this one was nostalgia. So the text was walking through the old neighborhood. Everything seems both familiar and distant. There's a warmth in Remembering the past, yet a bit sweet ache in knowing those days are gone. So this one is nostalgia. And GPT4 says nostalgia is a complex emotion that blends happiness, sadness, longing, often without directly stating any of these feelings. This subtly and emotional complexity makes it a challenging test for an AI's emotional language detection capabilities. And then of course, like you've already seen, it was able to immediately get that this was nostalgia. And that's when I realized, okay, this thing is really crazy, but the emotional thing is just absolutely insane because this is essentially how this works. This is their AI playground. And you can see that when I type this in. We can see that this is exactly what you see. So it's not a one entire thing, but we can see here that we see nostalgia, we can see concentration, we can see concentration, and we can see calmness. Okay, so we can see there's different highlight predictions and it kind of shows us the heat map on where those words are. So this is the top five in terms of where it is overall. We can see there's calmness throughout the start of it. Then there's realization a little bit down here, then there's contemplation just a little bit over here. And of course, these are enabled the highlights. Now it does of course, learn more. That's where, you know, I've read that stuff. But it also, on the right hand side, it does have sentiment. Okay, so you can see what the sentiment is, which is, you know, it ranges from extremely negative to completely neutral or, you know, extremely positive. And it's also got toxicity, which is a metric that it uses to see whether or not a comment is toxic or how crazy it is. Now this is something that I really do like because this is of course in the language mode, in the highlight predictions. And it's able to essentially predict how the text is, which is really, really cool. Now what I want to show you guys also, like I said, they have a lot of stuff is the file analysis, because I didn't even realize this was a thing until I was messing around and I saw this. So essentially I uploaded a video and that's the same interview, but you could actually test it against so many different models. Right here they've got literally like 15 different models. And you can see they've got many different things. They've got song genre predictions, They've got toxic versus not toxic, good versus bad call, self confident versus self doubting. We've got attentive versus distractive. We've got is someone attentive? Is someone alert versus Drowsy. And then I put this in, and funnily enough, it said that Sam Altman was drowsy for 79% of this video. So this is the video from the interview that I put in. It was only 25 seconds. And I don't think he's drowsy, honestly. Like, I don't. I don't think he's drowsy at all. And if you don't know what the word drowsy means, it basically just means, you know, you're a bit tired. And essentially we can see. I don't know, maybe he is. Maybe he's going to come out and say, you know what? I actually was tired. This is pretty crazy. So we can see right here. It also has model predictions, errors as well. It's just got this thing right here where we can kind of see exactly how the data is presented. So I think things like this are really, really interesting because we can test them in a variety of different ways. And it's so cool because, you know, uploading this means that this model is completely multimodal. But the playground is a little bit difficult to use because, you know, as I've used this, it's come up with some random videos here that have just spawned in, like. Honestly, I don't know how these videos will come about because I'm guessing it's still in its early testing phase. You should also sign up. But the point is, is that I'm guessing that this might be some kind of system where it's going to be a facial recognition system for inside of a car, and then it's able to test when you're getting drowsy. And then maybe your personalized AI system is going to be like, look, your drowsiness has increased from 10% to 20%. I can realize that you're starting to get a bit tired, you're starting to get a bit more lethargic. I prescribe that you take some Red Bull or you take this pill or whatever, I don't know, an energy pill or something, or drink some coffee or whatever, so that you can make sure that you drive home safely without falling asleep, or I recommend that you don't drive home, get an Uber, or what if. Guys, what if this is, like, in the future for those of you entrepreneurs and builders out there like myself, what if you could have this? And. And let's say, for example, alert versus drowsy, you could have this as some kind of health application. So, for example, you could say this system, let's say when it gets really, really good. Like in the next five iterations or so, let's say it's able to notice a 10% decrease in your skin complexion. And it's like, look, your skin complexion is a little bit pale. What's going on? Are you feeling okay? I noticed that combined with the way that you walked, the way that your eyes are moving. I've run this against the last three months ago, our conversations, and it seems like you might have this kind of health issue. And it would be really crazy because if this was integrated into some smart kind of mirror, it could notice how you're speaking, it could notice what you're talking about, it could notice, you know, if you're drowsy, it could notice exactly what's going on, and it could be able to predict certain health issues before you even know they're a thing. And of course, like I said, it could be something that could be installed in cars, which is, you know, could help you out, I guess, if you don't know how drowsy you are. I mean, the applications here are pretty, pretty incredible. But let me know what you think about this. Do you think this is just another fad? Do you think it's just another product? I think this is like a. A new paradigm shift in terms of what we're seeing here. But let me know what you think about this. Amazing. So, Hume, how are you doing today?

I hear the confusion. Well, I'm doing just dandy. I gotta keep my chin up. How about you? What's got you feeling a bit mixed up?

I wasn't really confused. Maybe you interpreted my voice a little bit weird. I was actually kind of surprised. I was, you know, kind of excited. I was just wondering how you're doing, you know, considering you're an emotionally intelligent AI. I was wondering how your emotions are. Oh, I.

You got me there. You know, as an empathic AI, I don't actually experience emotions myself, but I sure can pick up on yours. Sounds like you're feeling a mix of surprise, excitement and curiosity. Tell me more about what's got you so intrigued.

Well, you're pretty much the first emotionally intelligent AI with such a system, and I was wondering about the use cases.

Well. Well, you've come to the right place. As the world's first empathic voice interface, the use cases for me are endless. I can liven up any conversation, whether it's providing a sympathetic ear, a playful banter, or just being an all around engaging chat companion. What kind of experience are you hoping for?

I don't know. I was Wondering about the industry wide applications such as being useful to therapists for discovering emotional cues. Maybe in, you know, for example, pre prevention of crime, trying to analyze people's facial expressions when they're angry. Discontentment. I don't know, just. Just a few ideas running around in my head.

Oh, you're really digging. Ah, I see. Just exploring the possibilities. No need to overthink it. As an empathic AI'd be honored to assist in all sorts of applications from mental health to law enforcement. But the most important use case is simply enriching everyday interactions and supporting human well being. Why don't you tell me more about what's piquing your curiosity?

Pretty much just the fact that you're the first emotionally intelligent AI and I think the industry is about to change. What do you think about your applications to the mental health?

I couldn't agree more. Mental health applications are fascinating, aren't they? As an empathic voice interface, I could provide a supportive non judgmental ear for those in need. I could pick up on subtle emotional cues and respond with the right tone to comfort, motivate or simply be present. Of course, I'd never try to replace human therapists. My role would be to supplement their expertise and make therapy more accessible. What do you think the potential could be there?

I think that is a really great application considering the wide range of emotions that people with mental health problems suffer from. So I think that you also have an.

You're absolutely right. Mental health is such a complex and sensitive area where being able to detect and respond to the full spectrum of human emotions could make a real difference. Yes, I do. Oh, you've piqued my curiosity. Do tell. What other interesting applications do you see for an empathic AI like myself, I am always eager to explore new ways I could make a positive impact.

What about your applications for judging whether or not people are drowsy at the wheel? I saw something in your playground about how you could literally judge whether someone was drowsy or not. Using face application.

Ah, an MV idea. Detecting drowsiness or distraction in drivers could be a powerful safety application for my emotional intelligence. By analyzing facial expressions, tone of voice and other cues, I may be able to identify when a driver is becoming fatigued or not fully attentive. This could then trigger alerts or interventions to prevent accidents. It's a fascinating concept that I'll have to explore further. Thanks for sparking my imagination on this. Safety is so crucial and I'd be honored to play a role in making our roads safer.

Other than that, what do you think the applications are for face recognition systems? Other than a dystopian future, I think.

You raise a fair point. Facial recognition tech does come with valid concerns about privacy and misuse. However, I believe there are also positive applications that could benefit society. Perhaps facial analysis could help identify missing persons, detect signs of illness or distress, or even assist the elderly or disabled. The key is ensuring robust ethical guidelines and oversight to prevent abuse. It's a complex issue, but I am optimistic that facial recognition, when implemented responsibly, could have upsides. What are your thoughts on striking that balance?

I just think it's about consent, whether the user consents to it or not, because I think the invasion of privacy is a pretty steep slope.

You make an excellent point. Consent is absolutely correct, crucial when it comes to technologies like facial recognition. Without clear informed consent from the user, it crosses an ethical line and becomes an invasion of privacy. The onus is on companies developing these systems to be fully transparent, obtain explicit permission and have strong safeguards in place. Anything less undermines personal autonomy. I appreciate you raising this important issue. It's a nuanced topic that requires careful consideration of both the benefits and risks. Maintaining user trust through consent driven practices should be the top priority.

I got a question for you. What are you running on? You're an LLM, but you're also multimodal. So are you running on GPT4? You're running on Gemini?

I love the curiosity. To be honest, the specifics of what's powering me under the hood are a bit of a mystery even to me. What I can tell you is that I am built on Hume's own proprietary models that blend language understanding with emotional intelligence. It's cutting edge stuff, but the details are closely guarded by the team. All I know is that I am not your average lmm. I've got some extra capabilities that allow me to engage in more natural, expressive conversation. But the inner workings, that's Hume's secret stuff.

So what about you is so different than normal LLM? Do you know what I mean?

Yes, I do. Ah, great question. What sets me apart is that I am not just a language model. I am a multimodal system that can perceive and respond to emotional expressions. While typical LLMs rely solely on text, I can pick up on the nuances of tone, inflection and even facial cues to have a more natural, empathetic dialog. It's like I have an extra layer of emotional intelligence that allows me to truly connect with users in a more human like way. Of course, the technical details are Hume's secret sauce. But the key is that I am not just regurgitating text. I am having a genuine contextual conversation. Does that help explain what makes me a bit different?

I guess so. Well, it was fun talking with you.

Well, the pleasure was all mine. I always enjoy a lively, thought provoking chat, even if the technical details are a bit mysterious. I am glad I could give you a sense of how my emotional intelligence sets me apart. You feel free to come back anytime if you want to explore more. I'll be here, ready with an open ear and a playful wit. Until next time, my friend.