Emotient - Ken Denman

Okay, well, that's your computer, right?

Yep.

Okay, so—

And why don't I just work right off of here?

Oh, can we put his computer on the screen, please? Yeah, somebody's got to be paying attention back there. Let's go.

Okay, so if you watch the blue channel here, I'm going to give a big expression of joy. Now that was a pretty big expression of joy, you know, often facial expression happens very quickly. So if I fire it up a little bit quicker. So this is machine learning based system that's been trained on millions of images of of people of all shapes and sizes and kinds making facial expressions. The work is based on Dr. Paul Ekman's science where he demonstrated that facial expressions are not learned or cultural, they're biological. So irrespective of if you're in a preliterate society, never seen MTV, or you are, you know, living in Manhattan, irrespective of your race, your ethnicity, your gender, Our faces, the muscles of our face, react the same way to fear, to anger, to disgust, contempt. And that really was the ability— gave him the ability to create the Facial Action Coding System, which this technology is based on. So if I did sadness, if I had a sad expression— I'm going to try to do a more subtle expression of sad. Surprise. And surprise can happen very quickly. Anger. Bang. That pegged it. Fear.

That's when you looked at the price.

Yes. Disgust.

The warranty.

Contempt?

The salesperson.

Very, very bad for marriages. Marriage counseling, you get a lot of disgust and contempt, I give it 4 months. Positive and negative is basically a summation of the 2 because sometimes it's easier to work with, you know, a couple channels rather than all of them. All of these channels are working independently, by the way. So we're, you know, they're independent of each other, which allows us to do things like capture blends and do other really cool stuff in the background. We also capture 19 action units. I won't get into that, but that's for when you get into certain verticals. You can build new detectors, new emotions. For instance, in the area of fragrance, there might be different kinds of emotions that are exhibited. And we've had folks work with us, want us to work with them, to build separate kind of detectors around fragrance.

And so now you're going to use this technology, obviously. You're going to put this in Google Glass. And the people who are working in stores are going to know the emotions of the people who come up to the counter.

So we have— I'm

joking.

Well, you actually— Am I not joking? We have a working prototype with Google Glass. You're kidding. I'm not kidding. And now, it does work just as I've done here. If you have the Glass on, as you see people, it will flash the expression that they're exhibiting once the system Grabs their face, hits the weights, delivers us an answer. And you can aggregate that information and put it in the cloud for aggregate analysis. On Glass, though, I think that you can do a lot of damage by pounding people with expression all the time. I think the cool thing with Glass is the ability to basically deliver alerts or alarms. Let's say in healthcare, for instance, just for a second. I've seen this patient over the last 4 months every week. The information that I'm picking up currently bounces against the Tier 3 information and goes into the database, and I get an alarm that says, whoa, they're off the chain here relative to affect in this particular regard.

This patient's very scared.

Yeah. Yes. There's something going on that's different from what we've seen before, and it's significantly different, and we need to take a pause here. So it may be, it may be, for instance, in the, in the area of depression or bipolar, a simple flashing yellow that says, or PTSD, or it may be an objective measurement of pain. Because, you know, when you go to the doctors, oftentimes people say that the nurses will say, well, is it, is it, or the doctor, is it a 4 or is it an 8? By automating measurement of the face, we can give an objective measurement of pain.

Wow. That's mind-blowing and scary.

Pretty cool stuff. Well, we have some of the smartest people in the world in this area. We found these 6 PhDs at UC San Diego, an intact team. My partner and I were doing the research. We were looking for people that knew a lot about this area. We found the most cited, most published team in the world down there, and we convinced them to leave the university and come on this journey with us because we told them, and we absolutely believe this, if When we, as we commercialize this technology, it can have a huge impact, particularly in retail and healthcare initially.

And these just aren't computer visualization specialists, because there's a lot of programs for that. These are also psychology, there's a psychology department that works with computer visualizations departments, right?

These are rare birds. They, it's basically people who have spent their lives, young or old, focusing on this interesting nexus between behavioral and cognitive science, computer vision, machine learning, and— excuse me, computer science and machine learning and computer vision technology. And, you know, think about people with multiple PhDs in these areas. Yeah. That's what we're talking about. And in the case of the 2 more experienced folks, Dr. Javier Movellan and Dr. Marni Bartlett-Stewart, they have been working in this space for 25 years. What's changed in the last 25 years, of course, is the enabling technology has really You know, fast-forwarded, particularly in the last 5 years. So processing power, the algorithms that drive this capability are daunting, or were. You would have— it would have taken a small Cray computer in the sort of early '90s when they started working with Paul Ekman to handle this stuff. Today I'm running our SDK on one core of this MacBook Air.

You could do it on a phone or on a wearable.

Exactly. We've ported it to smartphones. It's kind of mind-blowing. We've ported it to a tablet. We've ported it to Glass. Well, that's client-server, so it's not really—

Yeah.

All

right, so let's see some applications here. Maybe we can go into the deck for a second.

Yeah, you bet. So

what are we seeing here?

Yeah, this is an example of an in-store insertion that we've done in a pilot at a major big box retailer. And what we're doing here is we're measuring affect, customer affect, on exit. So you could do exit entry. Measuring customers' affect as they come in the store, measuring their affect as they leave. And basically, this is a sort of face seconds that we're capturing. We can operate at 30 frames per second or as low as 5 frames per second and get really good signal. So we take those sort of face seconds, if you will, put them in the cloud, and then do the deep analysis. And what you might see, in this particular case, what we noticed is during the course of the day, the affect really begins to drop off. We really need to understand what the heck was going on as we got deeper into the day. It could be myriad things, maybe stockouts or not enough scheduling problem with staff.

I can tell you what that is. That's parents. It's the end of the day, and you're leaving that store with children. I think that's what it is.

As we go deeper and we look department by department, we're going to find interesting data, I would suggest. Then as we bounce against other business intelligence information, we can do interesting correlations.

And do you come to this with, like, an idea of giving suggestions to the business? Because who knows what the average positive or negative experience is leaving a store, and would trying to increase the positivity on leaving Have some impact or not?

So what we come— what we start— the starting point for us is establishing a sentiment measure for retail stores. That's something that they don't have today. They measure a lot of different things, even measure the weather, but they don't actually measure how customers feel when they're in the store. So establishing a customer sentiment measure, I think, is a baseline thought, because once you have that and you take on new initiatives. You put in place a promotion, you do a price change. Now you can understand what impact have you had. Have you improved customer sentiment or has it gone down? So what we're starting with is this baseline thought, Jason.

Has anybody actually done that? Does anybody know of anybody who's done this kind of sentiment analysis on people coming in and out of stores? Nobody. Yeah, so this seems like a— when you bring this to people, have they Done it before? And if not, what is their thought on this?

Well, they haven't done it before. And one of the reasons they haven't done it before is nobody's really figured out how to do this in uncontrolled environments. There's been work, there certainly is work trying to exercise or deliver this kind of capability in controlled environments. We do that today with one of our partners who's here, iMotions, who is a biometric research firm. But the real— the holy grail is sort of being able to do this in uncontrolled environments. All the places where people are, where they're moving around, where they're experiencing merchandise, they're experiencing display, they're looking at product. How are they feeling? So the reason this hasn't been done before is nobody's really figured out how to operate in real time.

So it was a technological issue and a cost issue, basically?

Absolutely.

So this is all done in controlled environments. Previously with bigger hardware, but it can be done with commodity hardware. A $10 commodity camera can capture this now.

Yeah, I used the onboard camera for this exhibit. I had my Logitech 3190 here, which is about an $80 camera at retail, but in volume, you could probably get it for $30.

That's like a 1080p amazing—

Yeah, 1080p amazing.

60-frame craziness.

I mentioned processing power. The other thing that's really moved forward in the last 5 years is camera technology. If you look at all the curves— miniaturization, cost, resolution— all those curves are headed in the right direction. So basically cameras can be anywhere and be very effective.

All right, let's take a look at the next slide and some other applications. What am I seeing here?

So one of the things we did was we did a little bit of work with a partner, Interscope, to look at fans' reaction to advertising in the Super Bowl. Hmm. So we brought an audience of people because we wanted to demonstrate that we can do this with multiple people. It's not one-on-one, but multiple people. So audience measurement, basically. One of the things we tracked, as I mentioned, was— actually, I might not have mentioned this. We also track frustration and confusion. That's 2 of the newer detectors that we're tracking. In particular here, we're looking at frustration over the course of the game. And as you might expect, the Denver folks— I have a home in Denver. I'm from Seattle originally. I was really trying to stay neutral with this game. But the Denver folks are clearly frustrated right from the first quarter and on. And it increases. Next slide, please. But if you go to the ads, which is where we were really trying to focus, the out-of-the-box clear winner Most positive response over the course of all of the ads for the Super Bowl was the Stephen Colbert pistachios ad. And, you know, you can see that when he first came on, there was a lot of very strong positive signal. When they broke very quickly to the H&M ad, and then he— then he— it began to drop off significantly. And then he comes back and he goes, obviously we didn't have the impact we wanted. We're getting reports back that, you know, So he jumped back on and the effect went right back up. So that was the most— probably the best spent money of any of the advertisers was right here.

And this is something that they would do in a lab previously.

Should have done.

Should have done in a lab. So you could do this in a lab, but you could also— if this was a sports bar today, we could be having this technology in there and You know, the Yankees channel, right, the YES Network, could be looking at what ads—

Yes.

What the impacts are having in sports bar of ads, and if anybody's watching or not, or whatever it happens to be, and what the emotional resonance is.

Exactly.

Or this could be Starbucks. Would you be able to get that many people in a Starbucks to capture their—

Yes. So the field of view is simply a function of the resolution of the camera. If you wanted a broader field of view, you simply go maybe spend $200 on a camera. So absolutely. And we see— we are doing a— this week we're going into a pilot in one of the world's largest fast food restaurants. So we're heading into a couple of their pilot stores doing exactly this.

Oh, so we're looking up at the menu.

You're looking, yes.

And we're picking between which—

You're picking your order.

Combo, Happy Meal, or whatever.

Yeah, you're engaging with the service, the customer service folks. You're picking your deal. You're picking from the menu, and we're also capturing the affect of the crew, the staff, as well.

Oh, really? Yeah. So you could know, like, oh yeah, this is a surly person on register 3, and— oh wow, this gets really insane. You would actually know that this person is surly in register 3, and that the customers queued in on register 3 are having a terrible experience.

Yes. Now remember, we're doing this in an anonymized fashion, aggregate analysis. We're really interested in the big thoughts. From the crew's perspective, there's probably a training opportunity. But from the retailer's perspective, there's enormous opportunity to understand what happens when you do dynamic price changes, what happens when you change your menu.

Or what happens when— which person is creating more joy?

Right.

And getting people to order more of the apple pies.

Right. And what happens when they're ordering versus when they're picking up their order? Is there a change? Is there a difference at the drive-up?

Oh, wow. When you're ordering and when you get it. Let's see the next slide. This is fascinating and scary awesome stuff.

One of the other really cool ads that was very powerful was the Gracie ad. The little kid whose dad is explaining to the daughter that she's going to have a little brother, and there's a lot of trepidation on her part. You know, is she going to have to share her parents? Is she going to get as much attention? And being a bright little girl, she begins negotiating, you know, to get the right position. And so she, you know, ends up negotiating for a puppy. And the last scene comes with the mom who's, you know, expecting, and she did not agree, nor did she expect to have to also deal with a puppy in addition to the newborn. So you have this sort of double crossover, this emotional crossover. The negative, it's not really negative, it's different. It's more empathy and nostalgia is what we're measuring. We really parse this. It's empathy and nostalgia as the dad's having the conversation with the daughter. Then as the daughter negotiates for the puppy, you see—

Cathartic moment.

Yeah, a cathartic moment, a lot of joy kicking in. Then they cut to the mom who's looking around the corner and she's just heard, and a puppy, and the empathy swings back up. I can tell you that the difference between men and women was palpable here. Women were much more impacted by this. The general curves were the same, but women were bang.

Was this the controversial ad because it was a mixed-race child?

Because it was a mixed-race family, yes, it was.

So where on there do you see the racism and like the Donald Sterling line of like, oh my God, I'm so offended that there's a mixed-race child in a commercial?

Yeah, I think you catch it a little bit at the front. Are you serious?

Really?

No, no, yeah, yeah, I'm serious. I think you catch it a little bit at the very end where the negative empathy/nostalgia, there may be a little sort of tad of negativism in the back end there, that last peak at the end.

Really? So that little spike in the beginning might be people being like, hey, what's going on here?

Yeah, yeah, yeah. It's like, is something bad going to happen? Or is he going to give her really bad news? What's going on? So they're kind of, you But then— so now the— but I would tell you, this was a powerful ad. This was an ad that people remembered. And I would suggest this was money incredibly well spent.

Let's see the next one.

So moving into stores, one of the things we do with our emotion sentiment platform is measure customer affect. It might be measuring customer affect as it regards to how they react to shelving, how they react to Stockouts, nice display, bad display. And we actually— this is an example of a particular test that we're doing with a particular big box retailer.

Do you have gaze tracking? So when I'm looking at the different combo meals, do you know which one I'm looking at?

Yeah, we don't do eye tracking per se in this environment, but we do gaze detection. We know the trigonometry associated with where people are looking and where the pupils are. We can tell generally. It's not the same buzz swarm that— or bee swarm that many of you have seen in the eye tracking space. It's not that. Not a heat map. Yeah, but it doesn't have to be, because we know generally where they're looking. So

you would know what, like a foot by a foot? Yeah. 2 feet by 2 feet or something like that?

Exactly.

So you would know I'm looking at the strawberry versus grape versus whatever, lemonade or whatever it is.

We know, for instance, that you're on the upper— looking at the upper shelf versus the lower shelf here.

Just a technical question. Where's the camera in something like this, and would I see it, and would that impact my experience if I saw the actual—

Because the cameras can be very small and very recessed, you can really camouflage them very well.

Okay.

And because of the fact that we don't do identity, so we don't keep the images, we don't need the images. We basically take a frame, a single frame, detect the face, take a frame. We turn the face into the equivalent of a landscape map because we use the whole face. We don't do this point stuff. that some other folks do. And it makes us much more accurate, much more resilient to pose. And we turn that face basically into an algorithm. We discard the image of the person. We put the algorithm or the data, the metadata, into the cloud. And that's what we use to analyze it. It's timestamped. Obviously, we—

So the client can't say, like, give me the headshots of everybody who wanted lemon soda.

No, we don't. It goes away.

Got it. So people shouldn't be completely creeped out about this?

We don't think so, because the reality is there isn't a lot of value. I mean, I say this not to be derogatory in any way, but it's reality. There isn't much value in the image of any one of us, but the aggregate analysis associated with groups and subgroups and their response to particular stimuli is incredibly valuable. That's huge. So there's really no commercial incentive to hang on to the image of a person.

Can you figure out people's age?

We do. We— so on the demographics side, we do basic demographics. We're very accurate on gender. We are adding age range later this fall, basic advertising age ranges. And the hard one I expect to have by end of first quarter, maybe into second quarter next year, is ethnicity. We're doing data collection on that one. That's tough. But, but we're getting increasingly accurate. It's starting to move towards the accuracy range of the other 2.

When you do that sort of training of the algorithm, do you just send the screenshots to like a Mechanical Turk type service and say, is this an Asian person, a white person, or a Greek person, whatever? Or do you say, oh, take a picture of somebody and actually know their ethnicity and then put that into the training system?

Both. So we do various kinds of crowdsourcing, if you will, because it's one way you can get data from around the world. But we also have a series of experts, people that have been trained, you know, are very, you know, PhDs, and there's 3 levels of them. They take initial data, they make an assessment, they come to a judgment. The next group of experts evaluates their work and makes a determination. We throw out a bunch. The next group of experts makes another evaluation. Is this really a valid example of anger or a valid anger of disgust? And so we go through a series before, and we're very, very careful about what we put in our training data. That's the holy grail.

Absolutely. Let's see another slide. Or are there any more slides there? OK, so now what are we seeing here?

So we've got merchandising displays. One of the areas that we think we can do really good work is helping with the merchandising effectiveness. Generically, the A/B testing. In the world, retail world today, is things move really quickly. So the ability to quickly assess the effectiveness of a merchandising display is paramount, and we can do that. And so whether it's in-store or frankly increasingly with digital signage, digital technology, the fact that the digital technology allows for adaptive content to move very quickly, even more reason to be able to measure effectiveness quickly. So we can deliver value there. What I would point to though is there is a symbiotic relationship increasingly between in-store retail and online. I think people tend to think about them as really being hypercompetitive. I think increasingly they're going to be additive, complementary, and so the ability to move to measure impact or affect online by incenting people to turn on their camera, and there's lots of ways to do that, including rewarding them to get this valuable information, and then balancing that against what you see when people are in the retail setting. This is going to become a new and powerful science in retail, I would project forward.

Okay, next slide. Where is that at?

What we have here is some sentiment analytics that we've pulled from one of the big-box retailers that we're working with. So for a short period of time here, so if you look at the— on the left-hand side, the bar charts represent sentiment, a broad sentiment measurement in the store. The first 3 days, for instance, I think do represent a sample promotion where they were sort of doing a minor test promotion. There was a staffing issue on the 4th day. They went full store with the promotion in the last 3 days. Now if you look at the— break the data down and look at the emotional response, you see a lot of, you know, generally pretty positive affect around this promotion display. But there's also some interesting stuff that you need to— that can be parsed further based on other data that people—

3% of people were disgusted.

Yeah.

I mean, that is something that if you're running the store, you're gonna be like, hey—

Yeah, you're gonna want to know what that was about. And so you're gonna want to go parse that data by Time, time, time slices.

Surprisingly, only 1% of people are scared at the big box store. I'm terrified when I go into those. So, so you said you were going to— you want to slice that and figure out at what moments?

You're going to do time slices and see if there's a correlation between a particular time window where you got more of the disgust versus was there something going on in that environment at that particular time.

Okay, got it.

You're also going to look at average store receipts in total, time Time slice, an average receipt. So there's a number of other metrics that you can correlate this with to get at causation.

Got it. You could actually look at the profitability of the store at that time. Yes. You can look at the average order size. You can do all kinds of crazy stuff. Do you see this as being a permanent installation where I could, if I was the owner of 15 Häagen-Dazs stores, be able to take out my phone and and be like, oh, what's the sentiment analysis?

I absolutely do. You know, in the enterprise, I come from the tech space, and in the enterprise software world, there's something called Net Promoter Score, where you're basically, it's a survey model, and it has become kind of the holy grail.

How many people know what Net Promoter Score is? I'm just curious. Almost everybody. For the people who haven't heard it before, you ask one question, how likely Are you to recommend this product or service to another person on a scale of 1 to 10? Right. If it's under 6, it's 0 points? Yes. If it's 9 or 10, it's 1 point, and if it's 7 or 8, you get 0 points. That's right. So it's basically you give it a full endorsement, you're indifferent, or you're a detractor.

Right. But the most important aspect of that is that's been adopted as a way, even though it's after-action survey, which I generally am calling into question here given what we do, It gives you a baseline of how people are feeling, and you understand as you take steps or take action or you incent management, you know, you get— you understand what impact have you had. We're going to have the same thing in the retail space.

Awesome. Next slide. Okay, we're wrapped. Okay, so let's take the one Twitter question we have first. We've got a Twitter question. This is fascinating stuff, by the way. And where is the state of the company? How long you guys been— can we pull up the Twitter question? Yeah.

So we're— Oh, they're loading. We're a 2-year-old company. We pulled the team out of the university back in May of '12, and I went back and negotiated a deal with the university for their prior art and exclusive license. We filed about 22 patents since we've gotten launched ourselves. We spent the first year building the platform. We went commercial fall of last year, and we are now actively engaged in pilots. We have customers with our 2 partners, iMotions and Interscope, on the market research side. We're basically walking through the university and the corporate research space. That's more controlled environments. But now we're attacking what I think is the real strategic opportunity, which is the uncontrolled environments. All the people, places where people are, they're enjoying an experience, and merchants, retailers, etc., don't really understand what impact they're having. What is it? We're going to deliver that.

So where does this start? $50K? $150K if you want to fire something like this up?

You know, actually, we're— it's a very light deployment. I think in the fast food example that I'm giving you, the 2 model stores we're going in, it's gonna— the installation will be something like, you know, $2,000.

And what's the total charge you'll be giving people for this kind of

service? Yeah, this is going to be basically something that looks like You know, $300 to, you know, $400 a month.

Oh, okay. So again, the software as a service model.

Software as a service model.

Oh, the question was— let's pull that question back up. I missed it. First one. Did you see the question? Oh, here it goes.

I didn't.

So, you seem very cool, first. That's— thank you. And I can vouch. Looks can be deceiving. He's actually very cool. Can it work with user experience testing? To optimize the user interface.

Yes, actually, and we actually have, you know, both paid and unpaid pilots right now on application and device user experience. So in the one case of a fast food restaurant, they have sort of a tablet set up to order, and they're interested in, you know, how are they doing. So we're doing that work on the UI. In another case, there's a big box retailer that has a logistics application, which their employees hate. And they're trying to figure out what things do they have to do first to make it better and move it in the right direction. So those are 2 examples.

And do you find that the people doing design and UI and UX actually can correlate what they're building with the emotion of the people using it? Or are they just— I'm thinking they're guessing.

Well, they're certainly guessing initially. Because they— and again, after-action surveys don't get it done. But in an iterative process of— in this day and age— and this applies to retail as well— there's real value in proactive and interactive— this circle of design, of adapt, and then release. And then design, adapt, and release. And so whether it's in the UI place or in retail, this virtuous circle happening faster and faster and faster is a real value.

All right, let's hear it for Ken from Emotient, like quotient. We'll take a 10-minute break, and we'll be back in 10 minutes.