The Future of Beauty Product Discoverability - Jay Hack, Mira | eCommerce Tech

Hello? Can you hear me?

Hey, I can hear you. I can see you coming across great.

Awesome.

I'm really excited to pick your brain. Oh my gosh. Um, so everybody, this is Jay Hack. He is the co-founder and CEO of Mira, the personalized beauty shopping platform of the future. So only fitting to have you on the Future of Ecommerce for Beauty and Skincare. With over a decade experience building intelligent data systems, Jay is an AI and computer vision expert. Sounds like a little bit of a nerd, which is great. I'm a nerd as well. Formerly at Palantir and Stanford University. Just to start, give me the background of Mira, how you came across, why you built this. You don't strike me as a big beauty guy, but something tells me you are more today than ever before. So I'd love to hear the whole story.

Yeah, for sure. I probably know more about makeup than pretty much, you know, 95% of men you've ever met. And so, you know, it's a pleasure to be in a domain that has become a passion of mine. But, you know, we like to joke that, you know, 3 years ago, and we sort of arrived at this idea, one day I was putting on my eyeshadow and I realized there's got to be a better way, which is kind of a joke on the side there. But no, the real story is, I, you know, when I was working for this company Palantir previously, I was living in Singapore for one of the deployments we were doing. We were working on, you know, sort of making artificial intelligence for banking infrastructure. So completely unrelated, at least at surface level. And I had a friend who was from Singapore, and she had a specific eye shape known as a monolid, which is sort of a flattened eye, if you can imagine that. And I'm sure you can imagine that when you have an eye shape like that, it's very difficult to put on eyeshadow if you only see it on people who don't share your eye shape. And she shared that with me. She said, hey, I actually have a lot of difficulty finding tutorials online for people who share my eye shape actually using makeup. And to me, as somebody who's been doing computer vision for at this point like 10, 12 years, something like that, that immediately stood out to me as like, hey, that's actually a great opportunity to apply the latest technology from the computer vision world and basically scan your face with computer vision, do the same for a bunch of influencers online, and basically use that in order to match you with people who can be very informative to you. So I built—

My mind is already blown. This is— that's— this sounds amazing. I bet some people right now are thinking, I don't even think this is possible.

Yeah. No, I am pleased to inform you that it is possible. And furthermore, you should check it out on askmira.com. Or download our app. But yeah, that was sort of how the ball got rolling there. Super interesting application of some technology that we were privy to, and fortunately it produces a lot of value to, uh, you know, the consumer who's shopping for makeup.

So you're, you're scanning my face or a picture of me or something like that. You're, you're comparing a few data points. How many data points are you comparing? What are the, what are the key differentiators? What do you find are most important?

You know, we, we ingest information about you from a bunch of different domains, so computer vision is just one of them. I'm sure you can imagine also as you're shopping—

I'm losing you.

Your skin type, the skin conditions you may have, the types of products you like to buy, all those are super valuable in informing the set of things we can show you in order to optimally convert you, if you will. And so in terms of the number of data points we collect on a consumer, I struggle to give you a number because data point is a little bit ill-defined here, but what I can say is that we've put together a massive data aggregation engine that goes across the web and gathers content that's informative to shoppers along their journey. And we have, say, 3.5 million videos, 9 million reviews from 29 different retailers. Wow. We have something like 15 million different images gathered from across the web, and we've applied artificial intelligence to all of them to distill sort of the core insight that people would gain from looking at that image so that we can present them in a very intelligent manner. Just like showing my friend a video of a, you know, woman with her eye shape to giving a product tutorial, we can show you an image of somebody in your skin tone putting on a product or show you a review of somebody who claims to have your same skin type in order to, you know, get you sort of across that threshold to actually purchasing a product online.

Okay, so this is maybe a little bit— this is a tricky tangent. Um, the skin tone— what about lighting? How do you know that this tone is the right tone? Does it like— do you aggregate it across 1,000 pictures of me? You just plug— maybe I log in with Facebook and you take like the sum average of every post I've ever made with my face on it?

What—

how does this work?

It's a really good question. I think this is something a lot of people are concerned about, is like, if you're going to recommend a specific product to somebody and claim that it has their same skin tone, then you have to be absolutely correct. And that's true. And, you know, we actually don't even really attempt to do that right now. We'll make sort of suggestions within a band of skin tones and say, hey, you might want to explore these, especially if you've used this foundation before. This one seems to match up in terms of the coloration. But when it comes to, you know, purely computer vision, instead of us saying like, this is an exact match for you, we just try and sort of filter out the 95% of content that does not apply to you. And so you can imagine that, you know, if you have a darker skin tone or something like that, and we're able to say, hey, here are just women who are from your ethnicity or from, you know, a band around your skin tone, that's useful to them and they can scroll through it. Unfortunately, I think there's a lot of applications of AI where if you get it wrong, somebody dies, and beauty shopping is not one of them. If we show you an image that's slightly, you know, off from your skin tone, you can just swipe to the next one and it's still very informative.

I see, I love it. This is super powerful. I'm actually, I have an application for this because we had Donna on earlier and her and I worked together at BoxyCharm. One of the biggest struggles we had because we're trying to personalize the experience for each user, but we can't put, some people just don't wear dark eyeliner as an example, right? And so we're putting dark eyeliner in their box and they're essentially like, They're putting up on forums, they're gonna trade it, they're not gonna use it. And so I can imagine that you could use this tool to predict that. The only way we were able to do it was just by asking, right? So asking what they like, what they prefer. And so I think that there's a next generation of that. Specifically to skin tones, skin tones are so tough to match that, you know, you can't just send somebody like the right color blush or like, it's, it's, yeah, it's got a— they're gonna try like 3 or 4. And so that's why I was really interested in this one.

Yeah, totally. And, you know, I think that there's, there's 2 types of input you can take from a customer or consumer. And one of them is explicit, where you ask them, you know, what do you use? And the other is implicit. And that's, you know, maybe somebody's browsing behavior. And, you know, I would basically put it this way: if you were to sit over somebody's shoulder and watch every Instagram post they looked at, how long they looked at it, as well as every YouTube video they watched, you'd have a much better idea of what types of products they're likely to actually enjoy if you put them in their box. And that's kind of And the strength of the Mira platform is that we're able to log all of their behavior on the platform. That's the conversations they're having, the products they're looking at, the things that they end up actually clicking on, engaging in with a significant amount of time behind it. And so that allows us to really sort of like reduce uncertainty as to what content we can show them, what products we can give them, et cetera, that it's actually going to be delightful for them.

So now I'm maybe a little bit confused. So on the backend of your platform, are you like integrating an Instagram post into the platform so that you can see that view duration? 'Cause otherwise Instagram and YouTube aren't gonna give these away to you, I'm assuming.

You know, we're not actually taking any sort of user behavior off the platform, but what we are doing is a lot of that content ends up being ingested into the Mira platform, and you can view it on Mira. So I was using that just sort of as an analogy, like if you were sitting over my shoulder and watching me browse through Mira, you would also probably get a much better understanding of, you know, the types of things that I was interested in, and our AI basically leverages that information or to populate the rest of the content that you see on the Mirror platform. But yeah, just to clarify, we are not actually looking at your Instagram viewing patterns.

But that's a kind of a goal of yours, or a dream perhaps, maybe a pipe dream, but—

Yeah, I think that if people were willing to give us that information and they didn't feel like it was compromising their privacy, then we would willingly take it. But also, you have to really, especially for an AI company in 2020, you have to be very cognizant of your user's data. You have to respect their privacy, and if you do that, then they're more likely to actually be attentive on your platform and respect you as a brand.

Yeah, absolutely. The first thing is their privacy and keeping the trust. The second thing is trying to, trying to help them, guide them in the journey, and like you said, removing 95% of the content that doesn't apply to them. So, um, You, I heard the other day, as you told me, only about 11% of beauty sales happen online. This seems super low. Why do you think this is occurring?

Yeah, totally. This, I mean, this really gets at the core of the problem that we're solving at Mira. So 11%, you know, out of context sounds low, and I'll tell you, it's super low. Like fashion does about 34% of all of its sales through e-commerce channels. And, you know, furthermore, you look at this category specifically, and the consumers of them are super digitally savvy. You know, they're on Instagram browsing beauty content. And another fun fact that's sort of similar to this 11% is the number 2 category on YouTube is beauty, actually. And so, you know, the question is like, why is it the case that you have this small product that ships easily, it's got a high SKU count, uh, you know, it's got an infinite shelf life for all intents and purposes, but people's browsing behaviors are not reflective of their purchasing behaviors? And the answer is, you know, coming from a cosmetics background, and probably a lot of people here know as well, is it's just such a complicated and experiential purchase that people traditionally have a lot of difficulty basically assembling that narrative for themselves, why they should be comfortable buying this product and spending their hard-earned money purely based on their online research. And you know, it's like you have this aesthetic concern of, is this product going to make me look the way I want to look? And kind of getting back to our earlier conversation, you need to understand what it's going to look like on somebody who looks like you, or you need to see a representation of yourself applying that product. And then there's The pigmentation concern around your skin tone. There's like the qualities of a product. Is it creamy? Is it long-lasting? Is it going to be good for humid weather? Because maybe you live in Singapore. And even getting into like the chemical composition of a product, because you're literally putting a chemical on your face today. And so unfortunately, it's the case that even in 2020, the resources you have available for sort of navigating this complexity around products are located across a super fragmented domain of online informational resources. YouTube for seeing representations of people who look like you putting on the product. And good luck searching for somebody like bald white guy wearing Kat Von D tattoo liner is what I would do in order to find that. But a textual query is just not very well optimized for that type of information retrieval, right? And I think the most egregious example of people incurring a lot of friction when they're trying to learn more about products online is if you want to understand what the ingredients in a product are going to do to you, you're really literally like copying the name of the ingredient from an e-commerce website like Sephora or Ulta or whatever, and you're plugging it into the Google search bar. They're kind of hopping back and forth, and it's a relatively simple insight, like, is this product going to cause me to break out, or is this going to be bad for my skin? But the amount of work you have to go through in order to actually, you know, get to that sort of result that you're convinced that it is or is not good for you is significantly higher than it needs to be. And so basically, you know, this, this 2-hour-long process we call attack of the tabs, where you have to go to 50 different websites in order to do this research and price comparison, et cetera, is what's really led to this epically low e-commerce engagement in the beauty industry. And so that is the sort of massive opportunity we're going after, is to eliminate that friction, use artificial intelligence to really sort of facilitate that journey towards purchase and get you to the point where you're actually confident making purchasing decisions in 2 minutes in the palm of your hand, as opposed to 2 hours.

This is really interesting. So Mira is really, you know, as you maybe reach a market maturity as a new platform in this space, Your goal is actually to take on offline retail and bring it online, specifically in beauty. So you're, you're actually trying to create your own trend of moving offline to online.

Yeah. I think if you look at, you know, other places we would consider like leaders in digital technology, like China, you see 20% of e-commerce happening online in China. And so to us, it's really not a question of if, but when. And we see ourselves as enablers of that trend. And, you know, also beneficiaries from it. If we are the place that people come to shop for beauty online, then there's a massive amount of value to be extracted from that sort of the wheels of commerce spinning.

Yeah, I love it so much. So tell me about how maybe brands today, maybe they have a local store or they're in a lot of big box retailers. How are we— how can we best maybe transcend this online-offline dichotomy How can we use your tool for online search trends to benefit offline? How can we use data from the offline world to benefit online? All of that, like, it seems really complicated.

Yeah, I mean, it is super complicated, and it's unfortunately the case that today there aren't great tools for brands to really figure out what people's sort of behaviors around their products are. And this is one of the early opportunities we identified. So, you know, a great example that I like to refer to is that If you sell your products in a Sephora or an Ulta, they're not sharing information back with you, the brand, and so you don't even know the average basket size, let alone what other products do my consumers tend to shop for this season and whatnot. And beyond that, you don't have any demographic information around them, and so it's really difficult to make good decisions when you don't have data or to refine your marketing narrative to figure out what types of products you should be building and selling and whatnot. What's your best option in terms of— well, the solution that we provide to people is basically we can provide you very comprehensive analytics that describe those consumer behaviors around your product in such a way that's very actionable. So you can, like I mentioned, refine your narrative, find better people to be marketing your product to in the first place, better types of formulations and ingredients to actually be putting into your product. And then that's obviously going to play out well both online and offline. If you have a better way of phrasing your value proposition to the customer, it's going to drive higher conversion. Conversion. But I think as of today, what I would suggest is the best solution is, if you have a Shopify landing page, the extent of your analytics are going to be what people say in the comments on your Instagram posts, what people, you see on Google Analytics as people come to your Shopify landing page, what percentage of them are converting, what assets end up actually increasing that conversion and whatnot. And fortunately, there's a much better way, like using data analytics and data science and machine learning, you can actually figure out what things should I be doing in order to optimize that funnel. And fortunately, we are the providers of said solutions. Yeah, that's amazing.

Is there any way as a beauty brand right now to use vision, computer vision, if just for my business, kind of separate from using you specifically, but maybe I want to use that here?

Yeah, sure. There's a ton of like white label solutions you can use for virtual try-on. Two that come to mind are YouCam, it's owned by PerfectCorp. They have, you know, they're actually in Sephora right now. And there's one called Revive, based out of Finland. And both of them will essentially, like, use computer vision to analyze your face, do so-called virtual try-on where they'll project makeup on your face. They will understand what skin conditions you may have. And it's actually a relatively mature technology. I mean, I think we can anticipate the next 2 to 3 years here, we'll have something that no longer looks obviously fake or gimmicky. Uh, it's very expensive. You know, I, I don't want to quote numbers on their behalf, but I think they end up charging hundreds of thousands of dollars potentially to larger brands and retailers in order to have this technology on their website. And I think also I heard a really interesting thing the other day, which is this doesn't actually increase conversion based on the value proposition of showing you what the makeup will look like, but rather Sephora saw— this is hilarious— Sephora saw an increase in sales when they had AR in store because the boyfriends and husbands of women who would visit the store ended up spending a lot of time playing with the tablets that had this virtual try-on. And so they spent more time in the store and women had more time to shop.

Interesting.

Yeah. And so I would say that the technology today is like, it's getting there. It hasn't been demonstrated to actually have the value proposition that people expect, which is eliminating uncertainty for them about what shade is going to be right or what a product will look like on them. Um, but you know, maybe if you give it 2 years or so, we'll get to that point.

And you said YouCam, and then what was the other one?

Revive, R-E-V-I-E-V-E, and they're focused a little bit more on skin analysis and less so on virtual try-on, although they might have a solution. There's actually, you know, there's sort of like a boom of activity around augmented reality. I want to say about 2 years ago when ModiFace, which was the leader at the time, they were the main provider for Sephora's in-store sort of kiosks for virtual try-on, they got bought by L'Oréal for an undisclosed amount. and then simultaneously Ulta bought one based here in San Francisco called Glam Street, and Shiseido bought one called, uh, what was it? JARAN, G-I-A-R-A-N. And all three, you know, these sort of massive conglomerates put their hands into an AR solution. They decided we're going to, you know, use this for our own, um, sort of internal, you know, technology stack. We have yet to see anything cool come out of it, although if you give it another 2 years, we might. Yeah.

I like that timeline. And this, this is, uh, it's, I think it's a good timeline. It's really good for the consumer. And I think it's important that we know that this is about to hit us because like you said, it, we, I think we've, we've heard about augmented reality for 10+ years and we've kind of been waiting for, for something to take effect. And so you're saying it hasn't been effective yet, but it's about to be. So I think we need to make that move today. Or maybe, you know, at the beginning of 2021 to make sure that we are on the successful side of the trend. And so we've been waiting. Now might be the time to take action in creating augmented reality tools. I actually did that at BoxyCharm. I launched the first augmented reality app. We were able to take somebody's— all of the products in a beauty box and get it onto their face using augmented reality. But like you said, it wasn't that successful. Because it's hard to get exactly right, especially with different faces and skin tones and matching. It is really, it's quite a lot of work. So perhaps with some more dynamic.

Yeah, and on that topic, I think what we see is that it's useful for sort of top of funnel discovery and exploration. So if you show somebody, here's a couple of different shades, find one that you're interested in, and then here are options that closely match up against that, that actually ends up driving a lot of momentum behind a shopper. But if you're trying to claim to somebody while simultaneously trying to sell them a product, that this is what it's gonna look like on your face, but I've completely fabricated what the shade looks like on your face, then people don't really have a good reason to trust what you're saying. And so as a result, that ends up driving very low engagement. I see. And it makes sense.

So there's a risk, it could backfire, is what you're saying. So be careful, do it, do it well, and I think that as Donna said in the opening keynote, you know, be authentic to your brand, but don't say that's not on brand. This is something you're gonna have to dabble in and try. I think it'll become mandatory the same way I believe SMS marketing is now mandatory, especially in beauty and any other time we need people to see the product or feel the product in other cases. I can't wait for tactile augmented reality, but that's probably not 2 years out.

Good luck.

It'll be there, since we all live in the Matrix anyways. No, all right, anyways.

Good point.

Speaking of AI, so outside of this example of AI, how might businesses look to use machine learning and AI for their e-commerce store today? Are there any other examples?

Yeah, for sure. I mean, I think you have to think about AI in terms of What levers are you pulling and what outcome do you want to produce? And so, you know, let's say you're a regular Shopify store or something like that and you're trying to get somebody to convert on your product. Well, levers you have to pull are like, what content are people seeing on that sort of journey towards clicking the buy button? That's going to be your branded asset at the top. That's going to be the copy that they end up seeing. That's going to be the other recommended products that they see at the bottom of the screen. And really what we're trying to do in the sort of AI problem setup or the AI context is optimize your conversion there and basically swap out those assets or the sort of things that somebody sees in order to get them to maximally convert. And so, unfortunately, you're in a pretty low data setup when you're just a Shopify website. All you're seeing is maybe the IP address of the person who's coming in, which tells you the location they're coming from. Potentially, you might also have a cookie that describes the other websites that they've previously visited, Maybe you have a Facebook pixel that'll tell you a little bit of demographic information about them. And so the question is, how do you basically select the optimal asset based upon this small amount of information you have? I'll give you an example. It's maybe not like AI, but it is intelligent inference. And AI is sort of like an overloaded buzzwordy term at this point, but intelligent inference is what it is. Where if you see somebody coming from Singapore, for instance, I think it's pretty reasonable to assume that they are going to have a different demographic breakdown than somebody coming from South Carolina. And so as a result, what you might want to do is actually show them a different asset of somebody putting on that makeup compared to the person from South Carolina. It takes a lot of infrastructure in order to implement a decision like that for your website in order to convert them, but it makes a lot of sense at the end of the day. And this ends up applying in retention marketing as well, where if somebody buys your product and they leave a review that has some natural language in it, says like, "Hey, I thought this was really great for my monolids," for example, which is going back to this example of an eye. Well then, if you can detect that from their review and send them in your attention marketing like, "Try these top 5 products from my brand for monolids," well then you've essentially latched onto some personalized information about them that allows you to refine your narrative and be more specific and actually speak to the things that they care about. You might say that's AI. I always suggest to people like, hey, Before you do AI, just implement very baseline personalization using basic decisions, decision trees and stuff like that. And then when you're trying to get a little bit more fancy, add AI on top of it. But AI usually adds another 5-10% accuracy after you get that basic infrastructure set up.

Oh, interesting. So a lot of it can be done. I think what you're talking about here is personalization, which I think is another kind of mandatory component of, at least at a certain level of scale, I'd say maybe at 10,000 visitors a month, it starts to become relevant, and at 100,000 visitors a month, it's basically a game changer for a business where you can look at whether it's regional personalization, weather personalization, and then product based on purchase behavior and a bunch of other things. If you have a static website today, 2 years from now, I don't think, you know, you will have I don't think homepages will be static. And I think that Shopify will actually design some sort of dynamic content creation tool and it'll just pop up like 5 demographics that it pulled from its own data and said, you have these common demographics, create a custom homepage for each of them or something like that. I think this will be the future of e-commerce and it'll be really powerful in beauty when we're, we're, you know, seeing people that look like us tend to convert better. It is the way that the human kind of mind works. Um, all right, Jay, um, we've got to run. Tell people where they can get ahold of you or, or what to do. What, what should they do? They should sign up at Ask Mira. These are all beauty brands. How are they going to— they get on the back end of your platform? What do they do?

Yeah, absolutely. If you're interested in collaborating with us, we've actually just rolled out a Shopify integration so you can start selling your products on our channel. That's 100,000 users on our app. Uh, so check us out on the App Store as well. It's called Mira Beauty on iOS. And then our website is askmira.com. And if you're interested in collaborating with us and doing something like an exclusive launch or, you know, some type of discount for segments of our consumer base who are specifically likely to be high LTV users of your brand, then go ahead and contact me at jay@miraai.com. I'd love to chat.

I love it. I'm going to share that with everyone here. I'm putting it in our show notes as well. I think in a couple years' time, or maybe it'll be next year, we'll have this conversation again. And you'll say our 5 million users on our platform and the thousands of beauty brands that we're working with. I think, I think you're about to hit a serious S-curve acceleration here.

Thank you so much.

All right, thanks Jay for hopping on and explaining the, uh, the, the future of product discovery and the future of AI and in beauty. I, I love it.

Real pleasure. Thank you so much for having me.