Jay Hack demos an AR beauty app that tracks your face in real time, overlays virtual makeup, and recommends YouTube tutorials from creators who look like you. The system classifies facial features like eyelid type and ethnicity, then surfaces matching tutorials with product links timed to the exact moment they're mentioned.
Chapters
Hi, I'm Jay. Hi, I'm Anne. And we're going to be demoing a product today, and Anne is going to help me out. So the first thing is when you hit our website, we can immediately begin to track your face as such. As you can see, I'm tracking Anne's face in real time, and as she moves, the lines will follow her face. This allows us to do things like project compositions onto your face. So this here gives a preview of what purple eyeshadow and eyelashes will look like on her face, and as she moves around, it follows her. As she's doing this, we're able to grab frames, send them to the backend, and actually perform a characterization of her face. So as you can see here on the left, we've classified that she has a double eyelid, she's female, she's not wearing glasses, she's Asian, and it also follows her head pose. So, Anne, can you look to the left real quick? And look to the right? And as you can see, the arrow actually follows where she's looking. Obviously, there's some room to grow here, but it works pretty well out of the box. Now, we can use this to be prescriptive about the types of content that we decide to show her in the beauty space. So over here, I'm going to pull up YouTube tutorials that I think are most visually similar to her face. So you can see this ranked list over here contains people who are all Asian, which matches the fact that she is in fact Asian and they actually look like her. Whereas, you know, for me as a white guy, the types of things that would pop up would be very different. Great. Good. And here's a demo. Where as we sort of roll over it, you can imagine different types of content appearing on her face. So here it might be something like, "You're going to apply foundation and stripes on your cheek, and this one is going to teach you how to do this type of eye makeup." So one last thing I'd like to show is we can also have an interactive way of watching tutorial videos. So here I'm pulling up a tutorial by the artist Desi Perkins, and one thing to note is up here in the top left, there's a GIF that shows what she's actually going to look like in the very end. So she always kind of has eyes on the prize. And we've made this through a UI that allows you to mark up the GIFs. Furthermore, the products that she's going to be talking about are presented here right in line. So if you found something you were interested in, you can imagine clicking on that and it would take you right to the point in the video in which they're talking about that product. Finally, Desi Perkins is known for being a little bit verbose, and what we can do is we can hop directly to the points in the video where she's actually talking about things that matter. So for instance, she's talking about lip lining, which is the process of drawing on your lips. And here you can see on Anne's lips, it's actually drawing where she should be drawing. And then we can be like, okay, hop to the next part, and it's gonna cover the entire top of the lips so she knows where she should be drawing, and then the bottom of the lip, et cetera. Cool. So this showcases a bunch of different sort of disparate technological components that can be woven together. In order to— oh, there we go— in order to make a more comprehensive product. And we are currently rapidly iterating on making something that solves the low-hanging fruit. Thank you, appreciate you watching.
