DYSCO 0.1.0 - AR/CV for Cosmetics Discovery and Exploration

My presentation is about using augmented reality and real-time computer vision to aid cosmetics discovery and exploration. Instead of providing a single narrative of how a user might interact with the product, I'll instead showcase a set of components that can be plugged into one another and used to rapidly prototype a product. So starting off, come to this website, and we're immediately able to start tracking your face. So as you can see, I have a model of my face, and it's following me in real time. Using this, we can project makeup or arbitrary things onto an individual's face in real time. So as you can see, I have this composition and I'm able to have it follow my face. In order to put these things together, I have a UI that allows you to go ahead and draw an arbitrary shape that you'd like to project onto somebody's face. So for instance, if you're a makeup tutorial artist and you want to tell somebody where they are going to have to apply a certain product, you might want to go through and do something like this, where we add in a nose corner as such, modify the color as such, save this, and then going back to the original web app, we can then project this. In real time onto the face as such. Cool. So this is sort of the augmented reality component. It shows you how you can take something and put it onto your face, and this might correspond to like, oh yes, put the product here, or these would be blinking, highlight a region of your face. Now another aspect of this is we can use computer vision in order to make intelligent product recommendations. So for instance, here I've pulled up a bunch of people who are YouTube artists. Who the algorithm thinks look like my face in terms of facial features, skin tone, etc. So the top hit here is Jeffree Star, who is actually one of the only men in the dataset of YouTube artists— excuse me, YouTube artists— that I pulled from YouTube, which is quite impressive. You can imagine doing something like this also for, you know, specific products, and as you roll over them, the thing that ends up showing on your face is like what that shade of lipstick might look like, etc. And right now I'm just doing it with tutorial artists because it's been shown that 50% of heavy buyers— this is a specific category of consumer in this domain— actually end up researching through blogs or some sort of social media first. So if you can recommend them content that's relevant, then that's highly informative. So finally, I think there's a lot of opportunities to make makeup tutorials specifically a lot more interactive and fun. And to that end, I've put together an analytics module. That over here you can see is able to real-time perform classification on my face as well as a couple of other tasks and figure out that yes, I do have a double eyelid, I'm male, I'm not wearing glasses, I'm white, and furthermore my head pose, which might correspond to something like where my attention is on the screen. So as somebody who's a content producer, this is a useful source of analytics for you because this can tell you what type of viewership you're getting and where people are actually paying attention on the screen, how engaged they are, etc. And from my perspective, if I'm actually trying to recommend content, then this type of information can be very useful because it can allow me to route you towards the correct endpoint. So whether that be, you know, okay, you're a white male, you probably should not be putting on Korean makeup. That's something that would be quite informative. Great. I hope you've enjoyed the demo. This is currently in development. It's been about 2 months, and I will continue adding features and updating as they come along. Hope you enjoyed.