GIPHY's celebrity detector

Bandwidth for Changelog is provided by Fastly. Learn more@fastly.com we move fast and fix things here at Changelog because of Rollbar. Check them out@rollbar.com and we're hosted on Linode Cloud Servers. Head to linode.com changelog this episode is brought to you by DigitalOcean, the simplest cloud platform out there and we're excited to share. They now offer dedicated virtual droplets and unlike standard droplets which use shared virtual CPU threads, their two performance plans general purp CPU optimized They have dedicated virtual CPU threads. This translates to higher performance and increased consistency during CPU intensive processes. So if you have build boxes, cicd, video encoding, machine learning, ad serving, game servers, databases, batch processing, data mining, application servers or active front end web servers that need to be full duty CPU all day every day, then check out Digital Ocean's dedicated virtual CPU droplets. Pricing is very competitive starting at 40 bucks a month. Learn more and get started for free with a 100 of credit at.co changelog again.co changelog. Welcome to Practical AI, a weekly podcast about making artificial intelligence practical, productive and accessible to everyone. This is where conversations around AI, machine learning and data science happen. Join the community and slack with us around various topics of the show@changelog.com community. Follow us on Twitter. We're @practicalai FM. And now onto the show.

This is Daniel Whitenack, one of the co hosts of Practical AI. I'm a data scientist with SIL International. I'm joined today by my co host Chris, who is a Chief AI Strategist with Lockheed Martin RMS APA Innovations. How you doing Chris?

Doing great. How's it going today?

Daniel? It's going great. We were just talking before the show. I'm dealing with a few allergy issues. I think you are too, but it's that time of year, I guess it is.

So a couple of coughs here and there is a normal part of a podcast as far as I'm concerned.

Yep. Well, are you up to date on your pop culture, Chris?

I'm trying. I'm looking forward to learning a whole lot more on this episode.

So my wife and I have been listening or watching Jeopardy. Now that it's on Netflix during our dinners recently. And I I fail often at the pop culture categories, so I'm really glad that we have Nick Hasty from GIPHY with us today who is an expert in such areas but also an expert in AI, to talk about some of the work they're doing in AI at GIPHY and where that crosses with pop Culture and other things. So welcome. Nick.

Hi. Thanks for having me. It's great to be here. Yes, extensive culture knowledge is a prerequisite here at giphy.

Yeah, I assume so. So hopefully I can. I can do a little bit better at Jeopardy. After. After the show. Yeah. So Nick is the director of research and development at giphy. Nick, do you want to just give us a little bit of background about yourself, how you got into tech, how you got into AI, and how you ended up at giphy?

Yeah, for sure. I'd be happy to get kind of a varied background, to be honest. So currently director of R and D here at gip. We're in New York, New York, and I live in Brooklyn, so I grew up in Georgia, actually, and.

Ah, that's. That's where I'm at.

Okay, good for you.

Georgia rocks.

Canton, Georgia. It's like, about an hour and a half north of Atlanta.

I'm in Kennesaw. I know.

No kidding.

And I grew up in Cobb county, so for those who don't know that the Atlanta metro area. I'm sorry, but, you know, we're practically neighbors growing up.

Yeah, totally. Yeah.

I feel a little left out here, but I'll let you guys continue.

I'm so sorry.

Yeah, that's really funny. Small world. Um, Elena's doing good. So then, yeah, I went to University of Georgia as undergrad, and I was actually an English major and always dabbled in computing in terms of, like, custom doom mods and, like, music technology stuff. I was big into music, and I actually didn't really get into, you know, web stuff and more computer science until grad school when I went to a interactive telecommunications program at ITP called ITP at nyu. It's kind of a art technology school in the Tisch School of the Arts. And they kind of take a bunch of random people who are good at a lot of different stuff and throw technology at them. Anything from algorithms and, you know, visualizations, data data vis stuff to, like, actual light, electrical engineering. Just kind of see where you end up. So I was, like, historically very into, like, creative tech stuff and even, like, art proper. So after itp, I started working for the new museum here in New York. They have a new media art branch called Rhizome, and it's been around since the mid-90s started off listserv, and it's now a website and museum and an archive and stuff where they house digital art. It's like an Internet art organization. And I was a director of technology there for a number of years. And I got a lot of my. My web chops there. And being at Rhizome, we were very invested in the GIF early on as. Because it was. I mean, the GIF is like a purely Internet media, I guess you could say. It's like something that was really born. Well, I mean, it predates the Internet, but I mean, it really came to its own.

Yeah. Once upon a time, there were only GIFs and JPEGs on the Internet.

You know, images. Yeah, like old.

Well, it's also. It's like a primary communication technique now almost native to the Internet.

It's a really wild kind of trajectory that the GIF has had. It's been really interesting to see. So, yeah, when I was a rhizome and we would do like, art shows where like, the artists would only work in the GIF format. This is like the early two, not early 2000. This is like. Well, they did stuff in the art shows like that since the 90s and the 2000s. And I started in the late 2000s there. And I've worked hard on their digital archive. So there's a lot of archive digital assets, including animated GIFs and those kinds of things. And after that, I started working. I was kind of like teaching myself Node and I was building a gift maker tool. And through school, I was introduced to. Through a schoolmate, I was introduced to Alex Chung, Giphy's founder. He was at Beta Works and he had just been pulled in as like, they were doing like a seed program for a number of different startup people. Alex is kind of like a serial entrepreneur. He's been in the game for many, many, many years and just has tremendous technical knowledge and just like business expertise. And he was looking for an engineer and to build a team. And we met and talked and just really hit it off. And so then started working at Giphy the next day. And this was Giphy's first week. So I was technically giphy's first employee for a while. There were two founders. A guy named Jace Cook was the other founder, and then Alex. And then I was working building the website and Alex was building the API. And we were just kind of cranking for a while and the team continued to grow. So, yeah, I mean, we definitely. Giphy kind of took off like a rocket. It was very interesting. I think it was a really cool way that it worked out. It was like the time and the place, like you were just saying, Daniel, the gif, it was coming into its own, I think with bandwidth increases and, you know, it wasn't take like 10 minutes for a gift to load, you know, in Tumblr. Dumping GIFs everywhere. There was a whole way, I think on Tumblr people really started kind of communicating with GIFs and using GIFs as reactions and using gifs as a communications medium. You got like pop culture and all these kind of different things you can load into a GIF and you know, so like if a celebrity is saying hello, you know, you got all these kind of different layers of meaning that you can unpack. And it became kind of a, you know, really interesting way for which people to interact with each other online. And we kind of realize that. And our goal was to just, you know, since GIFs are so old and since they kind of work everywhere, I mean, at this time it was 2013 so like there was still like contention around which video codec to use on the web. And you know, somebody supported, you know, Firefox supported this, but Safari supported that. But like no matter what, the GIF was supported everywhere. It was definitely an old format and you know, overweight and kind of slow. But it's kind of universal ability to just load and play in anything, including like imessage way, way back in the day.

So I guess at this point, with you guys being going for quite a few years at this point, what does GIPHY research do now? Now that you've kind of grown into a full organization, what are the typical types of things that your researchers are engaged in?

Yeah, so fast forward now. I've kind of moved out of engineering and I.

More.

I guess I'd say more. I'm like a product director person, so I run the R and D team. We are within the search division. So our earlier.

How big is GIPHY in general? Just to get a sense.

Yeah, it's a good question. There's about 100 people. There's roughly 75 in New York. In L. A. We have a, have our own content creation studio. So there's like artists, animators out there. There's about 25 of those. Engineering wise, we're probably 30 to 40 at this point.

Yeah, that's a good size organization. You're definitely past the, the early startup phase.

Oh yeah, totally. It's been a wild ride to see that happen. There's a lot of people here now and it's awesome to have a lot of super talented people here taking care of the things that, you know, just like they were just you could never deal with in the past. And just now we're just so much more mature and so much more able to handle everything that comes at us. So you know, like I was saying, my background is more like, I like making things. And so I have been able to move into the more of the R D, like special projects kind of research things, find fun stuff, you know, create prototypes for things and then other teams maybe, you know, find value in it and pick it up. But I mean, to be honest, our team, we're a good sized company, but we're not so big that we can just sit in the back and like mix chemicals and like play with our thumbs. Like we got to ship stuff. So maybe one day. Maybe one day. But yeah, so, so my team, alongside the celebrity sector stuff, I mean we've done, we were the kind of the first team to really use deep learning tools like Word2Vec. We use this open source model, we use that deep learning model like all over the place for recommendations and a couple of other things. So like on the GIPHY website, if you go to the GIF detail page, which is like a single GIF, all the related GIFs down below, like our team kind of overhauled that and used word2vec to take user data to approximate relatedness in terms of how visitors visited the GIPHY website in a session. We did a similar thing with our tag space. So like we do like related tags and recommended searches using deep learning Word specifically. We are doing other kind of things now where we're building trending tools using various types of models. Not all deep learning, some just machine learning or some of them just kind of old school statistical algorithms doing like language prediction stuff. We're doing head to head comparisons against search results so that we can kind of get a better sense of how our search is performing. We had a spell of time in which we were doing a lot of image annotation. Obviously GIPH has a whole lot of gifs and we relied on our content team to annotate those for a long time. We have a team of people who are ultra mega pop culture wizards who can prattle off anybody in a gif and then all the slabs know everything. But humans can only go so far. When you reach the tens of millions in terms of your catalog, it strikes.

Me as funny you're the company that can have a pop culture wizard department.

Oh yeah, totally. I mean we've gotta, I mean it's, we could call it that. Our content development team is definitely, you know, people like people whose jobs are to watch TV shows and cut gifts out of them or live gif. That's not the only thing they do a lot of stuff. But like we'll watch the live event, like they'll watch the Oscars and they will cut gifts of the Oscars in real time and catch all those awesome moments of people sassy looks or you know, celebratory speeches and those kinds of things. And we'll cut those in real time. And we work with partners, movie studios, these kinds of, you know, various other people in the entertainment industry to live GIF events and things like that. So yeah, it's very, it's very funny.

So how many, I mean you mentioned a lot of different things that you've done and are doing. Like how many people are on this R and D team working with you to come up with these projects?

Yeah, so right now there's five of us. We've grown a little bit. So we work with a team in the Ukraine called Rails Reactor. And so I have a Dmitri Wojtek Ihor Krush are working with us right now. We've had a couple of other employees from them, but they are really amazing machine learning engineers and kind of the specialists in terms of like the algorithms and building out the systems and stuff like that. So we work with them. And then giphy CTO Anthony has, Anthony Johnson works on our team a lot too. He has a pretty extensive background in machine learning and deep learning. And so right now we're tag teaming the team. It's been really awesome. We're like really cranking out some stuff.

So is it almost like having between the two organizations working together, kind of almost having a sub team of deep learning folks inside your R and D organization then?

Exactly, yeah, totally. It's like, you know, like I said, I've kind of moved into more product and vision and strategy and communication. I mean, I still will open up a jupyter notebook and kind of like work on the data a little bit and kind of figure out like, you know, things around training data and sources and making decisions. But in terms of like the actual implementation, I don't do that as much anymore. Just, it's cool.

So with it being like, I mean, obviously the things that you're building are filtering pretty quickly into the product and you're moving quick and you have have a smaller team. Do you feel like that? I mean you are R and D, but R and D in some organizations, like things take forever to get from R and D to product. Do you feel like that has that's influenced how you approach AI development in one way or another?

Yeah, I think, you know, R and D initially was like me trying to end up in a really, really removed, kind of fun playland. I'm kind of hoping for that and thinking if I put the name first, then the reality will follow. But the truth, you can hope. Yeah. And not that I don't enjoy shipping, but, you know, with the velocity of giphy would be cool. I was kind of hoping to like slow down and kind of like play a little bit, but truth be told, I mean, the work we did was like had a lot of value. It was very obvious that it needed to be integrated into the company itself. So we ship and it's a goal of ours. And I think a lot of companies say, oh, we're going to spin up an R and D team. And then, you know, the team disappears and nothing ever shows up and they're like, whoa, what are we throwing money into? You know, that can be kind of a gotcha. So we work hard to make sure that, you know, we're doing cool stuff and we're reading papers and like, we're like looking at the latest stuff, but we're also then taking what's out there and applying it directly to giphy so that there is real value that can be proven and integrated. You know, I think my team's probably taken over four or five of the GIPHY API endpoints. We don't do like the actual search algorithm, but we have provided a lot of metadata for the search that gets fed up into the search engine. But I mean like the Slack integration, we took that over and a while ago and we redid it over last summer. That's one of giphy's probably most well known integrations is, you know, forward slash Giphy and then whatever. And then you get kind of a random gif. Historically it was very random and we really like the juxtaposition of the weirdness, you know, just kind of like the funniness. Like it. I don't know, I mean, I think it.

Yeah, I remember those days very well.

Yeah, our philosophy is work is boring, you probably don't want to be there. So maybe if you can just throw a little weirdness into the mix, you know, it can make things a little funner. But. Yeah, but since we've, we have taken over that and did, you know, applied some learning techniques to that to make it a little smarter, make it a little more on point and measured engagement, you know, it's been like 20 to 30% increases overall in terms of how people have used the integration itself. So like Slack and that endpoint, which is our translate endpoint. Our related GIFs. Endpoint. Related tag searches. Endpoint. Yeah. So point being, like the things that we've done, we've been able to prove value in that and then those things have been able to incorporated by the company as a, as a whole.

So the AI stuff that you're doing, do you typically have like a target value add like product wise when you're going into that, or is it. Do some things just start as like, oh, I wonder if we could do that with gif.

That's a good question. A little bit of both, you know, like being with the company for so long and not having the immediate constraints of having to ship very specific things. I kind of wanted to explore the periphery of giphy's products and offerings that kind of been more lackluster or kind of unexplored or whatever. Like, you know, our tag space, which is something that like we know had really been maintained and it was very flat and it was just kind of no one knew what to do with it. And then, you know, here comes Word2Vec. Let's see what happens when we throw the tags in Word2VEC and play around like, oh, wow, this is really cool. We can see interesting relationships between our tags that we never knew existed. Hey, these are actually better than our current implementation, you know, of how we do related tags. Let's keep iterating on this product and then it becomes like an actual replaces the old method in which we were to do that. So it's kind of, that's been the approach, I mean, for the celeb detector itself though, that was something that we've always had a need for tools that can annotate images. Like I was saying earlier, due to like, you know, just general human constraints around our content team as obvious, you know, entertainment celebrities, that's our bread and butter. So if we have a tool that can annotate these GIFs as they come in through uploads and crawls and that kind of stuff and help us surface those things, that's a value add. But that being said, it was always something we wanted to open source and it was always something we wanted to do to kind of demonstrate where giphy is and what we're doing and kind of the different intersections we exist at. Because I mean, people, I think given giphy's kind of distributed nature, like, oh, I know you're from Slack, or oh, I see you in Facebook messenger or Instagram messenger or oh, I use stickers in Instagram or whatever, people don't always know like what it is, is kind of this amorphous thing. So this was kind of a flag in a way to say, hey, you know, this is kind of we're doing these things, we're using these technologies, these are the things we're thinking about and this is the data that we have to deal with. So like the list of the celebrities that we trained them, those are taken from our top 50,000 celebs that people search for. So these are the celebrities that people actually care about. Like there are celeb detection systems out there. I mean, you know, I think Microsoft has one and that you can pay for. There's a couple of third party hosts. But those services, while they may have a large number of classes in which they can provide, they didn't always hit the ones that we needed because we tended to just be more on the cutting edge in terms of like what's going on in popular culture and celebrity culture. So, you know, the thesis was we can really demonstrate that we can do the tech. We can build something interesting and fun. We have a very unique set of data that is not necessarily handled by other people. So we can kind of, you know, put all our different ingredients into this like stew and then give it to people as a way to, if you're new to deep learning or if you're new to these kinds of things, we can open this up for people. And GIFs are fun and easy. I mean, that's a, it's a nice entry point. Like if people are slightly intimidated by machine learning or deep learning or any of these technologies, you know, gifs and celebrities is kind of like a nice spoonful of sugar to help people kind of dive in. I mean, I've always personally found that when I'm working with fun content or working on a fun project, I tend to learn the technology better that way as, as opposed to just reading a book or, you know, hammering stuff out in class. I'm like, I want to build something and I want to incorporate this fun thing and this fun thing and you know, if I end up needing like a, some deep learning algorithm or whatever to make it happen, that's cool. Yeah, that was kind of the thought behind that, that.

This episode is brought to you by Discover Bot. Learn everything there is to know about bots at Discover Bot. Practical AI. Discover Bot was built by Amazon Registry Services as an online community for bot creators and makers of all skill levels to learn from one another, to share stories. And they regularly publish guides and resources to answer questions like how to set up payments to Your bot. How to stop shopping cart abandonment. What KPIs are worth measuring. How to write an engaging chatbot dialogue. You can even register bot domains there. Learn more and explore this huge library of bot resources at. Discover Bot practicalai. Again, Discover Bot practicalai.

So, Nick, to take you back, we kind of leapt right into the. The full company. But I wanted to ask you a question about you. You described when you had come into the company and you know, you were having that conversation and the next day you were doing it. But even possibly before that, as you know it, how. How did the project itself come about? What was its genesis? And also, why were celebrities the thing that you moved to? Or was there anything before that?

The project? You mean the celebrity tutor or giphy itself?

Maybe describe both of them. Maybe kind of discriminate between the two and where they each were.

Yeah, totally. So for giphy itself. So Alex Chung, who's the founder, he built, he's just a perennial creator, like, dude who just, you know, just makes stuff all the time and has worked in the Internet industry for a long time. And he kind of. He caught on the GIF thing and he built a GIF search engine that was just a website that he kind of built on his own. And he launched it and it would immediately. He got some traction. I think it was like a press article and crashed and burned and because of it's just the influx of traffic. And so Beta Works, which is a New York City incubator, slash, you know, venture capital firm, they knew Alex and they were introduced to him. And Alex is like, I got this thing building. And, you know, they're a cool company. They are very aware of culture and art and technology and like, kind of got what Alex is up to.

And they're like, listen, our episode with Hugging Face, Clem mentioned. Oh yeah, mentioned them as well.

Okay, right on. Yeah, totally. So they understood what Alex is trying to do. And so they were like, they had a hackers in residence program and they brought Alex on and his to build giphy. And then, you know, Alex, within the first week of Alex working there, I was introduced. I was doing contract work and I had been working on a gift stuff. I had been working on gift stuff already at my previous job at Rhizome, and I was building a gift creator because at the time in 2013, there was no good way to really make gifts on the web. And I met Alex through mutual friend and we just like really hit it off. And I had already kind of been, you know, it was very evident that we would probably do good together. And so I just started working the next day. So that was 2000 2013. Flash forward six years. I'm doing the R and D stuff and you know, the impetus is like, you know, giphy, we do a lot of different things. People know us in different ways. But let's really flex and demonstrate how the cool tech we're doing. I think people may have an idea of like the volume and the gifts and we serve and like kind of the scale at which we do. I mean we serve a couple billion gifts every day. But I don't know if anyone people know kind of like about the more interesting kind of projects that happen in house and how working a gift via working at like working with interesting technology at the nexus of culture and entertainment. So the sled detector kind of embodied this. We could do this project, say hey, you know giphy, we do deep learning, we're working. We've had this experiment kind of happening in the background. We use this tool in house to annotate our GIFs. But you know, the real goal was to like put it out there for people, hopefully get some press. Hopefully people are interested in it and also be excited about the community itself. I mean like any startup, you know, like we use tons and tons of open source projects and giphy would not exist categorically if it wasn't for, you know, awesome open source projects. So our success, we want to give some of it back and be able to demonstrate some of the things we've picked up and learned along the way and give something that, you know, people could play with. So yeah, that was where it came.

Cool. Yeah. So I mean that that describes a lot of the motivation. I'm curious you've talked about how the celeb project came about. I'm wondering like you talked about previously using like Word2Vec and other things for related gifs and all of that in particular on the celeb detector project, as you moved into that space, what were the kind of different things that you needed to learn tech wise to be able to accomplish that that you weren't doing before in your AI work?

Yeah, that's a good question. I mean we hadn't done a lot of convnet stuff. A lot of the image recognition, object detection stuff. We had used a lot of third parties for like hit API. Like you're not going to build if you need a generalized model for object detection and images, like not necessarily going to train your own for that. So we didn't necessarily need that in house. But for the celebrity detection one, yeah, it was, this was like our first reality, extensive deep learning project. So there was a lot of experimentation in the background. I mean first, you know, it was like just getting the data together against our searches. Like we had a lot of images around some of these celebrities, some of them we didn't. So we had to like scour the web a little bit to build the image data set and then making sure we had, you know, and this is probably like any real machine learning or deep learning project but you know, making sure the labels were good. Like I had mentioned earlier, you know, giphy's tag system was, has been going for a while and was kind of big and difficult to carry, are difficult to manage in some ways. And so we used a figure eight, used to be crop flower. We used them to help us make sure that the labels that we had for the celebrities was really good. And then I think since it is face detection, we did use previous existing models. So just finding the right one that we wanted to kind of do the, to build ours up ended up being the Resnet 50. I think that was all that was pretty standardized. I think a lot of it kind of got a little more hairy when it became things like clustering and finding out how to group our faces together. So when our team used the center loss approach for face recognition, that was like a real. In the blog post I linked to the paper, when we used that approach that kind of really the results really dramatically increased. That was a big gateway.

So when you're saying, I know in the blog post, which we'll link in the show notes by the way, so make sure and check that out. But you mentioned kind of the face vectors, face detection and then this clustering bit which you just mentioned around using center loss. So could you describe a little bit for those that aren't familiar, what do you mean when you say these spaces are kind of encoded or vectorized and then we're doing clustering with center loss? What, what does that process really mean? What, what are you doing when you're doing that?

Yeah, so. And again, I'm more of the product person at the point so I'm going to try to like give my high level. I wasn't into weeds as much on that.

Yeah, no worries, no worries.

But yeah, when you have a face coming in to the network, it you make an embedding of it so that it becomes like a series of numbers or whatever.

Yeah, so it's like face to numbers.

Yeah, exactly. And so then for all the faces that have a numerical representation. And what you want to do is be able to find similar faces and be able to match them. So if a new face comes in, when you translate that face into a group of numbers, you want to be able to check it against the previously existing groups of numbers that kind of live within the model to come up with a prediction. So for us, you know, there are pre existing models, like I was saying, like the resnet that can take an image and make it into a vector. And so that wasn't so hard. But finding the right way to take the vectors and cluster them and group them in such a way that when a new face vector came in, we could make an accurate prediction, that was definitely the biggest challenge for us.

So you mentioned that you had selected ResNet50. What kind of tools were using was the team using to be able to do it? Was this TensorFlow or PyTorch or MXnet? Do you have any sense?

Yeah, totally. So we went with Pytorch, we played with TensorFlow a little bit, but Pytorch was great. I mean, we've kind of fell in love with it. I guess it's very Pythonic to say TensorFlow and Python has its own kind of idioms and its own kind of ways in which it wants to be used. But Pytorch felt it was kind of easier to. It's just more malleable, we would say. And it allowed us to iterate faster and allowed us to kind of experiment quicker. So yeah, we really fell in love with that. And that's kind of become our go to. And the code that we released is all in Pytorch.

Cool. Yeah. So I'm kind of wondering, as I'm thinking like, like you're building this face detector bit which vectorizes the faces, you've got this clustering bit. And I'm just imagining pop culture in general, there's so much of. It's so dynamic, right? There's going to be new celebrities that are super popular in a super short period of time. So have you thought about how that influences how you kind of manage and deploy the model in the sense of as new celebrities come onto the scene, how you're updating it over time, how you're doing swapping out models in terms of like serializing them out of Pytorch and then swapping them out in a service or something? Like how do you think about that sort of updating stuff over time?

Yeah, for sure. I mean, we'll probably continue adding to the model based on you know, the search queries that come in because it's like we want to be able to identify the celebrities that like our users are searching for. And that by its nature tends to be the more, the newer, the cutting edge stuff. So we'll continue to, I'm not sure when the next time we will because we've got other projects going on, but at some point in the future we'll probably do, do another big sweep across our searches and see who we're missing.

Can I ask a follow up to that before you move on? Yeah, yeah, I'm just curious. So when you're tying, you know, the business side, so new celebrity comes on the scene and you guys are going through some sort of process about, you know, hey, we gotta, we gotta add this in and you have this deep learning model out there. How are you connecting the business side and the technical side together in a meaningful way to where you're able to get on top of it quickly and make it work and get it deployed out? What's the connection between the business side and the deep learning team?

I would say in terms of like, you mean in terms of like making this value proposition to the business.

Yeah, like process. Like how does it go from somebody who has nothing to do with deep learning, is maybe watching the celebrity scene saying, ah, there's somebody else that we need to start paying attention to. How does that get into a task level action from the deep learning team? You know, how does that move across the.

There?

Totally. I mean I usually, I'm in pretty good contact with like our content development team and they're kind of in charge of all the, the latest pop culture stuff and I worked closely with them through this to get the, for them to go through the list of celebs to approve, you know, the different ones and make sure there's nothing bad or weird in there. And, and they can certainly, you know, as things bubble up, they can hit us up and be like, hey, you know, this person's really growing. Can we add them at some point? And we'll probably wait, obviously because training takes a little time, obviously, and we have to make sure we got the right data set. So as we hit a certain kind of set of new celebs, we'll probably just do one big training and then re release the model after that. Hasn't been a huge ask so far. I mean, in terms of business value, I think this isn't something where we're like, hey, giphy's got an API that we offer to everyone that's Part of the giphy API so you can get celebrity predictions. You know, it's not like this isn't. The proposition was never deep learning as a service. It was always like, let's do this thing in house, set it up in house and then also give it out to other people to kind of show how we do things in house. That's why we included like some ops oriented code like Docker and those kinds of things to show how we're our philosophy now for deploying those kinds of things.

Yeah, that's much appreciated by the way.

Yeah, we've actually been working with for some of the latest stuff that we're doing looking into kubeflow and specifically Selden. I don't know if you guys are familiar with that because we use Kubernetes here at giphy for all our architecture and deployments and stuff. And kubeflow seems pretty magical and awesome, especially Seldon because it'll give you GRPC and rest APIs kind of out of the box and handles all the model management, all those kinds of things that historically have been. They feel a little ad hoc. I think everyone kind of just does what they can like oh, put it on S3 and then download it or put it directly into, you know, the image or something. That's. That's kind of where we're moving towards it seems like a sustainable and kind of a very efficient way to handle machine learning, deep learning deployments.

Yeah, that community is moving forward really quickly. I know. Chris and I are both big fans of things that are going on there and in our last episode, so the one that will air prior to this one, we talked about some AI infrastructure and Selden was mentioned there in kubeflow too. And just a lot of the great things that are going on in the Kubernetes community. So I definitely encourage people to check that stuff out and check out that episode if you want more details there as well, specifically. So you kind of moved to talking a little bit about how you really wanted people to try this out. You realized that there wasn't a great open source tool for this and you wanted to release it and kind of put your foot down somewhere in terms of your open source contributions. I'm wondering, being probably the greatest experts on using AI for GIFs, I was wondering for those that are interested in this subject, then maybe they'll look at the GitHub repo which we'll link in the show notes. What are the unique challenges of using GIF data in AI models as opposed to videos or images? Single images. Does it carry over the same techniques like you mentioned? Convolutional neural networks. Some of that I imagine carries through frame by frame or something like that.

Exactly.

How different is working with GIFs and AI models than working with videos and images?

Yeah, it's like you said, we do it frame by frame. For the most part we use a com that just kind of iterate through the frames. The challenges though are just like Nvidia is kind of the same way and truth be told, we do Transcode all our GIFs into different video formats. So we don't always work exclusively with GIFs. Sometimes it is easier just to, you know, if you're doing some stuff, to just work with a video file or whatever. But you know, the challenges are as well, number one, the formats heavy, you know, it's kind of a hefty format. So the compression algorithm is. Is it limpulsive welch or whatever? It's, it's older, right? And it's like kind of dictionary based from what I remember. So it compresses. But it's still a big format. You know, it still suffers from like. Well it hasn't been able to take advantage of like all the things that happen in video. So it's really just like a bunch of images that are squashed together into this file and the player just kind of goes through each of the frames and there's metadata at the beginning and end of the file that kind of say here's the speed and here's the colors and those kinds of things. GIFs are also limited in the color palette. So if you have a high quality video or high quality image and then you transfer it over to GIF or each frame is only limited to like 256 colors I believe. So you're going to lose some coloration there, which could very well affect how your machine learning system, you know, how your deep learning system like interprets the image properties of it and determines the values. So ideally, you know, you would if you can, it's good to work with this high quality, as high quality of images as you can in order so your model can be the most accurate. So this is the kind of the main thing and it's just slowness too. Like you know, if you're moving with big files and it takes longer to train and it takes longer if you're moving stuff over the wire, that takes slower. So those are kind of the biggest challenges I would say that we had to deal with. And maybe this happens with video too. But if you have so you know If A GIF has 90 frames or 120 frames, and it may become the situation where, like, evaluating every frame is not your best option just because of the overhead involved or the time involved. So you gotta, like, do some sampling across frames and then figuring out which frames to sample, and then, you know, how you want to aggregate it and how you want to. If the end result is a single prediction of a celebrity, getting to that single prediction by looking at a single GIF, which could count 100 frames, maybe multiple faces, those kinds of things. Since there are so many frames there and more than one frame. More than one. The frames are different, so more than one face could appear in a frame or not. So you do have to play a lot with, like I was saying earlier with the clustering and how you want to group things in order to end up with that single prediction. So there was a lot of time spent in that component.

So I wanted to ask a quick question. I noticed in the. In the blog post you talked about testing the model for different types of bias, and that kind of piqued my curiosity. I was wondering if you could give us a little bit of detail on that.

Yeah, for sure. We want to do another bias. I mean, we're going to do another blog post around that specifically. But I mean, it's been a. It's been in the news, obviously, that, you know, there are biases built in to. Or like, certain models that have been released, you know, tend to have different biases around race or gender, these kinds of things. And we were kind of curious to see about that. We took all the classes, and I think our methodology was to. We wanted to see the representation of cultures, you know, just in terms of our data set, to make sure that it wasn't something that was leaning any one specific way. Like, we wanted to make sure that we had a big, accurate, like a wide swath of people who were kind of being identified.

Sure, yeah. It's a big challenge for a lot.

Of folks, for sure. And then making sure that, you know, those identifications are proper. So, yeah, we. We looked at the different names and we checked out their. I think we used Wikipedia, you know, to get their ethnicities. And then we kind of did a breakdown and kind of see which ways we'd skewed. And there wasn't any kind of like, major red flags. You know, we had a. I mean, GIPHY itself, you know, we're younger company. Most people who work here, you know, millennials. So, like, we're very sensitive to these kinds of things. And Our user base, you know, tends to be kind of proper representation of, you know, genders and diversity, those kinds of things. So yeah, we found that the classes that we had was like, you know, a good representation and there wasn't any kind of like overweight in terms of any single ethnicity or like gender or whatever. So we put up the. This is probably one of my favorite things about the whole project is that we. There's a link in the blog post to the vector space of all the faces that we have.

Yeah, that was super cool. Check that out.

Yeah, and you can really see. It's interesting to see how a computer or when you take an image and you crunch it down into numbers and then you group those numbers together and then you re equate those numbers with faces. It's cool to see how the computer actually groups faces together based on their properties, you know, skin tone and hair color and all these kinds of things. So it's like it was really funny to see celebrities too who looked alike or who had shared similar characteristics in their face that you had never, never seen. In fact, I mean, to be honest, like our, maybe our very favorite thing about this whole thing was within the company itself, everyone would upload their own face and then they would get their celebrity prediction and there was like a huge email thread that the company shared out where everyone was like, oh look, you know, my match is Brad Pitt or oh, you know, I'm Jennifer Lawrence or Idris Elba or whatever. You know, everyone's got their. Your ego stroked a little bit.

Yeah, that's awesome. I for one, am I. I was just thinking while you're doing or while you're talking about open sourcing things and having that online, it's something I need to, I need to try out and share the results with my wife. I'm sure she would be, be interested to see that. But yeah, I really appreciate you taking time to go through the project with us. Just great work on everything. I really appreciate your, your practical perspective on things and also appreciate you diving into some of the finer points of things like bias and other things. We'll look forward to that follow up blog post. But yeah, I think all of this will be linked in the show notes. So nick mentioned the 3D face embedding explorer, the GitHub repo, the blog post. We'll link all of that in the show notes so check it out. But thank you so much for joining us, Nick. This has been really fun and I've definitely learned a few things about G Gift along the way.

And I learned a few things about pop culture.

Hey, my job here is done.

Then what? Right as we finish out. Do you have a favorite gif?

I do. And so within the Reddit release, when I released, when we put it on Reddit, someone asked for the favorite gift and I put it there. It's Vince McMahon. Do you know Vince McMahon from World Wrestling? He is like this crazy showman with insane faces and there's a meme situation where he has this progressively more audacious reaction to images. So there's this one where there's this body bodybuilder dude and then Vince McMahon looks at him and keeps getting more and more excited until he falls out of a chair and it's just, it just. He kept. I don't know, it's just really, really funny. And that whole the Vince McMahon meme format for me, I don't know. Crack it. Maybe this is the Georgia in me talking about wrestling. But yeah, that, that's maybe my all time favorite.

Awesome.

Yeah, I think wrestling is uniquely suitable there to that gift format. You know, with the expressions and the exaggeration in it.

It's incredible drama. Yeah, we've got some, we've got some great, some great wrestling. Just totally ridiculous wrestling gifts that I could just watch over and over and just kind of ponder what, what the hell, what, what all happening in. Within this thing.

Yeah. Awesome. Well, we'll make sure and add that link to the show notes as well. But thank you so much, Nick. It's been great. Thanks for joining us.

Of course. My pleasure.

All right, thank you for tuning into this episode of Practical AI. If you enjoyed this show, do us a favor us a favor. Go on itunes and give us a rating. Go in your podcast app and favorite it. If you are on Twitter or social network, share a link with a friend. Whatever you gotta do. Share the show with a friend if you enjoyed it. And bandwidth for changelog is provided by Fastly. Learn more@fastly.com and we catch our errors before our users do here at Changelog because of rollbar. Check them out@robot.com changelog and we're hosted on linode cloud servers and eleno.com changelog check them out. Support this show show this episode is hosted by Daniel Whitenack and Chris Benson. The music is by Breakmaster Cylinder and you can find more shows just like this@change law.com when you go there, pop in your email address. Get our weekly email keeping you up to date with the news and podcasts for developers in your inbox every single week. Thanks for tuning in. We'll see you next week. Foreign. Because you've stuck in here to the end of the show. Got a surprise for you. Here's another preview of our upcoming show called Brain Science. This podcast is for the curious. We explore the inner workings of the human brain to understand behavior change, habit formation, mental health, and the complexities of the human condition. It's hosted by myself, Adam Stakoviak, and my good friend Muriel Reese, a doctor in clinical psychology. It's about brain science applied not just how the brain works, but how we apply what we know about the brain to better our lives. Here we go.

That applied brain science really stood out to me because I don't want it to just be data. I want you to go, how can this fit? What can I take away now? How am I going to change? And that, that sort of is where you come in more. And even some of the questions like, so, like, I want to ask you, what are some of the most challenging things working in the tech world when it comes to relationships, probably the most.

Important one is isolation. More and more, the world and companies are being for good reasons. They're being okay with what they call distributed teams.

Yeah.

And that means that you and I, we work for the same company. But you work from your home office. I work from my home office. I might go into the office a couple times a week if I live local. But even if I live in San Francisco, I'm still probably a remote worker, even though I can hop in an Uber or hop on the train or whatever and go into the office and be there in a half hour. But why waste the time, you know?

And this is where I would revisit what I want to talk about with resonance. And that whenever we're learning, no matter what thing, it's really helpful when we get feedback that's both immediate and specific. And so when you're by yourself and you don't have any interaction with other people, how can you get any feedback? I mean, you're losing most of the non verbal communication and you also don't have all of the voice inflections or facial expression. Have you ever tried to, you know, be sad, feel sad and smile at the same time? Try it.

It's pretty hard, right?

Because facial expression is exactly what's involved when it comes to empathy, which is relationships. I was reading a research article recently and it talked about, you know, how couples who are together a really long time end up sort of looking like each other. Yeah. And so what they've looked at is when we actually empathize with other people. Facial expression is really key within that. And so when you, you empathize with the partner you're with over and over and over again, your face begins to make the same creases and facial expression as it relates to where somebody else is emotionally.

Wow.

Right.

Say it again. So that's creepy.

Well, again, this is sort of the hotbed when it comes to neuroscience these days is mirror neurons. And these mirror neurons are what are involved with empathy. And so, so mirroring, meaning I get another person's emotional world. And so one of the research studies looked at Botox and what they found is that Botox because it actually assists in paralyzing facial muscles, but then you can't contort your face so you don't get wrinkles, but actually levels of empathy go down.

Huh.

Right.

Because your physical appearance can't reflect your inner appearance.

You're.

Yeah, you got it. And so when you're working in these remote locations, it, it might facilitate better work or more focus and it allows people to be distributed and to capitalize on the talents across the country. Right.

Yeah. Wow. So that, see that's like a treasure trove in my opinion. Talking about in a scientific way, you know, not just like, hey, this is my opinion.

Yeah.

About all the cons of that. Because I think what we can do is still have remote work, but do it in more healthy ways. Because I'm, I'm fully, I mean, I've been self employed remote worker since 2006 now. I'm a unique animal. I, I know that, my wife knows that.

Right.

I'm fine with it. I'm a good human being. But I've got some flaws and I'm willing to accept and share those to some degree. And I think the problem is we just, we just lack more, maybe a more purposeful or intentional feedback loop.

Yeah.

Which I think is super important to being able to operate in this world in just good ways. I don't know. Healthy ways is probably the best way to use in this show context is healthy ways. That's a preview of brain science. If you love where we're going with this, send us an email to get on the list to be notified the very moment this show gets released. Email us@enreviews changelow.com in the subject line, put in all caps. Brain science with a couple bangs if you're really excited. You can also subscribe to our master feed to get all of our shows in one single feed at the changelog.com master or search in your podcast app for Changelog Master. You'll find it. Subscribe get all of our shows and even those that only hit the master feed again. Changelove.com master.