Making GANs practical

Bandwidth for Changelog is provided by Fastly. Learn more@fastly.com we move fast and fix things here at Changelog because of Rollbar. Check them out@rollbar.com and we're hosted on Linode Cloud Servers. Head to linode.com changelog this episode is brought to you by Digital Ocean, the simplest cloud platform out there and we're excited to share. They now offer dedicated virtual droplets and unlike standard droplets which use shared virtual CPU threads, their two performance plans general purp CPU optimized They have dedicated virtual CPU threads. This translates to higher performance and increased consistency during CPU intensive processes. So if you have build boxes, cicd, video encoding, machine learning, ad serving, game servers, databases, batch processing, data mining, application servers or active front end web servers that need to be full duty CPU all day every day, then check out Digital Ocean's dedicated virtual CPU droplets. Pricing is very competitive starting at 40 bucks a month. Learn more and get started for free with a 100 credit at.co changelog again.co changelog. Welcome to Practical AI, a weekly podcast about making artificial intelligence practical, productive and accessible to everyone. This is where conversations around AI, machine learning and data science happen. Join the community and slack with us around various topics of the show@changelog.com community. Follow us on Twitter. We're @practicalai FM and now onto the show.

Welcome to another episode of Practical AI. This is Daniel Whitenack. I'm a data scientist with SIL International and I'm joined as always by my co host Chris Benson, who is a principal AI strategist at Lockheed Martin. How are you doing Chris?

Doing well today. How's it going Daniel?

It's going pretty good. Got, you know, models, training and messy data to work with, so as good as any day could be, I guess.

What more could you ask for?

What? What more could you ask for? I guess in certain scenarios you might ask for interesting models, which is what we've got to talk about today. On previous episodes, I know we've mentioned Gans a few times and we've talked about some of the specifics, but not a whole show devoted to them. And so we thought we'd dig into this topic a little bit more, and one of the ways that we thought we could do that was to get some experts, and that's what we've done. So we've brought in Jacob Lunger and Vlad Bock, who are the authors of the book Gans in Action, and they're going to help us parse through all things Gans. So welcome Jacob and Vlad, thank you.

Yeah, thank you for having us.

So before we begin and jump into Gans specifically, let's jump into each of your backgrounds and hear about how you ended up where you're at. So maybe, Jacob, could you start things out?

Sure. So I was sort of working in machine learning since about 2013. Obviously back then GANS weren't a thing, but I sort of fell in love with the field and I was sort of firstly curious about all the latest and greatest things going on in research. And as I was sort of following conferences and like the researchers that I really respect and admire, I came across this thing called Generative Adversarial network, sometime in 2015, I believe. So that was quite soon after the original paper by Ian Goodfellow came out. And I just sort of fell in love with the technology and the whole idea that I'm sure we'll get into later just kind of really made sense to me. So I was from that point on really curious, but only sort of as a part time hobby. And then over time things sort of started picking up. So I started writing more blogs, blog posts about it. And then eventually Manning approached me with a book offer. So I think that was where it really started to take a more coherent form and I started dedicating more and more time to it, to now where it's taken over all of my life. You know, I'm working full time with GANS and you know, doing these types of out, you know, communication and outreach type of thing, which I think, you know, is just a testament to how far Gans as a field have gone. So it's kind of ramped up a little slowly. But, you know, now it's kind of really, you know, took over every aspect, which is great because I think I really got to see the field from its inception, which is not something you see every day.

Yeah, it's been a pretty quick ride, I guess. So you were saying like 2015 was around the time that, you know, Ian Goodfellow came out with a paper and that stuff kind of started getting momentum. Is that right?

Yeah, so I think the original paper was presented at then NIPS 2014 and then I think, you know, then you saw like a slow trickle of papers which kind of eventually turned into like an avalanche by like 2016, 2017.

So Vlad, I was wondering if you could give us a little intro about yourself as well.

Yeah, totally. I studied computer science, so machine learning was always one of my interests, both personal and professionally. And after college, after a brief stint at a Y Combinator Startup where I worked as a data scientist. I joined Microsoft and Microsoft has an ARM called Microsoft Research, which is essentially R and D division. It's effectively the Bell Labs of our time. And there I got involved with a research project where we used GANS along with my team. And it was just fascinating to see the margin by which on data generative tasks, this technique has exceeded everything else that used to be the state of art. So it was truly this stepwise improvement that is rarely seen in or used to be rarely seen in machine learning. And from there I stayed involved with the field.

And now are you working in a practical sense day to day with this technology?

A little bit here and there. Although I must say that when it comes to practical applications of most vast majority of machine learning and deep learning techniques, it's still very much in the supervised machine learning area and less on generative tasks. So when it comes to my day to day job, then GANS are almost no involvement.

Gotcha. Yeah. And maybe we can get into some of those practicalities a little bit later. But to kick things off into GAN world and I'll let you guys choose who wants to take on certain questions. But maybe one of you could just give us a brief, like what makes a GAN a gan? How is it different than what we might think of when we think of a quote unquote normal neural network or a normal machine learning model, whatever that is.

Sure, I could give that a whirl. So I think the two areas that I would highlight is that first of all from like very high level, Gans generally live in the unsupervised world, which Vlad alluded to already, that there's this distinction between supervised and unsupervised. So most GANs exist in a world where you don't actually need any training labels, which is where the supervision comes in. Rather, the most generative models learn from the data itself. So all it needs is just raw data and it manages to reconstruct it. So, you know, the prototypical example being human faces. And just by feeding GAN a whole bunch of human faces, it will eventually learn to reconstruct completely novel faces that are not in the original training set. So that's, I think idea number one is that it's unsupervised. And then idea number two is so the model itself effectively uses most of the time two neural networks that compete against each other, one being called the generator, one being called the discriminator. And the generator is almost like an amateur painter or something like that. And it tries to basically take Some inspiration, generate a new image. And then the discriminator is like an art critic who says this is a good picture or this is not a good picture. And through the art back and forth process, they both get better over time at generating and then telling apart real from fake. So at the end you can have a generator that's pretty good at producing realistic images. So that's kind of like the high level summary. Anything to add, Vlad?

Yeah, totally. I think it's a great intuitive description. I would just add some of the technical details in which GANS differ from traditional neural networks. And a core of it is the training procedure where a traditional neural network is effectively an optimization. You have a very complex loss space in which you are trying to minimize some loss function which is the objective. So essentially there is some measure of an error or how far the neural network is from its objective. And then the training procedure just uses calculus to minimize that objective. With Gans, since there are two neural networks, the training process can be better described as a game rather than an optimization which has far reaching implications on the training process itself and the outcome of the network.

So yeah, in this game, and maybe this is related to Jacob's discussion of being unsupervised as well, is the idea that you have these, let's take the faces example. So you have real faces or real pictures of faces, and essentially those are kind of labeled in the sense that those are real faces. And then anything that comes out of a generator model trying to generate real faces or trying to generate realistic faces, those are kind of automatically labeled as fake. So there's kind of this fact that you know, everything you pump in is real.

Yeah, absolutely. So actually there's an active debate among researchers in classifying Gans because when it comes to the training process, then exactly as you alluded to, they can be seen as a supervised machine learning because you do have implied labels in real or fake. And there are also some GAN models that actually have explicit labels as well. But even the traditional core gan, there are implied labels. However, when it comes to the training setup, then a more correct description is unsupervised because there is no need for humans to painstakingly label this data. So usually the bottleneck in machine learning is access to a large data set with labels. And Gans and the GANTT paradigm helps to solve for that problem by essentially having the labels implied. There is no need for a human to go image by image and label which one is fake and which one isn't, because that comes from the very nature of the setup of the GAN model, when we look at it from a practical standpoint, then they can be seen as an unsupervised model. Although when we look at the particulars of the training process, then we are measuring a distance to an explicit label. So from that standpoint, the training can be seen as supervised in a way. So they are definitely in this gray area between, you know, supervised, unsupervised. And the Google researcher Francois Schollay, who developed the cross Python programming deep learning library, he refers to some techniques like this as self supervised as well.

So you mentioned that there's these two models that are involved and it makes sense to me kind of now what you're talking about in terms of unsupervised and how to think about that. I guess maybe in training each of those models, does each one still have the idea of loss in terms of what it's trying to generate, or are they completely different? The training of each model individual, is it completely different in some way?

Well, I was just not 100% sure that I understand the question, but obviously the two networks have different inputs. For one of them, it is the latent vector. One of them is just typically a random sample from some somewhere between like 100 and 500 dimensional vector. And then it's just like a, you know, the same way that any latent space would be. For those who are familiar, it's kind of relatively meaningless vector, though there are some qualifications to what you mean by meaningless specifically in this case. But it's just a vector of random numbers and then uses a process to basically get to transpose convolutions or deconvolutions to size of the image. Right. So it reshapes that vector through learned transformations, something that looks like an image, ideally coming from the original data distribution. And so of course that is how you evaluate it as well. And the way that it gets the feedback is through the discriminator, which then tries to effectively, like we discussed, label images that the generator produces. Right? And every time the generator manages to fool the discriminator, it gets a lower penalty than the discriminator who gets penalized more because it got something wrong. So it's this iterative process, but of course, like they're, you know, in the original formulation, like Vlad alluded to, it could be thought of as a game. So the two networks are basically set directly against each other. It's a zero sum game, right? One has to lose in order for the other one to win. But it turns out that even that works relatively well. But in practice, people use more complicated loss functions that just have better, like, numerical properties. But the original formulation kind of gave it some solid theoretical grounding. So people were sort of more willing to accept why that works and then just kind of use these numerical tricks to make the training more stable or give it some other nice properties. And we can talk about some of the more advanced variants later. But that's basically what the generator is doing. And then the discriminator is basically just classification algorithm. Right. So, you know, you can think of it as a real or fake detector like we discussed. So the training process there is more natural to what most people are used to thinking.

So I guess could we turn to maybe a couple of examples and talk a little bit about, you know, what makes GANS kind of useful or interesting for certain tasks and what are kind of the range of tasks that you could use them on and kind of, if you could maybe throw out a couple of different examples about how GANS would be implemented to solve a particular problem more conceptually and stuff to give people a sense of how to fit this concept into their thinking as they're learning this topic.

Absolutely. So GANS are, as their name suggests, generative adversary networks. They are well suited to generative tasks, which is where you generate synthetic yet realistic looking piece of data. And GANS have been particularly well suited to generating fake imagery. So you may have seen in media that there are fake images of human faces that are at photorealistic quality, or even fake videos of statements by famous people that were never made. And yet the video footage looks as if a Hollywood studio made it. But it is something that researchers were just able to produce using this technique. A great way to think about it is in contrast to what machines used to be good at until GANS came along. So machine learning and later on deep learning is excellent at uncovering patterns in existing data and then using that insight to unsupervised machine learning tasks such as regression or classification. So, for instance, there has been huge advances in machines that were taking an image as an input and then categorizing it to the correct label. So you have an image and then the machine tells you it's a dog, or you have another image and it tells you it's a cat. What used to be extremely difficult until Ian Goodfellow came along with this innovation, was doing the reverse of this process, essentially taking the label dog feeding into a neural network and having an image of a dog being produced at the end of it. I mean, this is overly simplifying. It Especially when it comes to internal workings of it. But conceptually, it's essentially instead of having a computer classifying something, it's having a computer create something. So it's a. Philosophically, it's like a level of imagination or creativity that the machine would have.

Yeah.

This being something that was extremely difficult to do because it's extremely hard for a human to define what a realistic image of a dog is. And it's also extremely difficult to capture mathematically. So the other generative tasks that or other generative models that came before Gans, they were usually trained by recreating the same image. It was fed in that it was fed. That's essentially like taking an image, compressing it into a representation, and then recreating the image itself. And then you can tweak the internal representation to produce a fake image that is somewhat similar to the original one. What Gans were able to do is that the generator itself is not learning explicitly by something that a human programmer would define or a researcher would define. There is no explicit loss function for the generator. What is happening is that we have another neural network, which is the simple classifier, which is the discriminator that helps the generator generate something that looks realistic. So effectively we have another model that helps us do the teaching. And what's really fascinating is that what Ian Goodfellow accomplished is taking something that machines used to be are good at, which is classification. Right. Like taking an image and saying it's real or fake and using that insight to help machines achieve something that used to be very difficult for them, which is generating realistic data.

So maybe to just quickly add on to that, I think in terms of your original question, right. Like around the application side of things, my perhaps slightly sort of less, you know, something like, I guess this is an informal podcast, so some researchers I know would probably object to this, but.

All is safe here.

Yeah, exactly. So I think it's a reasonable approximation to how to think about Gans is to think as the like basically first learned general purpose generative framework. Right. So Gans have been sort of applied, as we discussed, in images and video, but also people might not know that they have been successfully applied in like tabular and highly structured data, natural language processing, audio. I've seen papers in network theory and graph applications, obviously lots of artistic applications, you know, even some defenses against adversarial examples, though that is still far from being a solved problem. So obviously Gans have had like sort of managed to replicate so much interesting data. But in terms of like the business applications that they tend to be non trivial. Right. Like you need to really think, where does this technique make sense? It can be extremely powerful. And, you know, unlike with classifications and sort of decision theory and things like that, we have not had the ability to recreate, you know, realistic data for that long. So it is not always immediately obvious to how to, you know, apply it in a business process or something like that. And the reality it is that this is a technology that's like literally four or five years old. So I think it'll take a while till there will be like some sort of, you know, mass adoption. But I think there's lots of interesting things in the fact that you can apply this technique in all these different domains. But, you know, the applications of like, how to use it, you know, sometimes will be. I think one of the earliest practical applications I remember was in something in dentistry where people try to actually propose, which is another area that I haven't even mentioned. Propose, like a 3D mesh of the crown, I think that they were trying to fix. So basically to create artificial crown that would fit into the patient's mouth and with the rest of the teeth using gans in the 3D space. I'm not sure when I was doing research on this, I'm not sure how far it actually went because there was a lot of like, I think 2016, 2017. There's a lot of talk about it and people like love to mention that example, but I have not seen that much follow up. So I'm not 100% sure what happened there. But, you know, there's definitely lots of really interesting applications. You just need to think about, like, how do I use this strength? Right. Rather than like very obviously applying it to some sort of business process where you have like, okay, you know, I need to make a decision here. I'll just train a deep neural network to do that instead.

So I've noticed over the course of the conversation that we have actually mentioned, we've referenced Ian Goodfellow half a dozen times. And I thought it might be worthwhile to ask you guys if you could just kind of tell us who Ian Goodfellow is. Obviously he's a pretty big deal in this industry and with you guys working on Gans, if you just kind of give us a little bit of a background on who he is and what he did and why it's significant to the conversation.

Yeah, so Ian, why is so important is that he's the single person who invented this technique. Of course there were other his colleagues that are on the paper but when it comes to receiving the credit for the invention of generative adversarial networks, it's him. He was a PhD student at the University of Montreal where in 2014, at the end of the academic year he went out drinking with some of his friends and they were discussing what I alluded to at the beginning of this dialogue, how difficult it is to have machines synthesize photorealistic imagery which used to be from a research standpoint, an intractable problem. And Ian came up with the idea of the two dual link neural networks which he then after returning home from the pub, coded up. And later that year he and his colleagues have published a paper that truly started this field. And then of course there was the research community took it up and there were since then huge innovations made both on the original GAN model, both when it came to the complexity of the model itself as well as its application, the use of labels during the training, both the generator discriminator or one of those. So the field has advanced considerably in just a few short years that it has been around. But yeah, and again he is so prominent because he's credited with his. With this invention.

Yeah, I think and just to note, we're often asked about learning resources, he is one of the three primary authors of the Deep learning textbook which we have referenced on this show many times. And though he has worked for various organizations in the past, I believe he's currently with Apple at this point.

Yeah, that's correct. Yeah, he was at Google OpenAI and now most recently he joint Apple which very excited to see what we will come up with because I'm fairly certain that the technology that is powering the emoji that you can manipulate with your face with an imessage, the underlying technology for that would be gans. So when we have seen a lot of the flashy applications of creative image processing and image editing, like the applications that make you look older, like the Face app, or the applications that make you look younger, like the baby filter on Snapchat, tend to have GANS as the underlying technology. There are also other photo editing pieces of software that have very advanced features that are also using GANS as the underlying technology. So when it comes to creative applications, when it comes to the immediate commercial use cases, then image editing is where GANS have shined. But I think that's only scratching the surface of what will be ultimately possible with GANS in particular and also the research directions that this technique has opened up.

So to kind of summarize and I'll make an attempt at this and you can tell me if it's a good summary. But to kind of summarize, it seems like that Gans, as opposed to other kind of quote unquote normal models that people might envision, is that there's actually two models, the generator and the discriminator, and they feed back to one another. And one is trying to generate something, whatever that might be, based on some randomness, and then one is trying to differentiate between the generated version and some gold standard or real version. Would that be a good overall summary?

Yeah, functionally, yes.

Okay. And I was wondering, like, for those two models, I guess I had a couple questions, but one of them is like, let's say we take the example of the faces or something like that in terms of the mechanism feedback between the two. Is it that when you are generating things, you generate a whole bunch of fake faces to mix in with the real faces and then try to discriminate with the classifier model or retrain or update it? Or is it a sort of one at a time thing, like you generate one face and then add that in. What's the sort of balance that happens there and the considerations you have to take into account?

So if I get your question correctly, I think it's about the training and how, you know, how to balance the two networks learning regime. And I think, you know, you kind of hit the nail on the head here because that tends to be one of the most challenging aspects of Gans is the training part. Because even though it might sometimes seem like magic, it's obviously driven by real algorithms. So to nail that dynamic can be very challenging. And in my day to day work, or even over the course of just playing around with new research papers and their code, that tends to be one of the biggest challenges people have proposed. Literally. I don't think it's an exaggeration. There's literally hundreds of papers, if not thousands written on just the training dynamic alone. And it's obviously like quite a bit of challenge to get that exactly right. There's like techniques that keep on popping up over and over again. People in and, you know, many of those would be like covered in the book. But to feel this ever evolving, right, there's new things coming out. You know, every major conference there is like at least, you know, 5 to 10, like new proposals on how to improve this training dynamic. And some of them take off and, you know, start to be incorporated by more papers. Others sort of might have been good, but like fade into obscurity through some, like, you know, the Pseudo random process of academic discovery. That's kind of my take on it. But yeah, I think in general, like I've noticed that, you know, having solid starting architecture like close to something that you know, has worked and look at the data set that you're applying it to because, you know, a lot of the academic work tends to work on fairly standard data sets. If you're applying it on something else tends to be very different. So you need to think about the data set as well as the network and the architecture, which I think just kind of talks about one of the differences between academia and industry, where the industry problems tend to revolve much more about the data set than thinking about the sort of, as Kaparthi calls it, like the data programming.

So is this part of the reason? Maybe why. So Vlad mentioned that when he was giving his sort of introduction that there is still a bit of a struggle to kind of make the transition from GANs to their application in sort of day to day data science work in a widespread manner. Do you think most of that is because there is still a lot of fuzziness around the best way to approach training? Or what do you think's factoring in there? What are some of the, is that the main challenge or are there other things kind of preventing that?

I mean, I personally think that there's quite a few challenges. I think training is definitely one of the them. But I think realistically, I mean, even if you look at the state of our field more broadly, like not that many companies are successfully deploying deep learning models, even supervised on a regular basis. So I think that, you know, obviously the infrastructure and the support and the business thinking about the whole machine learning space is maturing. But I think, you know, generally, I think the reasons for why gans have generally been applied mostly as like specialized startups or very specialized business units that have someone with a lot of GAN experience is for obviously the training difficulty, but also like having the right set of business sort of incentives or not even incentives, maybe more like intuitions around how to apply GANT successfully. I think there's a lot of really, really good applications for them, but you generally need someone who can, you know, sort of tell you where that extra effort is worth it because it will be somewhat challenging because of the training dynamics and other things to deploy. So I think you need to have someone who can like sort of guide you through what makes sense in this situation, but also like someone who can like pick out the right tool for the job. So even machine learning broadly is still relatively novel. Though to us it might not feel that way and I think businesses are still trying to catch up.

So I'm curious, as we've been talking about this and we've gone into some depth about how generators and discriminators work, and I'm wondering, are there other models, either other types of neural network models or other machine learning models outside the neural net space that you could use as a generator or discriminator?

It's a great question. So the discriminator itself, if you actually isolate it from the GAN model, then that's just a classifier in most of the incarnations of the GAN architecture. So these are two separate neural networks that can effectively function independently. When it comes to generative tasks, then there have been other models that were used for that purpose. You have the image in particular, you have restricted bots, machines and autoencoders that were used for this purpose. But when it comes to image generation, and this applies to both static imagery like photos or to video footage, then GANs are indisputably the state of art for those type of tasks.

So we kind of gone over the basic generator discriminator, the interplay between the two and the specific, specific models. I know that you talk about a few more advanced types of GANs in your, in your book though. Is there a whole, I guess this is a whole research area and there's probably a lot of different types of GANs, but are there some more advanced types of GANs that are starting to filter into maybe a little bit wider spread usage?

Yeah, yeah, yeah. Jacob, do you want to talk about cyclegan and I can then talk about semi supervised paradigms.

Sure.

So I think, yeah, I think like Vlad said, I think Cycle GAN is definitely one of the more prominent examples. I think when you talk specifically about the different cases and different applications. I think for example, Biggan is very popular with sort of as an artistic tool. So there is a tool you can check out made by one of my friends, Joel, who wrote ganbreeder. I think it's ganbreather app or something like that. And you can sort of create new like combinations of images and people have entire like art collection just off of generative art. That's been happening for quite a few years now. I was very surprised at how quickly like gans became popular among the sort of artistic community. There's like lots of like digital artists using GANs as their primary tool of choice. So there's Big Gan, Psychogan and I think there has been recently. Well, a couple of startups popped up Using Stylegan as well as a way to generate stock images of very high resolution faces and a bunch more. But I don't want to steal all of the material from Vod.

I was wondering, but as y' all are talking about this and you're kind of going through these different versions, if you could. I recognize you don't want to go do a deep dive necessarily on all of them, but if you could give us a little bit of explanation with kind of each of the titles that you just went through, so that those of us who are not as familiar with them can kind of categorize them in our own thinking as we try to absorb the material.

Yeah, I think maybe you mentioned Cyclegan and Stylegan, I think was it?

Yep, I heard both those.

Is there like a quick. Will really test your summarization skills, like a quick one sentence description of each of those.

So I think the way to think of Psychogan on a very high level is you have two domains and you're trying to basically always map one domain into the other. And it's very surprising how broadly that transfers. So for instance, satellite images to something that looks like Google Maps or night to day, these could be different domains. And you can then have a generative framework that can translate something that is a day image into a night image and back again. So you can kind of have these two domains and you can have one generator for each translation, basically, if that makes sense.

That does. That's a good one. I like that kind of hands on aspect because as I'm trying to follow what you're saying, that gives me a tangible example. Do you have something similar for the Stylegan?

So Stylegan. I think most of the advancement there is in the algorithmic. From the algorithmic perspective, I think one of the things that Stylegan had, one of the major innovations, just a big one, was the fact that you sort of keep. You don't source from the latent space just at the beginning, but you keep adding information throughout the generative process. So that means you can influence different levels of features and have much finer level of control. So because you keep adding sort of sources of information and inspiration, quote, unquote, throughout the generative process at different layers of resolution, you can sort of more finely tuned, you know, like the big, like broad aspects of the face or very tiny details. And you have a more granular control than just the initial vector. You know, if you just want to change the hair a little bit, you can do that. Or if you just. If you want to change like the Whole face or ethnicity or, you know, gender or these big. You can do that also. And you can kind of sort of have a better way of expressing what type of image you'd like to get.

This episode is brought to you by Brave. Big news from the brave team. Version 1.0 is official. That means our favorite open source, privacy focused, blazing, fast browser is ready for prime time. Their brand new iOS app landed just in time for the announcement. And the Brave team is celebrating by granting 8 million basic attention tokens to the community. That means when you download the iOS app, you get 20 bat absolutely free. Put it to good use by heading to changelog.com hitting the triangle icon in the upper right hand corner and flipping us a tip.

So, Vlad, I think maybe you had some, some other input as well in terms of maybe advanced or specific different kinds of gans that people are pursuing now in a sort of wider sense.

Yeah, sure, yeah, I can talk to the conditional gan. So in a regular gan, you have the data set of, let's say, real images of human faces that the generator, over the course of the training iterations, learn to mimic. But similar to what Chekhov was mentioning about Stylegan, there is no way in the classic gann paradigm to control what type of image would get generated. So once the generator gets trained on a data set of real human faces at any given time, when you feed it a random vector, as in that's the latent vector that Jacob was mentioning earlier, it would spit out a face, but the researcher would have no control over whether the face is a man or a female or a child, or let alone more fine tuned feature, as in like a human with glasses or somebody with long hair or short hair and so on. For the conditional gan allowed to do, which was one of the early innovations that was since then fine tuned by the research community, was to introduce labels during the training process, which allowed the discriminator to not only recognize whether an image is real or fake, but also whether it matches the given label. So in the example of human faces, it receives an image, it's told whether it's real or fake image, but it's also told what gender it is. So therefore, for the generator to be successful at fooling the discriminator, it needs to produce images that are not only realistic looking, but also ones that match the label. And the magic of it is that once you have the generator properly trained, you can then pass it latent space and the label that you would produce, such as I want an image of a child and it would given Sufficient training data set and properly trained, it would then produce a fake example matching the label of your choice.

And on the discriminator side, in that case, would it just be a matter of adding a feature to the input of that classifier? That would be whatever it is, gender or ethnicity like was mentioned. Is that just another feature that gets factored into the discriminator?

Yeah, exactly.

There are different implementations how this can be done on a technical level, but broadly speaking, you're absolutely right. It's essentially training classification that isn't only binary as in real or fake, but it is one that is taking into account also the correct label. And what's really great for the conditional GAN paradigm is that the additional information that the training process is conditioned on can be arbitrary. So it can be a description, it can be a single label or it can be a description. So there are also GAN models that can take in a set of tags or even like a word description and then produce an image that is matching the description. So you can for instance, feed it a description, say like bird sitting on a branch. And if properly trained and given sufficient data, then the generator would produce a fake image matching the description. You know, of course our imagination can go wild, but I could see this having tremendous practical applications, especially in spaces like animation, where currently you need a lot of effort by human animators to create, let's say, characters in a game or characters in a, say, Pixar movie. But with GANs, you can greatly optimize this creative workflow by having the trained model essentially aiding the human animator. Or if these techniques get advanced enough, you can imagine digital worlds, be it in VR or even like a regular PC game which self creates. So you can essentially going to be an infinite world where characters get generated on the fly, new terrains get generated on the fly, and are going to be extremely believable without the need for input for a human animator or even programmer.

Gotcha.

So I'm curious, there is so much research going on right now in gans, it seems to have really exploded in terms of just the. So many people and organizations are interested in this and trying to level up. What are some of the biggest open questions that are still out there that people are trying to address right now. Where do you see the top researchers really focusing?

So just to put my perspective, and I think definitely keen to hear what Vlad thinks about this as well, there's the training question that we've already alluded to. And I think the other big area is sort of these more complex data sets that I think are only kind of getting started the whole audio synthesis. I think the first papers that I remember seeing using like audio where maybe at ICLEAR this year iclr which is like one of the big conferences, maybe there was something before but that was like the first time I saw like good attempts at doing that. And I think just last month or two months ago DeepMind released GAN TTS. So there's just an example of a vertical where I would expect more things to happen. And so I definitely think that there's like a lot of scope in these sort of non visual types of data as definitely at least in research. I'm not so sure about production like in the next year but like research definitely. And the third area I'll mention is just having a GAN being incorporated in some bigger process. I think a lot of the time where I've seen some of the more successful applications of GANs that have actually gotten deployed where things that it was a sort of supporting process either on the training side or some sort of post processing side for a larger like machine learning pipeline. So I think gans have a lot to offer if you know how to like you know, added into your algorithm, whether that's as a domain adaptation algorithm or just some sort of, you know, better sort of tabular data like generator or anonymization tool, that sort of thing. So there's definitely a lot of scope for them to be incorporated as one of the pieces in a bigger puzzle.

Awesome. Vlad, did you have anything to add there or. I don't want to cause any friction between the two of you, but if you disagree or have any thoughts. What are your thoughts here?

Yeah, no, I think it's like the gans are great that there are countless research directions and different people can get excited about the different opportunities there. For me it's primarily the ability to leverage the internal representations that the GAN model learns along the way to succeed at the generative task.

So this would be a similar idea to what people talk about in sort of like I guess in the NLP case we're talking about word embeddings and things like that.

Exactly.

Is it a similar idea?

Yeah, precisely. Vert embeddings is a perfect example. So you may have heard of the simple arithmetic that we can perform on world embeddings to demonstrate that the machines or the neural networks develop a very complex internal understanding of the semantics of the human language. So for instance, you can take the embedding or the vector that describes the word king you can subtract like pure Arithmetic the word man, and then you can add the word woman, and the resulting vector is going to be very close to the vector for the word queen, which again, we take king, which is male royalty, you subtract the work from men, you add woman, and then you end up with female royalty, which is queen. And what Gans have been demonstrated to do is that you can perform the same kind of arithmetic on images. And it's really fascinating. So when you take the example that was published in 2015, so very early on in the short history that Gans have been around, you have an image of a man with sunglasses, you subtract an image of a man, you add an image of a female, and the outcoming image is a female with sunglasses. So you can perform arithmetic. And this is completely unsupervised. This is just based on the internal representations of this extremely complex space, I.e. images, that you can then perform, again, something intuitive as arithmetic. And then the computer, without being told what the correct answer is, would come up with an answer that a human would, based on our intuitive understanding of what arithmetic on images should produce.

Just as a quick interjection with that arithmetic as you're describing it. I mean, doing that on imagery, on video, I mean, that is a deep fake, at the end of the day, just to kind of tie two terms together. Or am I wrong?

Yes, it works on the same principle. I'm not sure if deepfake works. Deepfake might actually work more on the principle for Cycle Gan that Jacob was talking around previously. But deep fakes, vast majority of them, are based in one way or another on Gans. And it is. Exactly. It's essentially modifying an image in a believable way so that it looks like something else. And this is also at the core of the applications that I alluded to earlier, as in the face app, which you take a selfie and you can immediately make yourself look older. So it's translating yourself into an older version of yourself and things like that.

Yeah, I can do without the older version, because that's already happening quick enough anyway.

Yeah, exactly. Just like, just wait a little bit. But what's fascinating is that the older version is like the perfect example of something where machines just cannot have the data, because you would have to literally wait 50 years for people to get older and have images of them so that you can train it in a proper, supervised way. As in bad examples of this is what this person looks like young, and this is what this person looks like when they are 70 years old. That's Effectively impossible to get unless you get clever, which these generative models have been extremely helpful with.

Yeah, since Chris went there. I didn't go there. I don't want to say anything bad about Gans, but since Chris brought up the idea of deep fakes and all that that people are concerned about. Thanks for explaining that. You know, how GANs are particularly well suited for this sort of task because they develop this deep understanding of a deep representation of these images and features and different things that are important. You know, people put a lot of focus on that deep fake stuff and you've mentioned a bunch of other applications, but from your perspective, I guess, do you think that there are a lot of good examples, positive examples of GAN usage out there or are you concerned at all that the deep fake stuff and you know, obviously that's what gets retweeted a lot on Twitter and blogs and all of that. Are you concerned that that's kind of overwhelming the attention around Gans when there's a lot of good uses of them?

So yeah, so I would like to kind of set a couple of things straight. So first of all, like, so this is super important topic, right. Like, and I think there's a reason we like that's part of the motivation why we wanted to include at least like a short ethics section in the book. But because obviously we understand the same time that people don't. It's not a book about ethics. Right. But I also want to highlight one thing which is that the original deepfake algorithm and quite a few of the more successful are not actually gap based but there are other techniques. I just want to like equate the two terms. So you know, GANS are one of the ways to do deep fakes. And in fact like, so there's a London based startup that I know some of the researchers from called Synthesia, which kind of uses this, uses a mixture of several techniques including GANS as a way to sort of reanimate faces and sort of do a more realistic dubbing. So effectively when you have a movie that's in English and then you want to port it over to, I don't know, Chinese market, Russian market or you know, like any other country if just if you. Dubiously, you can sometimes see that that's not what the people are saying. So they have kind of been, been applying this in like the movie space to basically make improve the experience for moviegoers. And I think, you know that there are obviously many positive applications even of deep fakes. But as overall as a technology Whether that's a net positive or. And negative, I don't know. What I would say though is that I think, you know, Photoshop and other collation tools like After Effects have been around for at least a decade maybe. And realistically, you know, and when you look at where people can do a lot of damage with misinformation, it starts already with something much simpler. Right. Like things like articles or text. Right. The GPT2 and from OpenAI and that sort of thing. There's a lot of stuff in between. And I think it's a really important debate around like, do we want to. How do we want to approach it? Because of course you can't uninvent any piece of technology. And you could even guardrails that you set around them, I think are potentially problematic because if you set those guardrails in almost any way, there will be some edge cases, they'll fall outside and then you'll get use cases where. Or cases where people will just get fooled by something because normally they're protected. So I think the typical example being like the platforms trying to do the policing, I think it's a very long and complicated debate about, you know, what the boundaries around that should be. So I'll leave it fairly high level. But that's my take on it.

Yeah, I think that's a great take and I think it gives people a lot to think about. Of course, with a lot of things with AI, I think there's important ethics things to consider. But at the same time, I don't think it's worth writing off specific techniques as, you know, blacklisted or something because they could be used in a certain way. I remember. I think, Chris, maybe you remember, but I'm pretty sure at some point during our conversations it was brought up that, you know, Gans are. There are quite a few really, really positive uses of them, like in a social good sense as well, in terms of. I think that there was one I saw where they were generating like augmenting tumor imagery data sets to help actually train tumor detection algorithms, because that data is. Obviously there's privacy concerns, but it's also fairly scarce and hard to supervise, that sort of thing. So I think that there are, you know, and I'm sure you guys have better insight into this, but there are a lot of positive use cases. Right.

You know, just before you guys jump in, I remember when we talked about deep fakes some episodes back, we had a lot of people in the audience come back at us and say, you know, hey guys, I recognize you were being a little bit, you know, dark in that and we had acknowledged that. But a lot of people came back talking about good uses and I just.

Wanted to kind of call that out and.

Sorry, Vlad, I think you were starting to talk.

Yeah, I was saying absolutely, like medical space. I think it's a great example to bring up where again, augmented data sets can be tremendously useful in unlocking diagnostic applications, where previously there just wasn't a sufficient enough data set.

Awesome. Well, we've had a lot of great conversation here. I know I've learned a lot. And obviously we will link your book in our show notes so people can find it. Gans in Action. And that I'm sure is a great place to start in terms of getting to know about these subjects in a lot more detail. But I was wondering also if, you know, maybe to close us out, if you both have any insight into, you know, where's the best place for people to jump into this subject in terms of maybe like certain frameworks that are easier to work with Gans than others or maybe it's like certain example tutorials or problems or tasks that would be a good place to start and jump in and experiment and get hands on. What are your thoughts for our listeners that are interested in that?

Yeah, so we might be a little biased here, but. Yeah, we might be a little biased, but I'll let Jacob comment.

Oh yeah, well, so I think I'll try to provide one example for each. Right. So obviously I think keeping up to date with what's latest, I think Twitter and following the right researcher is the best way in terms of frameworks that are easier generally. I mean, I have not had extensive experience with TensorFlow 2.0, but I find Pytorch to be a lot easier to work with, especially for Gans, because there's a lot of complicated things that are going on and it's easier to kind of dive into the internals. So I generally kind of tend to prefer Pytorch over tensorflow these days. And I think, you know, I think it's especially with the latest improvements that Facebook's making into Pytorch, I think there's a lot of things that are picking up, especially amongst the research community, which is kind of where live right now, like in the bridge between like the research and industry. So George is good and I would say that I think in terms of the types of networks that are most tried and tested, I think like dcgan, Cyclegan. Yeah, those would be like the, like Stylegan, obviously actually is those would be like the three bases that I think are really important to understand. But of course, like, it's a massive field. So I think having, like, a comprehensive resource, you know, like, like our book is, it could be really useful as a. As a sort of map to get. At least get your head wrapped around it. Because there's much more than I can. I can say in this case. But yeah, hopefully that's good.

Awesome. That's great. I think it gives a lot of good perspective because people want to start poking around and I encourage people to get the book and dig in to this subject and, and work on good positive examples that can, you know, help demonstrate this technology. We certainly thank both of you. Thank you for taking time out of your busy schedules to help us parse through this subject, which is a complicated one and there's a lot there. So I'm glad that we, we had you both on here to. To help us parse through it. Really appreciate it and hope we can meet sometime.

Yeah, no, absolutely. Thank you for having us.

Thank you so much.

All right, thank you for tuning in to this episode of Practically AI. If you enjoyed this show, do us a favor. Go on itunes, give us a rating. Go in your podcast app and favorite it. If you are on Twitter or social network, share a link with a friend. Whatever you gotta do. Share the show with a friend. If you enjoyed it. And bandwidth for Changelog is provided by fastly. Learn more@fastly.com and we catch our errors before our users do here at Changelog because of rollbar. Check them out at Robot Changelog and we're hosted on Linode cloud servers. Head to lino.com changelog check them out. Support this show. This episode is hosted by Daniel Whitenack and Chris Benson. The music is by Breakmaster Cylinder and you can find more shows just like this@change law.com when you go there, pop in your email address. Get our weekly email keeping you up to date with the news and podcast for developers in your inbox every single week.

Week.

Thanks for tuning in. We'll see you next week.