GANs for Good- A Virtual Expert Panel by DeepLearning.AI

Hi everyone and welcome. My name is Ryan Keenan and I'm the director of Product here@deeplearning AI. We really appreciate you taking some time to join us for this event today and we sincerely hope that you and your families are doing well. This has been one crazy year. We have people joining us today from all over the world. In fact, I believe it's more than 140 countries joining today, which means that you're in pretty much every time zone. So good morning if you're in California like me or good evening, Good day, good middle of the night to you. Whatever it is, where you are, whatever time it is, we really appreciate you joining us, taking some time to join us for this session at DeepLearning AI. Our mission is to make world class AI education accessible to everyone. With that, we aim to empower you, our community of aspiring and current machine learning practitioners, to be part of the AI transformation that's happening across so many industries. Part of that work for us is building new courses. Today we're excited to be launching a new three course specialization focused on Generative Adversarial networks, or GANS for short. GANS are an emergent technology within the space of deep learning and have been used for a wide variety of applications in recent years. Most famously, you might be familiar with how GANS are being used to generate really realistic looking images, whether that's pictures of people who don't exist or other things. But GANS have also been used in applications that you might not think of, like improving cybersecurity or even generating music. And so this success of GANs in so many different applications has also come with some controversy because with this powerful technology, it becomes possible to do things like generate deepfake videos of people saying things they never actually said and other things that make people wonder whether this is a technology that's a good thing or a bad thing. So with these courses that we're putting out today and with this session, we're focusing on how you can use this powerful technology in exciting applications that'll have a positive impact on the world. So for today's event, we have a live panel discussion and we've assembled an amazing group of GANS experts. They'll be discussing some of their current projects, the importance and the future of GANs, and then also providing some practical career advice for those of you who might want to break into the field. So our agenda today will be, first off, keynote speeches by Ian Goodfellow from Apple and Anima Anand Kumar from Nvidia. Following the panel discussion. Following rather will be a panel discussion where Ian and Anima will be joined by Alexei Efros from UC Berkeley and Sharon Joe from Stanford, and of course our very own Andrew Ng from DeepLearning AI. After the panel discussion, we'll do a Q and A session and we've also lined up a course demo with Sharon Jo, the instructor of the courses at the end of this event, so stay tuned for that. For the Q and A session, we'll be selecting questions from the YouTube live chat, so we recommend you post your thoughts there and your questions, and that way, if we see a lot of people asking about the same thing, or if we see something that's particularly relevant to the conversation, then we'll go ahead and surface that to the panel in the Q and A session. And now, without further ado, I'd like to welcome our first speaker, Ian Goodfellow. Ian is the Director of Machine Learning in the Special Projects Group at Apple and previously worked as a research scientist at Google. Bring Basically all of us are here today in this event because back in 2014 Ian published a paper and brought Gans into the world. So it's a great honor and a pleasure to welcome you to this forum today, Ian, and I'll turn it over to you now.

Thanks Ryan and welcome everyone. Thank you for joining. I'm going to kick off the event with a quick summary of how GANS work and I'll also show you how some of my colleagues at Apple have been able to apply them to some exciting applications. Gans are designed to solve the generative modeling problem. Generative models take a collection of training data like these images from the cell of a data set shown on the left here and infer which probability distribution generated that data. Then the model can learn to do one of two things. Either it can learn an explicit density function describing that probability distribution, or it can learn to create new samples from that distribution as shown on the right here, where again has created two faces of people who look like they could be celebrities in an imaginary world. They come from the same probability distribution as the other celebrities, but they never existed before. They were created from scratch by the model. The way that Gans learned to do this is using a game theoretic setup where two different players seek out a Nash equilibrium of a game. One of these players is the discriminator. It looks at real images like the moth on the left and is trained to assign high probability of the moth being real rather than fake. The other player is the generator network. It produces data such as the photo of this cow like monstrosity on the right side of the slide. The discriminator looks at these fake images and attempts to learn to assign very low probability to these images being real. The reason that this is a game rather than an optimization problem is that the generator simultaneously tries to output images that the discriminator will assign high probability of being real. The Nash equilibrium of this game consists of the generator producing perfectly realistic models, images that are indistinguishable from real. So the discriminator will output a probability of 1/2 of everything being real. Over the first four and a half years after my colleagues and I introduced the first GAN paper, institutions and people around the world came up with new ideas to improve the performance of GANs. And in about four and a half years they went from able to produce only small, low resolution grayscale images of faces to being able to produce very high resolution photorealistic faces. Likewise, when I first had the idea for gans, they didn't work very well for producing a wide variety of different objects. But over about two years, starting in 2016, a variety of people came up with ideas that helped GANS go from struggling to produce low resolution images from the ImageNet data set to being able to produce high resolution images of all 1000 ImageNet categories. Now that GANs are working really well, it's interesting to study them and learn how to apply them to different tasks. One task that I've been especially excited about is using GANS in the physical world for medicinal applications. For example, Glidewell Dentistry has been able to use them to make replacement dental crowns. The GAN needs to actually make a crown that fits the shape of the other teeth in the patient's mouth and is able to bite and chew. So it has to be both personalized and functional. Traditionally, it took a human technician about two weeks to do this, so the patient would need to have their tooth prepared for the crown and then go home with a temporary crown for about two weeks and then come back with the permanent crown to be installed. With a GAN, it's possible to 3D print the new cran on site and get everything done in one day. At Apple, some of my colleagues have found other applications of GANs. One of these is GANS for Augmented Reality. In augmented reality, we often want to insert objects into a real scene, such as this teapot inserted into a scene with a table and a notebook. In the upper left, we see what it would look like if we rendered the teapot with reflections showing only things that are available in the incomplete Environment map shown on the lower left. This is the part of the environment that the camera can actually see when the camera is facing into the scene. As you can see in the top of the incomplete environment map, the camera has no way of observing anything like the ceiling, the walls, the lighting of the room. This is where gans come in. We can fill in the missing part of the environment app as shown on the right. This isn't just extrapolating pixels from what we can see. It's actually using our knowledge of the statistical structure of the world to understand things like that the ceiling tends to be relatively plain and that lights tend to be near the ceiling. Given this completed environment map, we're now able to make much better rendered reflections as shown in the upper right. Here's a video of what it looks like when we insert this Instant Pot pressure cooker into a scene. You can see from this angle that the reflection is capturing objects that are actually visible to the camera. But as the camera moves around, you can see that the reflection is pleasant and has an environment that spans the whole room, not just what's visible to the camera. Several other applications of GANs at Apple rely around generating training data for other models. The first published work in this direction won a Best paper award at CVPR 2017. A lot of people have asked me if you can use GANs to make training data for classifiers by training the GAN on a data set and then just producing more examples from that same data set. As far as I know, that doesn't work. You would need the GAN to generalize well and produce new examples that show new patterns in just the same way that the classifier would need to understand those patterns in new input examples. What you can do with GANs instead of just making new data from one source, is to meld two sources of data. In this example, we want to train a model that can tell the direction that an eye is gazing so that, for example, you can highlight an icon on a phone screen just by looking at it. Using unlabeled real images is difficult because it's hard to add a label saying where exactly the eye is looking. Instead, you can use synthetic Images where you3D render eyes such as the one shown on the lower left. The challenge with this is that it's not a particularly realistic Image after you've 3D rendered it. You know where it's looking because you positioned the model yourself, but the appearance is not quite good enough to actually train an eye gaze direction classifier. GANs can be used to learn from the unlabeled real images and make a refiner that upgrades the synthetic image to look more realistic, like the refined image shown on the lower right. So the GAN has melded two sources of data, real photos and their realism, and synthetic photos and their labels, so we can get realistic label data at the end. We also use this same idea in a recent application for QuickPath. QuickPath is a text entry feature for iOS and iPadOS where the user swipes out a series of letters in a single finger strip without lifting their finger. Here you can see three different ways that people swipe out the word anybody with the trajectory starting on the letter A and then turning from green to yellow over time as the trajectory is completed. These trajectories need to be recognized with a neural network because they're hard to recognize with handwritten code. Over the course of the trajectory, the finger visits many keys, but only a few of those keys are actually meant to be included. Most are just being passed over. In order to train this classifier, we can just gather data of real trajectories made by real users, but we can also use a GAN to combine sources of text with that smaller trajectory data set and make new trajectories for new pieces of text. Melding those two together allows us to train the model on a lot more data. One common way of looking at how machine learning models scale with more data is to make a log log plot. Work at Baidu showed that these log log plots often look like a straight line where as you move to the right across the data set size axis describing the number of examples and log scale, we see that the error decreases linearly if we also plot it on a log scale. The blue curve here shows that if we train our quick path trajectory recognition model on real data, we do get one of those linear curves. If we then switch over to training using new data coming from a gan, the error continues to decrease, but with a different slope in the log log plot. From this, we see that a GAN example is worth roughly 1/5 of a real example. Fortunately, it costs less than 1/5 as much to collect because it's easy to continue sampling from the gan. So this becomes a very efficient way to train our trajectory recognition model. Overall, I think these strategies for generating GAN data can be useful for a lot of socially good applications. You can generate a GAN that is differentially private and then it can make fake medical data for users researchers to use without compromising any patient's privacy. GANs can also be used for fairness, accessibility and inclusive personalization by over sampling portions of the data set that correspond to underrepresented groups or groups where it's hard to capture data for that population. I'm looking forward to seeing what the other members of the panel have in mind for how we can use GANS for good. Thank you.

Thanks, Ian. That is some really interesting stuff. I find it cool that you were able to quantify exactly how much GAN generated training data is improving the model compared to more real examples. I think that that's something that I'd kind of wondered about. Like, I'd heard that you can generate more training examples, but like, how good is it or how good are they really compared to being able to get new information, but then being able to say, okay, well, given the cost, this is a much more efficient way to improving the model. Very, very cool stuff.

Yeah. I was really excited to see this paper from my colleagues. Yeah, awesome.

Well, now I'd like to welcome our next speaker, Anima Anand Kumar. Anima is Director of Machine Learning Research at Nvidia. She's also the Bren professor of Computing at Caltech. Today she'll be sharing with us the role of interaction and disentanglement in the training of GANs. Welcome to this session, Anima. It's really a pleasure to have you with us. I'll hand it over to you now.

Hello. Hi everyone. Thanks for having me here. And it's really a pleasure to be part of this event. And thanks, Ian, for giving such a great overview of gant. So today I'll give you some of our recent work, both at Caltech and at Nvidia, on how to understand GANS better. There is complex interactions happening between the generator and discriminator. How can we make more sense of it? And can we design better optimization methods for training GANs? That'll be the first one I'll introduce. And the second part is how to understand disentanglement in GANS better. The example that Ian showed about the dental application, there may be so many different properties you may want to for the teeth figure. So how do you ensure that you can separately control all those different style factors? That's something I'd like to go a bit deeper into. That'll be the two agenda for this talk today. The first one appeared at ICML this year and the question was can we think about the role of implicit regularization in training GANS for standard supervised learning? It's now fairly common knowledge that implicit regularization of stochastic gradient descent is responsible for getting good generalization and for being able to Train models such as the ImageNet dataset. So what is the counterpart for GANS when it comes to the role of optimization method? And this is actually very important because ultimately what we would like the GAN to do is have a discriminator that can really separate the real images from the fake images, right? I mean, those images that don't look realistic, for instance, this data sample here is not within the image distribution. So you want the discriminators to say that this is not a real image. But then the discriminator could potentially cheat because of the Minimax game, that the generator and the discriminator displaying the discriminator could get really good around the training data points. It can just get very peaky here, meaning just around the neighborhood of the training point. It can say this is real or fake, and everywhere else it can just say that this is not a real image. But this is not happening. Because if we are not constraining the discriminator, it can get perfect, but we are going to regularize the discriminator. If you've trained any of the GAN models, you know, to regularize the discriminator, and the best way to do that is to do some penalty on the gradients. So given that we are adding this regularization, we are no longer talking about the original min max optimization between the generator and discriminator. And so for this, we did a very simple exercise. You know, we trained GAN to a good checkpoint, and then we wanted to continue training. So we could do that either by simultaneously updating both the generator and the discriminator. So doing gradient descent on both discriminator and generator, or only update the discriminator as expected, if you only update the discriminator, its loss goes down. So the discriminator can get really perfect. On the other hand, by training simultaneously with the generator, it's not able to get really good. That's what we call this as the implicit competitive regularization. Because this competition between the generator and discriminator simultaneously updating their weights is responsible for it not to lead to pathological solutions. The question is, can we further improve this interaction between the generator and discriminator and lead to good stable training of GANs? That's where we introduce a method known as competitive gradient descent. And the main intuition is can we go beyond just simultaneously each agent updating gradients on their own and introduce a term that models the interaction between the agents? And just to introduce some minimal math here, if you're just doing gradient descent individually on both the discriminator and the generator, that only looks at the gradients, there is no interaction that is being modeled directly between the players. On the other hand, if we now want to ask how to incorporate interaction, we can now look at this Taylor series expansion of the objective function and introduce a term that captures the next order interaction. And that would be the mixed Hessian between the two players. So we now have the Taylor series expansion be a little more accurate. But at the same time, this is still a good structure in the equation because it's only a bilinear interaction in the two players. So it ends up now viewing the game between the generator and discriminator as a local bilinear game. And so once we can make this Taylor's approximation, we can in fact solve this in closed form. And I wouldn't worry about the exact form, just to showcase that what we are now doing is a modification of gradient descent. Right? So in gradient descent, each player would only update their own gradients, but instead we are introducing certain correction terms based on how they're interacting with the other player. And intuitively, what this gives us is more stabilization because you're also looking at, if you're the generator, what the discriminator is going to do, in which direction is the discriminator moving and you'll account for that while making the moves. What we see is, if we now tried this on Gantt, we get state of the art results. And in terms of the inception or FID score, but with no penalty, the intuition is you don't need explicit regularization. And if the implicit regularization is strong enough to accommodate for stabilization between the two agents, that's where introducing the new optimization method can potentially now help us to train more in a more stable way and tune better for training Gans. This is not just for training Gans. We also see that in reinforcement learning environments. For instance, this is a racing game where both the agents have to learn from scratch on how to win the race. The one that is faster wins the race. And if both were just doing gradient updates on their own, you can see that it leads to poor strategies. First of all, there is a big gap between the two competing players, and even the best one isn't that fast as the one where you incorporate the interaction term in terms of the mixed Hessian and design policy gradients based on this interaction. What we see in these examples is that rethinking optimization in terms of the dynamic interactions between the agents, either in training Gans or in this racing game that I showed you can help us lead to better outcomes. We can train now Gans that are more stable during training and lead to better image quality without the need for a lot of other external regularization that would entail additional tuning. This is one aspect of understanding the dynamics of GANs and not just thinking about the equilibrium point of what the GAN represents as a minimax. G. So the other aspect I wanted to cover was understanding disentanglement in GANs and in particular StyleGAN, which is a popular architecture that's developed by Nvidia researchers. And so when I talk about disentanglement, what we really want is different style factors you should be able to individually control. And humans are great at this. We have different concepts we've learned as infants, and we've done that in an unsupervised way. And this way we can now compose and make entirely new images or new concepts. And so disentanglement is the key for compositionality. So if you're looking at the stylegan architecture, the idea is you also now have style factors or these factor codes that give instruction on what property do you want in this image. Right? So like the example that Ian showed with the dental case, you know, what is it that you want functionality in that image to be represented? And we have both the versions where you may generate an image from scratch, or the editing version where you can input an image such as the one of Professor Fei Fei Li, and then you want to control or change the characteristics of that image. It turns out that if you are doing this in a completely unsupervised way, that's challenging. It's very hard to have the different factors, say, of the mouth, eyes and so on, be completely disentangled. What we came up with was a semi supervised approach where we could have a very small number of examples that have the labels that say whether the mouth is open or not. And if you can then combine together and design good losses to encourage disentanglement, then we can make use of this supervised data. And that's the results we see on the CELEBA data set, the same data set that Ian also earlier showed, we can now introduce all kinds of attributes such as banks or glasses, and do that in an effective way with as little as 0.5% of the label data. So of the total training data, if as little as 0.5% is labeled, that is sufficient. And more labels actually didn't improve disentanglement. So it seems like as long as you have a little amount of labeled information, that's enough to encourage good disentanglement and good image quality. We also see that with also any external images that were not in the training data be introduced. And we see that it's able to also do this in the editing mode. But again with training having very little amount of label data. Just to quickly understand the losses that go into encouraging this disentanglement, the idea is now to combine unsupervised and supervised learning together. Right. So what we mean by unsupervised learning is here the real image doesn't have the label. So the only way to encourage disentanglement is on the fake images. So if you now generate a fake image, say with the mouth open, you want to make sure that the encoder in the discriminator can tell that the mouth is open. So you're now checking between the discriminator and the generator that they're consistent with each other. And for those that have the labels, you can just directly check if the reconstruction of the style code is being done at the encoder in the discriminator. So combining these two now leads to effective disentanglement and also good image quality. And that's the idea that sometimes you may want to have a little bit of label data to encourage this disentanglement and you can get significantly better results. So I want to conclude by saying that there's been a lot of exciting work in Gans recently and it's important to think about the underlying dynamic interaction between the agents. And we designed a new method known as competitive gradient decision that can better stabilize this interaction by incorporating the mixed Hessian term that looks at what the other agent is going to do. And this shows that you can even sometimes do away with explicit regularization and still get good results in training Gans and also in other applications such as competitive reinforcement learning, problems with multiple agents. And the other aspect I showed was on looking at disentanglement. So not only generating images, but a controllable generation with various different style factors. And here, if all the images are unlabeled, disentanglement is ill posed and really hard to get at the right answer. But a very small amount of supervision can greatly enhance this disentanglement. Thank you.

Thank you, Anima. Wow, there's a lot there in thinking about competitive gradient descent. It sounds like just an amazing step in the direction of the learning from each other thing. It's not just at the prediction, it's actually through the process of gradient descent as well. That's incredible. Next up, we'd like to continue the conversation by welcoming our other panelists today. These are Alexei EFROS. Professor of Electrical Engineering and computer science from UC Berkeley, Andrew Ng, founder of DeepLearning AI and our moderator for the panel and the instructor of the courses of the GAN specialization that we're releasing today, Sharon Jo. Sharon will be facilitating the panel discussion today. So just a reminder to everyone watching, if you have questions for the speakers or just thoughts about all this stuff, you can post them in the YouTube live chat and we'll be selecting questions from there for the Q and A session after the panel discussion. So Sharon, I'll turn it over to you now.

Thanks so much Ryan. And if what you saw in the keynotes caught your fancy or went over your head, that's what the specialization is here for. And you'll get to learn from the very basics of what is a GAN to the state of the art and Stylegan that ANIMA had shown. So with those fantastic keynotes from Ian and Anima, let's start off by letting the new faces in the room, Alexi and Andrew say say hi to our audience.

Alexi hi.

Well done.

Alexi maybe a sentence about yourself.

Go for it.

Alexi hi, I'm Alexei Afras. I'm a professor in at UC Berkeley doing computer vision and computational photography and I been a huge fan of gans. I I don't fall in love with papers often, but this was definitely love at first sight and so I'm yeah, I'm really excited to be part of this, of this GAN story.

Here.

Thanks to all of you from all 140 countries and all the time zones for joining us. Middle of the night, morning, evening. It's really exciting to be here. I had the pleasure of knowing Ian for a long time. It's great to reconnect with him and Alexei, whom I think I first met when we were both PhD about to enter PhD programs and we ran into each other touring different university campuses and really grateful as well that ANIMA and Sharon are here. GANS of big movements. It's one of those amazing technologies that frankly wasn't working so well six years ago when Ian published his first paper, but now has really taken off thanks to thousands years ago of thanks to the work of really Nvidia and many other groups around the world and if music is poised to find a lot of exciting applications. And I was really struck by Ian and Anima's comments on the concrete use cases where GANs are no longer something that generates cool pictures. You look on the Internet so things that are really useful and are going to people's mouth in the dentist's office. I hope that some of you watching online today will take the GAN specialization and learn about these tools and it go help build amazing things. They'll go make life better for a lot of people.

Thank you Alexi and Andrew. For the next 30 minutes or so I'll be asking some questions to our panelists and while I'll be directing questions to a specific person, I'd also like to encourage any of you panelists to jump in and offer your input in the conversation. So starting with Alexi, can you tell us a bit about what your lab has been working on recently? So students in the specialization will be spending a large chunk of time learning about work from you and your students.

Sure. As I said, I'm a big fan of gans, so there is a lot of things that happen in gans. Thank you Ian for giving a shout out to our work with Gladwell on the dental reconstruction. This was done with my wonderful colleague Stella Yu. We are lately we have been really thinking hard about disentanglement just like anima and because I'm really focused on unsupervised learning we have been looking at disentanglement in an unsupervised setting and we have several papers on that trying to use a Hessian penalty for disentanglement and also thinking about using contrastive learning for kind of disentangling texture from structure. We have a couple of very recent papers. One is on kind of replacing the cycles and Cyclegan with a contrastive learning that was in ECCV and we have a just brand new paper at Neurips called Swapping out Encoders where we are kind of separating the style and the content again in an unsupervised way that we we are pretty excited about. Apart from that I've been working for a long time and self supervised learning again going away from I don't like labels so I'm trying to stay away from labels and I've been also pushing against I also don't like data sets even though I love data but I don't like the fact that they're stationary. And so we have been really focusing on online kind of streaming data learning and so we have a paper on test time training where we're basically like adapt to the streaming data in an online way. And so I think but this is just starting and I think there is a lot of cool directions there.

Yeah, fantastic. I love your contrast of learning paper. And we are also in our lab with Andrew looking at disentanglement so that seems to be a trend here, but.

In a sense I think Gans are doing a lot of the disentanglement. Even if you just take the, you know, just off the shelf style Gan or big Gan, there is a lot of cool disentanglement there already and we just don't quite understand, you know, how much is in there. So I think this is, this is one of the kind of exciting directions now. Yeah.

Yes, very much so. And a huge trend in research right now. Speaking of trends, Andrew, what trends around Gans are you most excited about?

By the way, not sure if I should say this, sadly our submission to Neuros on disentanglement did not get accepted. But that's okay. It happens to happen to everyone. You live and you move on and then you try again. Excited about trends I'm excited about. I think that there's so much basic research on GANS that's still going on and that's great. Clearly technology innovations, new modes still going on the street. What I see is in the tech deep learning world over and over, a pattern you see is there's a new technology, it works amazingly well in the lab and this opens up the door for a lot of exciting creative applications. I think maybe 10, 12 years ago we started to see this supervised deep learning and then we saw a lot of dominoes topple. Deep learning has its first major impact in speech recognition, followed by computer vision, ImageNet and then I saw same transformation in NLP and so on and so on. Again, pattern matches to me to one of those, those technologies and now it works so well. You look at pictures, you go, wow, this is so cool. But I'm really excited about exploring all the ways to take these and put them into useful applications, which is actually really difficult. The whole AI world is not great, frankly, at bridging the research, the proof of concept to the production gap. But when Ian talked about the gaze tracking, you're mapping one eye, the computer graphics eye to another one, it turns out that in order to apply that broadly across many industries and many problems, I think there'll be a lot of important stuff to solve. What happens if the mapping doesn't work? How do you make sure you don't have mode collapse or training problems or all of those issues, not just when Ian is the one at the keyboard working at making it work well, but how to make it systematic so that many teams, hundreds of thousands of teams across the industry can make it work. And I'm also excited about the creativity that Gans could unleash. Everything from Photoshop 2.0 to all the ways we have to manipulate images. I hadn't realized Apple had such cool work on AR using Gans, but I think all the creativity, we need more people to understand these algorithms so that they can be the creative ones come with the cooling applications.

Thanks, Andrew. And I think Anima also has maybe shares that perspective since she does spend her time between Nvidia and Caltech. Anima, what are some of the unique challenges of applying Gans to business that you see even if you're at Nvidia and have a seemingly infinite supply of GPUs?

Well, you know, it's really exciting to be working at Nvidia, but like everyone else, of course, you know, we want to make a good case for why we use GPUs. Right. And use it for good benefits to humanity. And that's where I'm very excited by my colleagues at Nvidia who've been pushing so hard and making Gans photorealistic get to these really high quality images that have now passed the Turing test, whether it's Stylegan. Gogan has, I think now maybe even up to a million downloads where you can turn into an artist and you can, you know, give like rough, broad brushes of the landscape you want to draw and then it turns it into a really impressive image and then also moving more into the 3D world, you know, how to do generation there, what are challenges and how to do it at scale. That's where there have been a lot of great researchers at Nvidia looking into these cutting edge topics in generative models. So more on the Caltech side, what I'm excited is looking at interdisciplinary research and especially talking to neuroscientists such as Doris Sow and looking at how we can get inspiration from how our brain does this. Does the brain have a generative model? Apparently that question is still open and unanswered. But we do have some form of feedback. We do have some form of representations of the the world around us. When we are seeing and perceiving, we are hallucinating to get to the actual image. We are not just taking an external input from the world, we are also building based on our internal representations and doing it as a feedback. Some of the recent work we've been looking at is how to add this feedback to any standard feed forward neural network architecture and, and that can be a lot more robust to all kinds of corruptions that it hasn't seen during training. Because that's the other important aspect to bring it to the real world that these models have to be robust. The current brittleness that even if you have a small amount of imperceptible noise, it's completely going to throw off your predictions is not one that can be brought into the real world. And so looking at how we can get inspired from the mechanisms in our brain and, and trying to bring some of that into neural network architectures like feedback is something that's been very exciting.

Very cool and very salient. Going off of kind of the utility of GANs. Ian, can you expand a bit more on some of the topics you mentioned on using GANs for good from your keynote, particularly pertaining to perhaps differential privacy or combating bias?

Yeah, and a lot of these areas that have only had little nibbles on them in the research literature that I'm hoping people from this event on to explore in more depth. One of them is differential privacy. The idea behind differential privacy is you can train a machine learning model in a way where it doesn't memorize the individual characteristics of individual examples in the training set. So if that model you're training is again, it can then make you new data without revealing anything about the real data that went into it at the start. That's super powerful for medicine. We've seen a proof of concept from Casey Green's lab where they actually trained GANs on medical data and then they're able to make new fake medical data that can be released publicly and effectively an infinite amount of it. Especially because it's really hard to pool data from different clinics because of things like HIPAA considerations, things like differentially private generative models. Seems like a really good way to get over the data scarcity problem in medicine for a lot of other topics. There's reasons why you might want to generate more data for a given area with a gan. If you want to, for example, support a language that isn't spoken very commonly, it might help to generate more data for that language. Or there's a startup called View AI that allows people to visualize themselves in clothes available from a retailer. Traditionally, you'd have to rely on the retailer having hired a model who looks kind of like you. And now you can use the GAN from View AI to generate somebody who's your race, your skin color, your hair color, your body shape. It makes the whole model photography process a lot more inclusive in that sense. So I think there's a lot of different things that you can do. Those are just a few things starting to scratch at the surface.

Thanks. I think ANIMA had touched on this a little bit before in terms of how GANs. There's still a gap between GANs in the real world and decision making environments versus understanding where they do learn and are effective and where they might fail. Ian, did you want to touch a bit about this?

Yeah. It's also pretty similar to what Andrew said about how supervised deep learning went from the lab to the real world. The state of GANS today kind of reminds me of the state of supervised deep learning, maybe like circa 2012, that it used to really take a wizard to train a deep learning system. And to some extent today GANs are still like that. Now deep learning is considered relatively reliable and it's because we found all these nice recipes like always using ReLU's, always using Momentum, maybe having a few technologies that didn't radically change the paradigm, but made it so much more reliable, like Atom and resnets. I'm hoping that we get those kinds of reliability technologies that help us to apply GANs in lots of applications without needing a GAN wizard.

Yeah.

One of the interesting things to me, just really going off Ian's comment, one of these things to me, is that a problem, if I may, about the whole GAN world is we don't have very good evidence evaluation metrics and so we can generate it either. Go, ooh, this looks great. In fact, one of the pieces of work that I found really interesting was some of Sharon's work on hype, showing how challenging it is and how problematic it is to use some of the automated evaluation metrics. And I think this contributes to the need for GANTT wizards because the right wizard looks at it and their eyeball says, oh, I got to, you know, it's clearly mode collapse. I'm going to do this or something. Actually, Sharon, don't mean to put you on the spot. I know you're the moderator, but since you're a world expert on this, do you want to, do you want to say anything about this problem?

I think evaluation is a big problem in GANs and you'll get to learn about it in the specialization. But because it is a problem, I think it very much depends on your downstream task and what you want to use your GAN for. So your GAN can help your other AI models, which is cool, but then you can evaluate your GAN based on how much it does help your downstream AI model, like classification, segmentation and the like. Or it could be around realism. And then I think it's really important that we have humans in the loop. Who we humans are the gold standard perhaps of evaluating realism.

And in terms of democratizing GANs, which Ian was talking about. I think some of the really interesting work was the work by the Nvidia group, open sourcing stuff so the rest of us could use it. So I've gone to the Nvidia website and looked at maybe anima.

Yeah.

And I'm happy to announce, just a few days ago we announced the release of Imagineer. So that's all the very sophisticated and cutting edge GAN models. We've now put them all in one place. Mingyu's group has worked hard to have a hackathon for the past six months and get all that in great shape so that everyone can use it. So we'd love for more people to check this out, through this, you know, throughout this course and maybe adopt some of those models.

Very cool. Thank you. I'm really excited about that and was following the tweet on that pretty closely. Maybe switching gears and thinking about our audience members a bit and thinking about what they can do to prepare to be a good GAN researcher or practitioner or student in general. Alexei, maybe to start, what makes a good AI researcher in your lab and what characteristics do you see in your students that you like?

Well, I mean, my lab is. I'm pretty happy to say that the main characteristic of my students and me also is we're all a little bit crazy. I think it's kind of important to be a little bit crazy if you want to do research. And I think really it's the same things that, you know, is any discipline you want to, you know, you want to be imaginative, you don't want to focus on short term, you don't want to have, you know, focus on getting papers out every, you know, three weeks. I think it's all about having some kind of thing that you really, really want to do and just trying to go there. What I tell my students is don't stress out about all these papers. The faucet is open on Arxiv and there's so many papers coming out every day. There's so many papers being published. There is this idea in, in medicine, when they graduate from law school, from medical school, they tell you there is a concept of half life of knowledge. So they say, okay, remember, five years from now, half of what we have taught you in medical school will turn out to be false. Right? Just because science moves forward. And I think in ML and in GANs, that number is maybe three months. Half life of knowledge in GANs is about three months. So I would not worry about every single paper, every single thing, every Single trick, you know, it's fine. You can wait a little bit and see if it actually works. If it sees, if it takes off. I don't read every paper if like for example, you know, if the paper only has faces in it, I don't read it because we know that faces are easy. Faces worked 20 years ago using active appearance models. So if they need to try something harder or if they just have MNIST or something cifar, I wait until they try it on harder data sets. So that kind of cuts out a lot of the chatter. So I would just not worry and not be stressed out because if you have a good idea, just go for it. It will come out.

Definitely. And I will say I have been scooped before. I think I was thinking about doing a semi supervised GAN at some point looking to Nvidia. But Anima, as a fellow woman in AI, I and others definitely look up to you for advice and inspiration. And it's without a doubt that you've lended incredible support to your students and your role model to a lot of women looking to go into AI. What advice would you give to the women and girls tuning in and who are learning AI?

Thank you, Sharon. And yeah, I'd love to see more diversity and inclusion in our field. I think there's been incredible awareness in the last few years and you know, the majority has been supportive and positive. Right. So despite all the trolling and some of the negativity, I think the positivity overwhelms at the end of the day. And that's why I continue to speak out and I continue to make sure that we create a healthy environment for everyone in the community. And so what I would say is to keep fighting. If you are seeing a problem, whether it's a technical problem or it's a societal problem, it's something you care about. You may not see the immediate returns. You have to not only fight for it, also find the allies, find your support network and also learn how to communicate well. Because whether it's a research aspect or everything else, we want to see change in the community. I think at the end of the day we need that support network and we need to make sure that we create the awareness.

Definitely. And on the business side, for those tuning in on the business side, as a successful CEO, Founder and professor Andrew, what's your secret in managing your time and keeping yourself updated on AI? And what kind of advice would you give to folks tuning in from the business side? Should they take this specialization?

Yeah, actually, I'll answer just a shout out to ANIMA for really consistently speaking up on social matters at issue that matter. I feel like we live in a world where there are lots of ups and downs. And I guess watching the US Presidential debate just last night, I think it's more important than ever that when all of us have a strong opinion on something, right or wrong, that we speak up. Because I think every voice matters. I think ANIMA has been a consistent voice in that, and I think that's great. And I think Sharon released an app on the Internet to help anonymize protesters, you know, to protect them. I think it's wonderful that Sharon just woke up and said, this is social. Cause I'm a deep learning researcher. I want to build this because I can make the world a better place. I think every one of you watching this online, your voice matters. Don't ever think that. You know, individually our voices are limited, but collectively, we, the AI community, are incredibly powerful. But only if all of us speak up about the things that matter so that we can shift the whole world to doing more of the things we want that make people better off. And so to answer Sharon's question, so much is happening, really honestly, one of my honest sources of knowledge is the badge which Deepen AI publishes. There's a large team of writers that are Editor in Chief, Ted greenworld organizers to try to synthesize the most important AI news to share in a very succinct form with everyone. The issue that's going out later today of the vatch is a GAN special issue. So covers a bunch of cool research on gan, some of which I actually honestly did not know about myself until Ted and his team found those stories and synthesized them. So I actually used that. And then I think fortunate also to have friends. To be completely honest, my personal number one source of knowledge on GANS is actually Sharon. Sharon tells me when she sees something cool, so that's fun. But I find that having friends you can talk to, to read papers together, to brainstorm together, it has been a very important part of how I keep up to date. And so I encourage others, you know, to form a community. None of us should have to do this alone. So make friends, form a community and work with them because we're all really much better and much stronger together.

Thank you. That's powerful. The newsletter, the Batch, actually has an interview with Ian as well in it, which I hope you all check out. And I think this would be a question for all of you on how do you stay up to date on the latest machine learning Research, maybe starting with Ian.

Yeah. At Apple, we actually have an event called Paper Party where once a week we have people get together and share a paper. And I think that's really powerful because only the person presenting it needs to have read it. It's hard to get everyone in a discussion group to read a paper, but one person can explain it succinctly to other people. And then beyond that, just like how Andrew relies on you for updates about Gans. I have friends in different subject matter areas that keep me up to date. I actually read very few papers. I mostly discuss them with other people and that's. And through events like Paper Party, that's how I stay up to date.

Great. I know my favorite kind of paper group was definitely one where there were no expectations going in to have read a paper. We just read while you're there together, which is nice. Anima.

How do you stay up to date? Yeah, I mean, some of the reading groups are great, especially if they are involving people. Right. In a broader community or coming from different areas. So we can look at different papers, not just in a narrow topic, but mostly, again, I don't read papers most of the time. I also don't encourage my students to read papers just on archive. So we almost like in a. Most of the time it's a thought process, right. We are saying, why is this, say, problem hard? And what could be a new thing that can be done? Right? What is the barrier now? And then maybe look back and see if others have looked at similar ideas. So I encourage them to first come up with ideas on their own and then maybe check back and see if that's already been done. And if somebody has done it, great, that means they were thinking in the right direction. If not, then there are new topics. So I think encouraging more creativity and original thought is really important in this mad rush of papers we see on our curve.

That's very true. When I saw Semi Stylegan, I was like, I'm so glad Nvidia did this. I would never be able to do something that awesome. Alexi.

I. I also don't read as many papers as I would like to. And I rely on my students really to tell me what's cool and what's working. I like to read older papers. I feel a little bit bad that we are reading less papers, especially because I put so much blood, sweat and tears into my papers. You know, it's kind of a. Every, every paper is like the introduction. I try to make it all very, you know, Tolstoy, but it's kind of. It's harder and harder because I know that less and less people read them. So I think it's good to read papers. I would just encourage you to use time as a filter. Read older papers. The older papers are still good. In fact, you will find a lot of really cool things in older papers. And so just wait until you realize that this is actually something that's really working and then go back and read the papers. That's my advice.

And to those older papers, the ones that have stood the test of time, or perhaps your newer ones, did you want to share any of your work in art using Gans to the audience?

Well, actually, I do want to mention about art, and it's something that's really, kind of connects with people talking about the different, different things that Gantts are used for. For us, it was really kind of amazing. So we came out with pics to pix, and then within, you know, we put the date, we put the code online, and then within like a couple of weeks, a friend of Philla's, Christopher Hayes, created this very cute little online app to create, you know, cats from edges. And then what kind of started really quickly is that people just started doing crazy things with this app and doing things that none of us really expected. So this is kind of different things that people have gone online and people started to do it in faces. And we realized that there was some kind of a creative energy. And even Jan did one for us here. And then my wonderful colleague Aaron Herzman from Adobe, who is really kind of interested in this kind of connection of art and technology, he pointed me out to just an avalanche of absolutely amazing artists who are using Gans to improve their. To create art. And so Mario Klingbin is one of the first ones that really, I mean, this is something that to me feels like this is we are actually doing something good here by giving the artists a new tool, a new brush. In fact, this is an artist I met, Helena Sarin, who, you know, she's a software engineer and she's also an artist. And then when she learned about Gans, she was like, wow, I can connect those two things together. And I like what she says. She says, my watercolor teacher used to say that let the medium do it through that say, my sketch provides a foundation, and then the network does its thing. I don't fight. I constantly tweak the Ken brush, right? So it's kind of the idea of Gan, not as computers creating art, but as a brush to unleash the creativity of artists. And this is another wonderful artist, Scott Eaton, that I. I was happy to visit. And what he does is he basically he sketches things and then he essentially runs pix2 pics using some of the data that he has photographed himself. In his studio to create these kind of impossible sculptures. And it's kind of amazing how well the GAN is able to kind of capture the shading, the consistency of tone, and. And yet it is very much something different and something weird. And so finally, I wanted to kind of shout out to this wonderful website by Joel Simon, who is called artbreeder. And it's a wonderful thing to try to unleash your creativity with very simple set of kind of rules where you basically, you're just kind of. You're working in the GAN space. I think they use big gan and they kind of combine different latent vectors together. And the power of it is that it's a community effort, it's shareable, and some really, really impressive and amazing works come out of this. So this is just, I think, something that is kind of, to me at least, it was unexpected. I think maybe Ian would also not have expected this, that there's the whole community of artists kind of jumping on this and doing things that we technology folks would never think of. And I think that this is just wonderful.

Absolutely.

I really like Helena Sarin's book Using Gans and Illustrations Using Gans as well as Art Breeder. I love artbreeder. You can actually.