GANs, RL, and transfer learning oh my!

Bandwidth for Changelog is provided by Fastly. Learn more@fastly.com we move fast and fix things here at Changelog because of Rollbar. Check them out@rollbar.com and we're hosted on Linode Cloud Servers. Head to linode.com changelog this episode is brought to you by DigitalOcean, the simplest cloud platform out there and we're excited to share. They now offer dedicated virtual droplets and unlike standard droplets which use shared virtual CPU threads, their two performance plans general purp CPU optimized They have dedicated virtual CPU threads. This translates to higher performance and increased consistency during CPU intensive processes. So if you have build boxes, cicd, video encoding, machine learning, ad serving, game servers, databases, batch processing, data mining, application servers or active front end web servers that need to be full duty CPU all day every day, then check out Digital Ocean's dedicated virtual CPU droplets. Pricing is very competitive starting at 40 bucks a month. Learn more and get started for free with a 100 of credit at.co changelog again.co changelog. Welcome to Practical AI, a weekly podcast about making artificial intelligence practical, productive and accessible to everyone. This is where conversations around AI, machine learning and data science happen. Join the community and slack with us around various topics of the show@changelog.com community. Follow us on Twitter. We're PracticalAI FM and now onto the show.

Welcome to another Fully Connected episode where Daniel and I will keep you fully connected with everything that's happening in the AI community. We'll take some time to discuss the latest AI news and we'll dig into learning resources to help you level up your machine learning game. My name is Chris Benson. I am the Chief Strategist for Artificial intelligence, High performance Computing and AI Ethics at Lockheed Martin and with me is my co host Daniel Whitenack, who is a data scientist at SIL International. How's it going today Daniel?

It's going pretty good. It's been a good week. How about with you Chris?

It's been good, just the usual busy stuff. I am excited about today's episode. We were talking about it prior to getting on air and you had some great ideas. You want to go ahead and talk about why we're doing what we're doing?

Yeah, sure. So some of the listeners might know that I do industry trainings in AI and other things for companies and sometimes at conferences and that sort of thing. And one of the frequent questions that comes up during those trainings and just kind of conversations about AI in general are questions about kind of the difference between an AI model So you might think about having like a, a convolutional neural net, you know, model or image detection model, like something like that. Like a model. Is that the same or different from like, things like reinforcement learning? GANs, transfer learning are like GANs and reinforcement learning and transfer learning, are those like, types of models like you have a transfer learning model, or are they like different sorts of things than models that, you know, like then specific architectures that might be associated with like, specific neural units, like recurrent neural networks or convolutional neural networks. So my thought today is that maybe we could just kind of go through a few of these, like, methodologies or approaches, maybe we can call them that aren't maybe like models themselves, but are connected to the AI world in some way. So I'm suggesting that maybe we kind of talk through what reinforcement learning is, what GANs are, and what transfer learning is. How does that sound?

That was, that's a great idea as far as I'm concerned. And it's the kind of thing that, that we hear in feedback a lot, because as people come into the podcast, they're coming, you know, some people are already experts in the field. A lot of people coming in are brand new, and they're trying to understand what the field is about. And there's so much to learn these days, and it's evolving so, so rapidly that I thought this was a great idea to just kind of go through each of these and just identify or define what each one is and basically how it works and allow people to kind of get up to speed on those a little bit faster.

Yeah. And as we go through these, we can maybe just give a little bit of a sense of what they are, but also give some places that they've shown up in the news recently and in the AI news or news in general, and then some learning resources for each one. If you want to kind of get started in reinforcement learning or one of these other things. Some, some, some links that we'll for sure put in our show notes so that you can follow up on those things and start learning them practically. Remember, we're all about practicality here at, at this podcast, so I want to make sure and include those links as well. So which, which one of these do you want to get started with, Chris?

You want to dive into reinforcement learning up front?

Sure. So again, we're kind of thinking about approaches or methodologies that involve AI models, but might be kind of slightly different than a single one, end to end model. So with reinforcement learning, where have you Seen, first of all, where have you seen reinforcement learning being applied, Chris?

I think the thing that made reinforcement learning big was its application to simulation and to robotics. And it's, it's, you know, reinforcement learning has been kind of a core technique for doing simulation robotics for a long time. And then in recent years, deep reinforcement learning, which we'll dive into in a few minutes here, has really come in and revolutionized that process itself. But in real life, at a previous employer, I, we were working on a large multi skilled team with different people specializing, and we had reinforcement learning specialists on the team that were focused on doing robotics. So I mean, it's definitely real world stuff. It's not just academic and, and it works, and that's why they're doing that.

Cool. Yeah. So you mentioned that there's kind of this thing reinforcement learning and then there's like deep reinforcement learning, and that's kind of where part of this AI or neural network stuff gets plugged in. But in reinforcement learning, one of the main pieces of reinforcement learning is this thing called an agent, and this agent takes action. So in your case, Chris, with the robots, do you remember kind of what this sort of agent or these actions were that the agent was taking?

Sure. So you'll have different software components within the robot, and they may be integrated with different types of models, and they each have a particular job. And for simplicity's sake, let's just say it's about moving the robot around its environment. And so initially you have to have an algorithm where that the agent is going to use to make decisions based on what's happening to it in the environment that it's operating in. And the way you do that is every time the agent actually takes an action that changes the relationship it has within the environment, which is called state. And so that might have been, it could have been a good action that's kind of going toward what you're trying to train it for, or maybe not such a good action. And the way that is determined by the person that's trained the model is by offering a reward for, for the appropriate actions being taken. So you're essentially, you could kind of think of it as, since, you know, we're always talking about pets and stuff like that. Treating a dog for doing the right thing with positive reinforcement training for those of us who have pets, same kind of ideas. You want to let the agent know, hey, that was good, you get bonus points for this, you know, for doing the right thing, or we're going to pull back something if you don't and so that's kind of the basic idea. You go through that iteration many, many times to try to get your robot or your simulation, could be a video game, could be whatever, to start behaving in the way that it has been most rewarded along the way.

All right, cool. So let me. Let me try to. I'm kind of trying to parse through in my mind some of the things that you said, which was really good. So there's this first thing that's called an agent, and that agent can take action. So let's say in a very simple scenario with a robot, maybe the robot can only do two things. It can move left or it can move right. So that agent has to determine if based on some external factors. So the environment and its current state, or maybe where it's at or which way it's facing, if it's to move left or if it's to move right. So it takes in some inputs from that environment, and it's supposed to determine if it moves right or left. Now, I think what this. So the agent employs what's called a policy to determine that next action. So let's say that the robot is in this place with these coordinates, and. And maybe there's other external factors or something. And so it's got a current state. It's somewhere in its environment. And that policy is to determine that next action based on the current state, whether it's maybe move left, move right, do this, do that. So, yeah, I think that there's. So I think so far, I'm kind of trying to count up the things that reinforcement learning involves, in my mind. And we've got the agent, we've got the policy, we've got the state, and then we've got the environment. Now, you mentioned the reward. So the reward also is kind of how the model gets feedback. Is that right?

Yeah, it's the feedback loop. And the purpose of the reward is to shape the policy. So your policy is being evolved so that at the end of your training, the policy hopefully always does the right thing that you're training towards. And you're essentially giving it little bumps with the reward to get it there. And so you're trying to shape the policy, which is the strategy that the robot is using to move around. And so there are many different ways of doing that. There are lots of different algorithms that have been used over the years. And, you know, in one of those, which we'll talk about, has moved into what's called deep reinforcement learning.

Yeah. So in my mind, if I'M thinking about this. I kind of see this loop where the agent takes actions, and then at some point in the feedback loop, the environment or so the environment infuses a reward or feedback into the agent. Now, one of the things that people ask me kind of is when they're trying to figure out this reinforcement learning thing, they kind of get the idea of, okay, you can give a dog treats and kind of help train it. So this idea of training kind of makes sense, but they have a hard time picturing kind of where the neural network fits in this scenario or where the model fits in this scenario. So one example of that might be, like, if our robot has a camera, right, and it's looking at its environment, or maybe it's looking at a simulation, one thing it could do is, like, image recognition, right? And then based on that image recognition, it could determine whether to move left or move right or something like that. So instead of, like, the image coming in and then the output just being like this object is in this image or not, then in this scenario, the image would come into the model, and. And the output of the model would be like, the action, like left or right or something, something like that. So there's still this kind of neural network model there, but it's tied into this feedback loop where the output is the action. Am I representing that correctly, Chris?

I think that's a very good explanation. So a lot of times, what the reinforcement learning is acting on may be maybe a camera, the camera images coming in. And so the type is a little side note. The type of neural network that is most often used for that is called a convolutional neural network or a cnn. And we've had several episodes where we've talked about that, including one that was a deep dive on the technology early on. And so typically, when we've talked about those, we'll. We'll talk about the convolutional neural network, basically classifying what it sees, essentially putting a label on it with a percentage of, you know, I'm looking at something, is that a horse? Is it a cow, is it a dog? And there's some level of a percentage of confidence that is being assigned to those. Traditionally, with CNNs, the difference when you put it in with this particular approach with reinforcement learning is you're talking about influencing the policy. So what you really need is the output of that convolutional neural network is, what action should I take for my next action? And that way, it feeds into how the reinforcement learning algorithm is trying to do that reward to change the policy over time on how the model is acting on the environment.

That's a great point. So you mentioned the convolutional neural network, but people could see that this reinforcement learning algorithm or approach is really just that it's an algorithm or approach where within that approach, you could apply a convolutional neural network in your agent to kind of learn a certain policy to take in images and output actions. But people use reinforcement learning for all sorts of other things. And that approach is kind of independent of the specific kind of model that comes in. So you could perfectly well use other architectures of neural networks, recurrent and other things within your agent. But this reinforcement learning loop or approach would kind of still be there. That would still be kind of an RL approach to maybe a different sort of problem.

Yeah, and you raise a really great point there. And we've kind of alluded to it several times, and that is, and just to kind of back out of the specific RL reinforcement learning focus conversation, we're talking about different approaches that have different algorithms or architectures. And you know, when you set aside all these buzzwords, they are there. Each one is trying to solve a particular class of problem. Whether it's, whether, you know, we were talking about CNN's looking at images and, and trying to solve that and reinforcement learning being able to have an agent take actions that are rewarded to get to the right policy to, to act in your environment. We've talked about several others. And the point is you can use a lot of these together. And so to avoid confusion is if you're working on a particular problem and you might be in reinforcement learning and say, well, it's, it's images that I'm acting that I need to act on. In this case, you would stick a CNN there. And, and that is just one possibility of, of how you would combine different types of, of architectures or algorithms in deep learning to get where you want to go. So it's not always the case that one architecture, one algorithm gets you where you want to go. You may, a little bit like Legos, you may plug some of these together. I just wanted to clarify that in case there was any confusion.

Yeah, I appreciate that. So maybe before we move on to the next thing, let's maybe just think about, okay, where, where is reinforcement learning showing up in, in kind of AI news? And what are some learning resources that people can, can look into if they're trying to learn reinforcement learning, if this has piqued their interest. So one of the Things that I've seen in the news recently, so very recently are, you know, and people have probably seen related things on Twitter or wherever is DeepMind released a reinforcement learning approach to that resulted in human level performance in a video game, Quake 3. And so this is pretty cool where, you know, a lot of these reinforcement learning techniques have been applied to kind of fun things like, like video games and, and things like that if you're, if you're more interested in reinforcement learning. We've actually had a couple episodes, so episode 14 and episode 40 of Practical AI that talk about certain applications of reinforcement learning with a little bit more explanation. And as well, one of the learning resources that I found that looked really good on this front is there's actually an official Pytorch tutorial on reinforcement learning and we'll make sure and link that in our show notes if you want to go ahead and dig a little bit deeper into reinforcement learning and actually try some things out on your own.

Foreign.

This episode is brought to you by Discover Bot. Learn everything there is to know about bots at Discover Bot. Practical AI Discover Bot was built by Amazon Registry Services as an online community for bot creators and makers of all skill levels to learn from one another, to share stories. And they regularly publish guides and resources to answer questions like how to set up payments to your bot, how to stop shopping cart abandonment, what KPIs are worth measuring, how to write an engaging chatbot dialogue. You can even register bot domains there. Learn more and explore this huge library of bot resources at Discover Bot practicalai. Again, Discover Bot practicalai.

So, Daniel, now that we've covered reinforcement learning, what do you say we dive into gan Generative Adversarial Network.

Yeah, that sounds, sounds good. So this sounds kind of, kind of scary. Adversarial things.

Chris, are you going to talk about terminators now? Are we about to all die?

Yeah.

Is it?

Yeah. I don't know. How adversarial are these networks, Chris?

Well, I'd say that they're adversarial with each other, which is the whole point, which is why they're called calling it that. It is a, it is a really, really interesting innovation that came about in 2014 where a.1 of the, the very famous figures in this space, whose name is Ian Goodfellow, was put out a research paper with several of his colleagues and about generative adversarial networks. And what that is is basically you have two different types of neural network architectures designed to work together or more specifically against each other to try to get where you want to go. It's a way of often creating outputs that are creative from this type of technology. Pretty interesting stuff. I've seen it used for lots of different use cases, too.

And I. Yeah, you mentioned kind of the creative element of this. One of the places I think this has received a lot of attention for good and bad, in some ways, is in generating images. So there's kind of examples of creative uses like generating specific artwork or generating things in the style of certain other things. There's also examples of generating kind of pictures of fake people and all of these things. So this all involves this kind of generative element of GANS or generative adversarial networks. So you mentioned that there's kind of two elements of this methodology, Chris. So there's obviously some sort of generative element of this, which is what people call the generator of this approach. What's the other thing that's involved here?

So you have the generator, and then you have. Which is the one part of this combined architecture. And then the other side, the other algorithm is the discriminator. And so essentially the generative architecture or model in this case that's being trained is creating outputs that are inputs for the discriminator. And on the discriminator side, it's essentially trying to classify which ones are real and which ones are fake. And it has those mixed in with the ground truth data set. So that if you're trying to create images and you're. And this might be something that's completely new, the discriminator has access to a data set that has a bunch of real images that are the ground truth that you're training against. It is that baseline data set. And the generator is also looking at those. But it's creating images that are meant to look like. Like whatever it is that the. That the data set is representing. And so it might be. I'm just making this up. Might be cats, since we like to talk about cats on the Internet. And so you might have a bunch of images in the actual data set of cats. And then the generator is trying to create new images of cats and slide that in with the ground truth data sets. And it's up to the discriminator to determine which ones are real and which ones are not and put a percentage on that. And so there's this feedback loop between the two to where the discriminator is making its choices and giving that feedback to the generator. And in turn, the generator is learning from what the discriminator is able to do, right or wrong. And produce more and better images. So it's a neat thing where the adversarial side is that these two models are literally trying to beat each other. One analogy could be a policeman against a counterfeiter with the generator being the counterfeiter and the discriminator being the policeman. And they're each trying to do their thing and get better and better at it. And by doing that, they both get better.

Yeah, I've also heard the analogy kind of being the generator is the artist and the discriminator is the art critic trying to, trying to examine the output of. Output of the generator. So similar to, so in some ways similar to reinforcement learning. There's this kind of overall scaffolding in which in this case two models are, are interacting. So there's, there's, there's more going on here than just kind of one end to end model. There's, there's a couple things happening here and there's this loop between the generator and the discriminator. Now, each of these pieces, so the generator itself and the discriminator, each of them could be a single neural network. So the generator might be a neural network that takes in, for example, some random inputs and generates an image on the output, like a art image or something like that. So its input might be, you know, some, some kind of random input like that, and the output might be what you're trying to generate. The discriminator, on the other hand, it's taking in a whole bunch of images and it's kind of like a classifier. So it may just be another type of neural network that is trained to be a classifier to classify as like, you know, human generated or, or computer generated or good art or bad art or something like that. So it's a classifier that classifies that set of images. So you kind of got two quote, unquote models here, and that's where the neural networks are fitting in here. Of course, there's specific types of generative models that work particularly well in this framework. For the image case, DCGAN is fairly popular. There's a OpenAI article that we'll for sure link in our show Notes that kind of describes some of the generative models that are used in GANs. But maybe as we kind of look a little bit more at gans, maybe we can talk about where they've been showing up in the news. So where have you been seeing GAN show up recently in AI News or news in general? Chris?

Well, one of the things that we have talked about on a couple of previous episodes was that there was a portrait that was created by A. Gan that Christie's auction house sold at auction for $432,000. And it really, nobody, including the people selling it, were expecting that. And that was, it was for a unique, an original piece of artwork that again created. But you know, and that really suddenly it really shook that industry, you know, because it was, it was one of those instances that nobody saw coming. But we're also seeing it in all sorts of other places creating original music. We've talked about that in the past. It's. I know that Ian Goodfellow uses it in the security industry, which is completely different. And so there are, there are so many different use cases where you want some sense of originality or creativity to, to play into it. And using Gans to actually generate the stuff is, regardless of what the medium is, is, is becoming a better and better option for doing that.

Yeah, I know one thing that I've seen that kind of even people that have. So like my, my brother in law who isn't involved in the, in the AI industry at all. I mean he's kind of interested in tech things, but not really a programmer or anything like that. He even showed me this one website. So people are probably familiar with this, that it's thispersondoesnotexist.com have you seen this, Chris?

I have. And it's gotten better and better over time.

Yeah, it has gotten better over time. And this website, if you're not familiar with it, you can go there and basically all it shows you is a picture of a person and it looks exactly like a real person. So it's kind of, it takes you off guard when you realize that this person does not exist. In other words, this picture of this person, which looks real in every way, is a picture of someone that is completely generated. So everything about that picture is generated using this type of methodology. And of course that's really interesting and kind of amusing in certain ways, but also it's kind of concerning in other ways. Of course, everyone is concerned with fake news and fake content on the Internet now. So there's definitely a concern with these around if what you're looking at is actually real or not. I remember talking on one episode, I forget which one about. You know, there's actually people out there that will create a fake Persona, fake picture for you, for Instagram to be kind of your company's influencer, you know, on, on the Internet. So there's a, there's a question here of like, you know, how real are the Things that we're interacting with.

Yeah. So it's interesting. One of the responsibilities I've taken on at Lockheed Martin is, is contributing to developing AI ethics and figure out, you know, not just about what we do, but about how we react to what's happening in the world. And there are obviously bad actors out there. And so one of the things, you know, Gans are so powerful and as a quote, Facebook's AI research director is very well known in the industry, Yann Lecun. And he, he referred to Gans famously as the most interesting idea in the last 10 years in machine learning. And obviously Ian Goodfellow and his partners that were working on this are among the brightest minds in the field. So there's so much potential for the use of GANs, both wonderful, interesting, and some bad use cases as well. So the advent of Gans has changed the conversation in terms of AI safety and AI ethics and, and how these technologies can and should be used.

Yep. So if, if people are interested in diving a little bit more into Gans, there's definitely some good resources out there. So for, for reinforcement learning, we, we mentioned there's a Pytorch tutorial and there's a bunch of other tutorials out there for that, but there's a really great TensorFlow tutorial for GANs, so we'll make sure and link that in the, in the show notes. Actually, if you, if you go to that tutorial, they have some nice pictures as well, talking about the generator and the discriminator and cat images and all of those good things. But then they walk you through all of the code with Keras and TensorFlow to actually create the scan and they have a link to kind of pop that up in a Google Colab notebook so that you can go ahead and get started. Started with Gans.

Well, hello there listeners of Practically I. How are you? This is Adam Stakoviak. If you haven't heard yet, we're launching a new show called Brain Science. It's a podcast for the curious. Are you curious? Because if so, we're exploring the inner workings of the human brain to understand things like behavior change, habit formation, mental health, and what it means to be human. It's brain Science applied not just how does the brain work, but how do we apply what we know about the brain that can transform our lives? Learn more about the show and subscribe@changelog.com BrainScience until then, here's a preview of episode one, where we talk about the fundamentals of being human.

We're also all designed to be in relationship. We are fundamentally hardwired to have social groups and this sense of attachment. And because I'm sort of a geek when it comes to research, what researchers have found is that attachment, which that's what we label how we relate and connect with others, attachment is 100% learned. Which means our genetics don't actually contribute to how we learn to stay in proximity with other people. And with that, that we all develop ways to manage the threat of the loss of a relationship. But nobody gets to opt out of going, I need to be in relationship with others. I mean, think about it within the context of the prison system. Like why is it that the punishment for prisoners when they don't fall in line is isolation?

Yeah, that's true.

Right. That wouldn't be significant if in some way that doesn't actually harm our brain.

It's almost like we need to have that echo from another human being to let us know that we. Yeah, we're there or we're alive or just some sort of feedback loop. I'm not really sure how to describe that.

Well, it really is this sense of being with. Right. Like I can't fight battles on my friends behalf or on my kids behalfs.

Right.

But the simple fact that I know of what's going on makes the difference because I would contend it sort of like I help them hold that weight emotionally. And so that actually leads me into the third thing. And the third thing that I would say in regards to the fundamentals of being human is that we all struggle. Oh yes, right.

Big time.

And that, you know, we don't always get to pick the way in which we struggle, but we all struggle.

Well, if you like what you hear, you should go to changelog.com brainscience the show is not out yet, so don't get too excited. But you can subscribe and be notified as soon as this show launches. Once again, changelog.com BrainScience.

Okay, lastly, so we talked about reinforcement learning. We've talked about Gans. Let's go ahead and jump into this last thing that I hear people asking about, which is transfer learning. And we've certainly touched. Touched on this in previous episodes, but we haven't kind of put it in context like we're putting in context these, these other things. So transfer learning is another one of these kind of methodologies or approaches that's used in, in AI by AI practitioners to do a bunch of different things. But transfer learning isn't kind of a model in and of itself. It's. It's a, it's another one of these approaches. And I would say in, in comparison to Gans and reinforcement learning, it's actually one that I've. I've leveraged pretty heavily in, in my own work. I don't, I haven't touched as much as much or reinforcement learning, and Gans haven't touched my life as much as transfer learning has. I think transfer learning is something that pretty much all AI practitioners should be familiar with and utilize heavily. What do you think, Chris?

I would say that pretty much all AI practitioners have utilized it, whether they realized it or not. Yeah, that's totally true. If not before, certainly when they were learning how to do this and they were initially going through and creating their first models, they were almost certainly using transfer learning, even if they didn't realize it. It's kind of the secret weapon of kind of getting yourself going. And it's probably almost always used in certain types of use cases, such as computer vision. And that'll. As we get into defining what it is, it'll become apparent why.

Yeah, and it's definitely impacted the natural language processing, or NLP community very heavily, and there's been a lot of efforts in that direction recently. I know on one of our very first episodes, we had the guys from Machine Box on. On that was.

Episode. Yeah, that was episode.

Was that the first. Oh, that was episode two. The first one with guests.

It was the first one with guests. That's correct.

Yeah. So Machine Box has this really great service that you can spin up that will do facial recognition. And really all you have to give it is like one or two images of a person's face, and it automatically kind of updates the model and does really great facial recognition. And of course, it's not just utilizing one or two images and training a whole neural network on two images. That just wouldn't work. So there's something else being leveraged under the hood. And as Chris mentioned, in that computer vision context, or nlp, a lot of times that thing is transfer learning. So at a high level, how do you think about transfer learning, Chris?

So the way I think about it is when you're creating a model, you don't just go and do it and it's done. It is an iterative process. And so kind of going back to the basics of what deep learning is, what a deep neural network is, is you have a series of layers, and each of those layers is responsible for generalizing something, understanding something, and they tend to build on themselves. So in the context, to make it real of Computer vision, you may have a deep neural network and the early layers are there to recognize just simple things like lines or corners and things like that. And you tend to build those features up to where now it recognizes, after it combines some of those together, what lips look like or what an eye looks like. And then, you know, you go up a little bit and it starts to recognize how you put those different features together and make it a face and, you know, and then a full head. And so each one builds upon the other. So the really cool thing about this is let's say that you need to go recognize something and maybe some of those baseline features, like recognizing at the lowest level, recognizing lines and curves and such. Obviously in every image you're going to do that. So if you have a model that's really good at doing that, already taking that, and if it's getting close to human recognition or maybe animals or certain common objects, you can move higher up the stack. And then at whatever point the purpose of that pre existing model might diverge from yours, you can take those layers that were consistent with what you're trying to achieve and build upon those. And since they were built with a, with a general data set that is different from the data that you're about to train it on, your new model is more likely to generalize better as well, since you have a more diverse data set by definition, since you have pulled in a partially trained model from somebody else's data set. And so it's kind of like we're all standing on the shoulders of giants. You build upon what other people have built and you get it. You can take that pre existing model that might work really well up to a point, and then you take your specific data with your specific data set, a set of images about something that you care about, and then fill that out and you end up being able to get a very robust model that does something very useful with much, much less training. And it's, it's a lot less brittle since it has a broader data set to base it on.

It's kind of like, you know, most programmers don't write every single line of code from scratch when they're creating a new application. There's a lot of copy and paste that goes on because they've done this before, they've done that before, they've created this sort of service and it just needs to be kind of slightly different this time around. And so they, they never start from scratch, they kind of copy a bunch of things over. It's very similar here in the Sense that like, yes, you can in many cases take a model that has not been trained on any data yet and train it to do a certain task. Let's say that we want to translate text from English to Hindi. So what we do is we get a parallel corpus, so we've got a bunch of examples of English phrases and then a bunch of examples of the corresponding Hindi translation of that. And we train a model on English to Hindi so that when we put in an English phrase, what we get out is the corresponding Hindi translation. So that's kind of like training from scratch. That would be like, you know, creating every, in my, in my analogy, creating every line of code from scratch. We're, we're initializing all of the weights and the biases of our model, all of the parameters of our model from scratch. So from some random seed or all starting at zero or whatever that initialization is, but we're, we're using that English to Hindi corpus to train all of those parameters of the model from scratch. Whereas now let's say after we've done that, we don't want, later on in our work, we don't want a model that is trained to translate English to Hindi, but we want English to Urdu, which is a related language. And you know, this means that we could do one of two things. We could either get another huge corpus of like English to Urdu data and train from scratch again, or we could leverage the knowledge that we already, that we already created in that English to Hindi model. So we could take that model and all of the weights and parameters that we trained for English to Hindi and then we could just kind of slightly modify it or fine tune it by retraining those on the new data set, maybe a smaller amount of English to Urdu. So this has been widely used in NLP because in a lot of cases maybe you want to take a pre trained model that's very general. So it's applied to maybe do translation for all domains. And you want to really fine tune that for a specific domain of text or of some content. And so what you'll do is you'll fine tune or slightly modify that on this new data set. So there's kind of this initial pre trained model and then there's the fine tuning of that pre trained model on a new data set. So it could be on a new data set or you might fine tune it by kind of adding additional layers to it as well.

So to kind of bring this back full circle on that, if any of our listeners have taken classes from, you know, Maybe Nvidia's Deep Learning Institute or maybe Coursera or whatever on specific things like NLP or computer vision or something. Chances are in that class, one of the things you did when you started creating those, the models for your class was they would have you go in and select an architecture to base that on and that itself is transfer learning. And you're going to find libraries of these models that are pre trained that you can build upon in all the common frameworks out there. TensorFlow has them, PyTorch has them. It is truly the most common way certainly to get started or to do or to build upon. I have, in my experience, I have more often than not seen people use transfer learning in their work than start from scratch and try to build things completely for the ground up. You would have to do that if there is not the right type of model that you can build upon. But this is normal stuff, this is what we do. And I thought you were analogy Daniel, for in terms of using libraries, if you're a programmer, is you know, you're truly using lots and lots of code that other people have built. Maybe a lot of that's open source, maybe some of it's proprietary, but you're still using those APIs to build whatever thing you're building, whatever application you're building. And it's. Exactly. That's a fantastic analogy you gave on matching it up to transfer learning in ML.

And another thing is like in a lot of cases you may just not have access to the data that you need. So for example, you know, you may not have access to the huge number of face images that someone else has trained a model, a facial recognition model on. So they might have, you know, 200 gigabytes of images that they trained their model on and you only have a handful. But that doesn't mean that you're kind of totally out of luck, right? Because a lot of people have released these sort of pre trained models for facial recognition and other things to where like we were talking earlier with Machinebox, you might just be able to utilize that pre trained model and update it with a couple new images or a handful of new images. And you know, that kind of removes the burden on you to gather all of these large sets of data, maintain them, update them over time, run really long jobs to train these models using GPUs, which is really expensive. And so it can also be kind of operational and cost saving strategy as well.

Absolutely.

So I've seen transfer learning in the news recently in a few different places. One of the places. As I was searching around in preparation for this episode, I saw a recent article from forbes about Google's AutoML. It mentions actually transfer learning in that article, which I thought was reasonably technical for Forbes. But yeah, they talk about how Google's AutoML services are using transfer learning, leveraging transfer learning to allow people to create these sort of customized models maybe for translation for their specific domain, like for law or for medicine or for customer service or something like that. So it's definitely being utilized in a lot of production services. Where else have have you seen transfer learning kind of come into, you know, recent news or recent releases of things? Chris?

Well, we've, we've had several episodes that made reference, you know, some of the, the algorithms that we've talked about were Bert in episode 22 we talked about GPT. I'm sorry, yeah, GPT 2 in episode 32. And those are models that you can build upon as well. And that's really, I think it's really important to note that this is kind of the standard way you start thinking about a problem is you go out and look and see if there is something out there that makes sense to build upon. It's almost the root into machine learning today. And so a lot of these great institutions are, are in fact building things that all of us can then take it thereafter in the tool of whichever one we want to use and apply that.

Yeah, I think the, the sort of Bert and GPT2 and other large scale language models are good examples. So for example with like a, as we talked about Bert or GPT, GPT2 and other episodes you can basically take that pre trained model and in a lot of cases how you would quote unquote fine tune it is by adding a layer that would do named entity recognition or adding a layer that would do text classification or something and keeping all of that knowledge from the BERT or GPT2 embeddings on the front end of your model. So you're kind of only adding or changing it a little bit, but you're kind of leveraging all of that knowledge that Google or OpenAI has already built into it for you. So a couple of things that I've seen even over the past couple days, if you're looking to get hands on with transfer learning, there have been a couple resources that have, that have been published that are, I think are really great. So one of those is blog posts from the Hugging Face team. So if you Remember on episode 35 we had Clem from Hugging Face on had some really interesting and fun stuff to talk about, but their team has released this tutorial on how to build a state of the art conversational AI with transfer learning. And I think that builds on some of these large scale language models. And then even today I just saw there was NAACL workshop on transfer learning. So that's the computational linguistics conference that's happening I think right even right now up in Minnesota. And there was a workshop there and they released so released all of the code and colab notebooks and information, I think slides from that tutorial. So we'll make sure and link that in the show notes as well if you want to get hands on with transfer learning. But yeah, I think that. So talking through these things with you, Chris, has definitely helped categorize some of these major components of AI methodologies in my mind. I hope it has for you as well.

It definitely has. I hope that we get feedback from our listeners that certainly I know when we were talking about doing this before recording this episode, we were hoping that there might be some of the confusion out there that we could, that we could alleviate. And if we would love to hear back from people through changelog.com community or on our LinkedIn group, which we invite people to join as well, you can find it out. You can search for practical AI podcast on LinkedIn and do that. But we'd love your feedback to know if these were helpful. Are there other specific questions we left unanswered and are there other topics that you would like us to cover in future shows? Awesome.

Well, thanks for, thanks for talking through these things with me, Chris, and I'll look forward to hearing from our listeners out there of how they're using these techniques and if we messed anything up or misspoke or if there's additional great resources that you know about on this front, please, please reach out and we will. We will talk with you again soon.

All right, thank you for tuning into this episode of Practical Way. I if you enjoyed this show, do us a favor, go on itunes, give us a rating, go in your podcast app and favorite it. If you are on Twitter or social network, share a link with a friend. Whatever you gotta do, share the show with a friend if you enjoyed it. And bandwidth for changelog is provided by Fastly. Learn more@fastly.com and we catch our errors before our users do here at changelog because of rollbar. Check them out at robot.com changelog and we're hosted on linode cloud servers@eleno.com changelog check them out. Support this show. This episode is hosted by Daniel White Knack and Chris Benson. The music is by Breakmaster Cylinder and you can find more shows just like this@change law.com when you go there, pop in your email address. Get our weekly email keeping you up to date with the news and podcast for developers in your inbox every single week. Thanks for tuning in. We'll see you next week.

Sam.