Deep Reinforcement Learning

Bandwidth for Changelog is provided by Fastly. Learn more@fastly.com we move fast and fix things here at Changelog because of Rollbar. Check them out@rollbar.com and we're hosted on Linode Cloud Servers. Head to linode.com changelog this episode is brought to you by DigitalOcean, the simplest cloud platform out there and we're excited to share. They now offer dedicated virtual droplets and unlike standard droplets which use shared virtual CPU threads, their two performance plans general purp CPU optimized They have dedicated virtual CPU threads. This translates to higher performance and increased consistency during CPU intensive processes. So if you have build boxes, cicd, video encoding, machine learning, ad serving, game servers, databases, batch processing, data mining, application servers or active front end web servers that need to be full duty CPU all day every day, then check out Digital Ocean's dedicated virtual CPU droplets. Pricing is very Competitive starting at 40 bucks a month. Learn more and get started for free with a 100 of credit at.co changelog again.co changelog. Welcome to Practical AI, a weekly podcast about making artificial intelligence practical, productive and accessible to everyone. This is where conversations around AI, machine learning and data science happen. Join the community and slack with us around various topics of the show@changelog.com community. Follow us on Twitter we're PracticalAI FM and now onto the show.

Welcome to the Practical AI Podcast. This is Chris Benson. We are the podcast that brings AI to you in a practical, productive and accessible way. I am at Nvidia's GPU technology conference in March 2019 and we have a lot of world class people in the AI space here and normally I would have my co host Daniel Whitenack joining me. He was not able to make it this time, but I have the pleasure of introducing to you Adam Stuck. Adam is a graduate student who is studying robotics and deep learning at UC Berkeley and he's really focused on scaling deep reinforcement learning and hierarchical reinforcement learning. And so it's a pleasure to have you on the show. Adam, welcome.

All right, thank you Chris. It's a pleasure to be here.

So I wanted to note, and we'll dive into it in a few minutes, that you had done a talk called Doing More with More Recent Achievements and Large Scale Deep Reinforcement Learning. But before we dive into the meat of it, could you tell me a little bit about your background, kind of how you got here, how you discover deep learning and deep reinforcement learning and what your story is?

Sure. I've been working and studying deep reinforcement learning for about four years now as a PhD student at UC Berkeley under my professor, Peter Rabil. Previous to that, I had studied physics and mathematics in undergrad. Actually did a master's in physics here at UC Berkeley. And then I was in the Air Force at the time, so I was pulled away on active duty. I was very lucky to land a position at the Air Force Research labor in Albuquerque, New Mexico. So I kept my technical chops up pretty well during this period. And yeah, about four years ago, my time in the Air Force kind of came to a natural end and I came back to school and was looking for what's hot, what's next. And yeah, the robots caught my eye.

So I got to start off by saying a couple of things, connections here. First of all, Daniel is going to be completely jealous that he's not here because he is a physicist who also came into the AI space as well. So he's going to be going, ah, gosh, why did you take it, Chris? So sorry, Daniel, on that. And number two, being in the Air Force, I work for Lockheed Martin in the AI space, and we do a lot of work with the Air Force. So anyway, I guess. So as you. As you started, were you already interested in deep learning, reinforcement learning when you were in the Air Force, or did that come along sometime thereafter?

No, this came along after, actually, only once I returned to graduate school. What I did get out of my time in the Air Force was just kind of by accident. It became a first exposure to me to a lot of programming and modeling and simulation in order to make informed decisions about technologies that we were going to invest in and try to fly in space.

What kind of tools were you using while you were doing that?

In that case, I was programming pretty much entirely in Matlab and a little bit of Excel. Why not?

Yep.

But that's actually a lot of people I know get into it from that, so that's totally valid.

And so, yeah, that kind of gave me the first impression of like, wow, this is really powerful. If you can craft a good simulation and run this program in the computer, it can tell you a lot of important things about what's going to happen in the real world. And that was the first thing that kind of turned me on to this idea, so that when I came back to school, it was kind of a natural thing to look to a computer science department.

Yeah. So tell me a little bit about that process. You're out of the Air Force, you're getting into school. How did you discover it? What, you know, what specifics did you get into? How did you find your way eventually to reinforcement learning. Yeah.

So I came back to school and I rejoined the physics department at Berkeley, where I'm actually still technically a member. But after about a semester and a summer of kind of floating around to different research groups and finding a lot of really, really deep and interesting projects going on, but not really getting the personal feeling like I was finding a place where I wanted to be continuing the work after graduating. You know, doing the PhD in physics still has a really strong drought on me, but I started to look to other things to do after graduation. And then eventually I decided, well, if I'm going to do something different after graduating, why don't I get ready to do something different before graduating? And so I started looking a little broader and found the robotics work happening kind of across the street on the campus there in computer science. Very lucky to be at such a powerhouse research university like Berkeley with a hand in so many leading in so many different fields. So, yeah, I just kind of went across the street and knocked on the door and said, hey, hey, can I play? A year or two of basically re accomplishing an entire set of graduate coursework and projects later, the professor finally says, yeah, you know what, Adam? Okay, you can stay, you can do this with us.

He made you earn your chops there.

Which is code for I can see you're not going away, so I might as well.

Persistence there, intelligence and all, but sticking with it there. So they were a robotic shop at this point that you're moving into?

That's correct. Yeah, it's a robotic shop. And at the same time, it's a lot about applying deep learning to robotics. So kind of my personal path since then has actually been and much heavier on the deep learning side, the deep reinforcement learning in particular.

And before we even go on, we've mentioned reinforcement learning and deep reinforcement learning several times already. Could you tell us how you would interpret those in terms of what they are for anyone that doesn't know what that is?

Sure, yeah. Let's start with reinforcement learning, which has a little bit of a longer history and kind of pedigree behind it. It's basically kind of a formal way of doing learning by trial and error. So you've got some sort of task that you're trying to figure out how to do, you know, what are the different possible, you know, small actions that you can do in order to try to achieve that task. But you don't necessarily know ahead of time what the task is or how to do it. So you just explore, you try things the key thing is that you get a signal back from the. From the task. That's what we call a reward. When you receive a positive reward, it says, oh, you did a good thing, so do that more. When you receive a negative reward, it says, oh, you did a bad thing, so do that less. And yeah, based on queuing off of those signals, basically, again, through trial and.

Error, gradually, over time, make an adjustment.

You make an adjustment, you try again, and then eventually you can learn to do some pretty sophisticated tasks through this. So there's a whole mathematical formalism kind of like, built up around this, that we don't necessarily need to go into the depths of all the terms with that, but it's pretty well established. And a more recent development is deep reinforcement learning, which simply refers to the application of deep learning to reinforcement learning. Deep learning, meaning for all practical purposes, using deep neural networks.

Sure. And at a high level. Without diving too deep on this, can you talk a little bit about how you're applying deep learning to integrate that into reinforcement learning? Is there a high enough level for kind of an understanding of what that integration is without diving too far into the math?

Sure, yeah, we can do that. So one of the key components that you have in a reinforcement learning system is the learning. What you would call the learning agent itself. So the learning agent is interacting with the environment which contains the task, and the learning agent is where the deep learning comes in, because the learning agent is going to be somehow making use of a deep neural network to express its decision making.

Sure. And so this gives performance advantages over what you could previously do with just reinforcement learning. Given the fact that there are so many more options available in terms of adjustments that you can make going forward.

I assume that's exactly right. Yes. So in terms of the interfaces between the environment and the agent. So the environment provide some sort of information to the agent that the agent gets to look at and use, and we're making its decision of what to do next. And it turns out that deep neural networks basically open up a whole new class of environments of much more complicated sets of information that the agent is able to digest in order to figure out what to do next.

Sure. So I guess. So what. As you got into this environment, you're. Now they've accepted you into the fold. What did you do? What captured your imagination as you got into some of your first projects with the group?

Yeah, I think early on, the projects that I kind of keyed in on the most were some early work out of DeepMind on learning to play Atari Games from scratch, using only the screen image as input to the learning agent. So in a very similar style to the way that a human would actually operate to play the game.

So I remember reading about that. Can you tell us a little bit about what that was and how it inspired you to move forward?

Sure, yeah. So this is about five or more years ago by now, maybe was the first result of again playing Atari from only the screen image made a big splash when it came out. There was a Nature article about a cover of Nature even. So this went a lot more, published a lot more broadly than the usual learning only conferences and this sort of thing. And then very shortly after that, after the initial success is there with the first algorithm, then another paper comes out with a second learning algorithm that's a little bit different, but is also getting good results, maybe running a little bit faster. And then over the next several years, kind of in rapid fire, these teams that came out with the original ones are making little enhancements here, little enhancements there, piling them on, and before you know it, these learning algorithms are just completely destroying these Atari games way, way beyond human level. And it really is just a springboard for lots of more sophisticated and more interesting games, for instance, to come after that.

So, you know, obviously that's been a while since the Atari news that came out in Nature and stuff. Where did that take you at that point, as you're getting into this? What. What captured your interest? What did you choose to. To do? What activities and from. In terms of research did you want to engage in? What was that journey like? Because a lot of us out here, very interested, may not have had that experience and just very curious about how that evolved to where you are today with the talks that you're giving at this point.

Yeah, I think it actually ended up in combination with a certain course that I was taking on parallel computing. It actually led me down the path of scaling up implementations of deep reinforcement learning. So taking an algorithm, okay, this algorithm is like shown to work and here's how we run it. Maybe we're running it on a small computer and it takes five or 10 days to do one learning run. And to see the thing go from losing every point in Pong to winning every point in Pong. Pong say, okay, this is exciting. This is working. This is something. We've got something real to go on here. Now let's refine it and scale it up and adjust it to make much better use of the computers that are available today. Long story short, that has a lot to do with why I'm here at gtc.

Yep, we're working our way there. So you mentioned already the compute being a huge limitation and obviously we are at Nvidia GTC where it's all about compute in so many things. So I guess what bridged you today, you did this talk doing more with more recent achievements in large scale deep reinforcement learning. What kinds of things are you covering in that and what was the evolution to get to that point and what were some of the milestones that you went through along the way to get to where you're at now?

Sure. I think some of the early steps started out with just taking kind of an in house algorithm, reinforcement learning algorithm that we had and identifying kind of the key pieces in a way that the algorithm could be broken down to operate in parallel over more compute resources. So instead of just one CPU core running the computation or whatever, maybe you can use all the CPU cores in the machine, 16 of them running in parallel in a smart way that gets you 16 times faster to the answer. And so this is kind of how I started was just with one example kind of algorithm like this and just a lot of practice of figuring out how to set that up, how to set that up in a computer and get it to run and actually get the efficiency out.

And you hadn't even gotten to the GPU world yet?

Hadn't even gotten to the GPU world? No, no. And then let's see with this, I actually took a little, once, once we had this running up and running smoothly, took a little detour through, through actually an Intel Knight's Landing development kit that we bought. So a CPU based architecture, but with maybe 60 or 70 cores that we could successfully parallelize across, but ended up finding in the end that no, really.

It wasn't quite up to what you needed.

It wasn't quite up to what we needed. And really there's no way around it. Like the GPU was the way to go.

So at what point, how long ago was that where you realized you needed to go GPUs and you started grabbing those because you realized that you were limiting yourself until you could speed up your training?

Yeah, I think by now this was maybe about two years ago or so. So it was a bit of a journey. Yeah, starting on the GPUs about two years ago and then about one year ago kind of wrapping up the project and more or less having it in shape.

So one of the things I was thinking as you were taking me through this, what were you trying to achieve with some of these projects, as you're doing these experiments and carrying it forward, what was your end goal in mind? What were you trying to get to?

So the main, the main project that I was working on over this period was really just, man, we had a need for speed. We're just going to explore these deep reinforcement learning algorithms and see how fast can we learn one Atari game and see what kind of records we can't set and otherwise blow out of the water. So it turns out that a lot of the same techniques for parallelism that were useful in the CPU setting, the same principles applied in the GPU setting for using multiple GPUs inside of one computer to run the algorithm. So about the same time that I was setting down that path anyway, we were very lucky enough to have Nvidia come by and donate actually a DGX one as part of their university research support.

That's a great donation. That's the kind of donation I want right there.

So yeah, this is a really fortunate moment for us and good and good timing for me. As about the time I was getting into this anyway, and my advisor says, hey, hey Adam, we've got this DGX one and well, here you go to it. Go to it. Exactly.

Here's the first AI supercomputer. Have at it.

Exactly. And so I couldn't say no to that. I mean, didn't want to say no to that.

Of course not.

Couldn't say no to that.

That's Christmas right there.

And then. Couldn't stop once I started either.

So I'm guessing it was speeding up your process a little bit there, you know, to go from CPUs straight. And it sounds like you didn't have an in between. Did you go straight from CPU all the way to using DGX1 just like that, or was there intermediate steps along the way?

No, that's right. We pretty much just went straight to the DGX1.

Wow, that's like stepping out. That's like stepping out of this old jalopy car and getting right into the Lamborghini and just of kind going, that's it.

That's it. Yeah.

So I'm just curious, you know, it's non trivial to. You have to kind of to learn how to use a DGX1. They suddenly plopped one into your lap and said, hey, make use of this thing. And you know, what was the. Were you already familiar with containerization and kubernetes and things or was there a learning curve for you to get ready to use the box or what was that like to make that transition?

There was a little bit of a learning curve. And so at this point I was already pretty familiar with the deep learning, particular deep learning library that we were using, which at this time was Theano. And luckily enough about the same time, Theano released support for nickel. So Nvidia's collective communication library, which is for direct interconnects between the GPUs within the DGX1 for very fast operations, very fast communication. Sorry, the learning curve was mainly around. Okay, I already am familiar with this deep learning library, but let's see how to plug it into use for multiple GPUs. This was a pretty new thing at that time, about two years ago. And so there wasn't really built in support from Theano for multi GPU use. There was some kind of preliminary stabs at that, but what I ended up doing was kind of writing our own libraries for the parallel processing aspects. Sure. And then again, luckily, having the Theano guys expose an interface to this nickel, this communications library to tie the multiple GPUs together in the most efficient way possible based on the Nvidia hardware that was inside got you.

And I guess, were you already in the middle of a particular project when the DGX1 came into, into the picture or was that, did you kind of pick it up as you picked up a new project to move forward?

No, I think looking back on it, I hadn't really thought about this before, but it was just incredibly fortunate timing because I was already, I was already on this project, had already been tinkering around with, with the Knight's Landing for some time and was ready to, to move on for that and move to GPUs. And so no, it just like the timing couldn't, couldn't have been better happened.

Oh, that sounds fantastic. So I guess as you're making this transition, how are you. So you now have this DGX one that's been gifted to you and they're saying, go do stuff with it. What's going through your head? What kind of projects are you thinking about? You're now into deep reinforcement learning, I assume, at this point, and you have the power to drive that forward. What were you thinking and what kinds of projects have you taken on over the last couple of years to take advantage of that?

Yeah, so I mean, again, a lot of the work at the beginning was just scaling out the reinforcement learning itself, taking existing algorithms and showing, discovering that they can be scaled up to run on the entire system so that we could use all eight GPUs and all 40 CPU cores within a DGX1 to learn a single Atari game and get basically linear speed ups with that. So instead of taking 10 or 15 hours to master Pong, you know we're getting it to like four minutes. Yep or so, which was that probably.

Helps your productivity a little bit. Go get a cup of coffee and you come back and it's done.

Come back and it's done and then you can iterate and try the next thing. And that was actually, I think, a pretty interesting finding because one of the key techniques to scaling up to using multiple GPUs that we also see across a lot of other scaling efforts is increasing the training batch size. So how many in this. There might be a couple points where we need to stop back here and make some definitions, but when training these neural networks, we often use this algorithm called stochastic gradient descent.

And can you define that for the audience loosely?

Sure can, sure can. So basically you're going to have some way that you want to change the change the behavior of the neural net, which is this decision making function inside your reinforcement learning agent. And you're going to change it based on experience that you have in the game. But instead of making updates on all the possible experience that you could gather, you'll gather a little bit of experience at a time and then make a small adjustment and then gather a little bit of experience at a time and make a small adjustment. Foreign.

This episode is brought to you by a rally Open Source conference in Portland, Oregon, July 15th through 18th. We'll be there. By the way, as you know, OSCON has been ground zero for the open source community for 20 years. And this year they're expanding to become a software development conference. Because in 2019, software development is open source. At OSCON, you get to see what's shaping the future of software development. The program covers everything from open source AI infrastructure, blockchain, edge computing architecture and emerging languages. Hear from industry leaders like Holden Caro, Rupa Datri, Julian Simon and Allison McCully. Learn more and register@oscon.com changelog prices started just $925 when you register before April 19th. After that the price is going to go up. Plus you can use our code changelog20 to get 20% off your bronze silver gold passes. Once again, Our code is changelog20 and head to ozcon.com changelog to learn more and register.

Okay, so having defined stochastic gradient descent, take us from.

Okay, so the next step into making STOCHASTIC gradient descent run more efficiently on a gpu, which is itself a highly parallel computing platform, is one way to do this is to increase the training batch size. So increase the amount of experience that you gather and use together. Each time you're going to make a slight adjustment to this decision making function and that gives you full utilization of this gpu, which is kind of like a fat and wide computing pipe. And if you want to make full use of eight GPUs, then you need to have an eight times bigger training batch size in your algorithm because you need to fill up all eight of those GPUs in order to run them efficiently. So it's a pretty interesting finding that we found that we were able to scale up the training batch size, even in a game as simple as Pong, all the way to the point where we're making efficient use of eight GPUs in the full machine. And we're getting, we're getting good linear scaling where you're learning the game basically eight times as fast when using eight GPUs.

So I know at some point Nvidia is working on some of the same problems as they're learning to get the parallelism of the capacity out there so that you can take advantage of all the GPUs. Are you still using the code that you wrote to handle that or have you switched over to some of the stuff that Nvidia was producing? I had the same experience working at a prior employer where some of the things that we needed to do was out ahead of any release and stuff. So we had a kind of a similar problem. Have you just stuck with the code that you wrote to accomplish that?

Yeah, so far, for new projects that I've been working on that aren't focused on scaling RL but doing other learning experiments. Yeah, exactly. We're using the same code base that we established before during the scale up project. So far, the new projects we've been running, it's interesting, you end up running many different experiments in order to see what is working and what is not working. So in the end, an efficient way to run is actually to use only a single gpu, but maybe stack multiple different experiments running on the same computer, each one on a separate gpu. And so we had kind of like baked that into our code base anywhere where it was flexible. You could either throw all eight GPUs at one problem or have each of the eight GPUs running a separate problem. And you have many different settings that you're trying to Test anyway. And so either way, in the end you get a good turnaround of results.

Gotcha. So now that we've kind of gotten up to more or less the more recent past and you're now using DGX1 and you have these great tools available to you and to kind of come full circle back to your talk today. As you talked about, and I quote, recent achievements in large scale deep reinforcement learning. What things did you cover in your talk? What kinds of things is useful? So some of our listeners out There are using DGX1, DGX2s and other equipment from other companies as well. What are the learnings that you found that are going to help us along there in terms of recent achievements?

Sure. So most of the projects that I talked about earlier today were to do with large scale research projects happening at other organizations such as like Google DeepMind in London and OpenAI here in San Francisco. Some really, really impressive recent achievements coming out of those places.

Can you share some of those achievements from each that you talked about?

Certainly can, yeah. So kind of building on this like foundations that happened under the Atari experiments over the last several years, now there's new results coming out in other domains to include Dota 2, Starcraft 2, a version of Capture the Flag that's in an actual first person action 3D game style. So it's really exciting to see a lot of more sophisticated video games that are actually interesting and still maybe addictive for humans to play today that are now being tackled and solved by these learning agents.

Yeah, the Starcraft II work really captured my imagination and those of some friends of mine. We were talking about that just in terms of just this field is leaping forward so fast. As you talked about going from Pong to managing games of that level with this technology and doing that, I guess tell us a little bit more about DeepMind and OpenAI and how that's influenced you.

Yeah, sure. I was fortunate enough to do an internship, research internship at DeepMind over the last fall. So I had some firsthand exposure to some of their techniques and their working methods and such. I'm not going to say anything that isn't publicly.

Of course not. We would never ask you that.

So I got to be careful about that. But yeah, no, it was a wonderful experience to see the whole organization from top to bottom. Amazing resources in terms of people, amazing resources in terms of compute. And it's no wonder once you're inside the door, it's no wonder that the amazing things that are coming out.

So of the things that are Publicly known. Now, what was the coolest thing that you experienced there while you were doing that internship that you can share with us?

Yeah, I think the thing that was the coolest for me again was kind of the combination of these two factors. I just saw the, the people and the compute resources and both of those organized well in the same place. So actually was sitting next to much of the StarCraft II team really when they were in the heat of their development last fall. Oh, that's cool. In the run up to these professional matches and so just like I wasn't personally involved on that particular project, but I just heard a lot of background chatter, looking over their shoulders, saw the very intense meetings going on, saw people screaming, scrambling to get presentations together for the meetings with their latest developments and learning curves and everything. Saw people debating about how much computer they were going to use, whether they were going to bring down all Google Cloud or something like that.

Oh wow. That would be a bad moment.

There would be a bad moment. Yeah. So it was really amazing to see this great organization of really interested and, and genuinely excited researchers working together in this team environment that was very.

I.

Don'T know, it's like professional and eager.

Yeah. So what about OpenAI? You mentioned that a moment ago. What are some of the things that they have been doing that have inspired you?

Yeah, and OpenAI also has a very exciting project going on with a slightly different game, Dota 2, which I think is even more popular and widespread than Starcraft 2. I think I read something on their blog about maybe the annual winnings of esports contest in Dota 2 is like $40 million a year or so.

So this game is happening, some real money there.

This game is happening, there's real money. And yeah, likewise they have like a totally independent branch, a very different approach and their own run to train an agent to play at a professional level in a very complicated environment such as these games.

So it sounds like pretty amazing experiences to be exposed to that. Where do you see yourself going? Is there stuff that you're working on now that you can share or things that you have in mind for the future and kind of what you want to do specifically?

Yeah, there's a couple of projects that I've been working on recently that are kind of, I think are interesting. One of them actually is working towards real world applications of reinforcement learning. So it's very exciting, all this development that is happening in these video games. But obviously we're not developing reinforcement learning to play the games better. We're learning to play the Games? Well, we could be, but we're, but we're learning to play the games better to develop reinforcement learning maybe for other things. So I think another really exciting result that came out recently is in job scheduling, which is like managing parallel computing resources. So I give a shout out to another group. Hung Xi Mao from MIT is the first author on a paper that came out in the fall to do with reinforcement learning for job scheduling on data clusters under the Apache Spark setup.

We'll include a link to that in the show notes as well for people to go. Reference.

Great. This is really, really exciting work to see reinforcement learning applied to a real world problem like this. Scheduling hundreds or thousands of CPUs basically under diverse workloads from multiple users. And they did a really nice job of laying out okay, here's a couple of heuristic job scheduling algorithms that you might have for deciding which computing tasks need to run on which CPU and when they have some really nice heuristics that they use and then also some more advanced but recent handcrafted algorithms and then they show deep reinforcement learning of course, taking the cake and blowing them all out of the water. It's a really beautiful thing to see.

So where do you think we're going next in terms of applying deep reinforcement learning to robotics? This field, both of those individually are moving so fast. Now what do you do a little fortune telling for us and lay out what you think we're going to see in the near term or maybe even the longer term. Be a little speculative.

Okay, sure. I think we're already seeing interesting robotics results coming out of like OpenAI for instance. They had blog posts recently on learning dexterity where they had a robotic hand which could manipulate a cube in very, very human like looking ways. Trained entirely on reinforcement learning using basically the same algorithm and setup as they used for Dota 2, which is really cool to see this techniques kind of crossing into new applications. But I think definitely robotics is, is a very, very ripe field for application where obviously there's so many things that could be enabled where not only is the hardware getting there and in place, but man, it's really hard to. It's just really hard to hand program these things to do interesting activities in new environments and new places that they haven't seen before. And learning is definitely going to be the way to get behaviors out of robots that generalize to new scenarios that will let them roam free in the world and be useful and safe.

So do you dare to put any timeframes around when we'll see different levels of that out there or am I getting too out there? The look on his face, by the way, you can't see it listening to this, but he was like, oh God, I just poured gas on the fire right there.

Maybe what we can do is I'll just record, record 5 years, 10 years, 15 years, and then we can go back later, depending on which one was right.

There you go.

Edit it in. How about that?

Fair enough. I had to try there. So a lot of people that listen are kind of getting into it. You know, we have a lot of people that are practitioners, but we also have a lot of people out there that are in the technology world, academic world, business world, who are trying to figure out how they're going to move into this space. As someone who, who has ramped up, done that, understands what's necessary to be productive in the world, what kind of advice do you have for people in a couple of different scenarios? If they're in a business and they may have a data science team or something or group of developers and they're wanting to kind of retrain in reinforcement learning, deep reinforcement learning, and maybe even robotics that might be an application that they're interested in. What would you, first of all, for that kind of person already out there in the real world, so to speak, what should they be thinking about? How should they start making a transition to take advantage of these technologies at this point?

Yeah, I think one possible way to approach this, taking on a new field like this that I think could be productive, is to take the time to practice it and do it and do it yourself. So there's starting to be more and more implementations of, of these deep reinforcement learning algorithms available out there on the Internet. GitHub, for instance, I'm sure is full of them. I mean, I just released the code for XLRL. My project for scaling up on the DGX1 is now out on GitHub and.

Of course we'll add a link to that into the show notes there.

Okay, so this is one such example, but really taking the time to, okay, read off of those, but implement it yourself, make yourself rewrite it from scratch and run it and get all the way to the point where you're recreating the learning curve that is published in the literature. Because a lot of what happens with this work is there's so many programming concepts that are not too difficult to understand mainly and get in place, but there can be lots of little bugs and lots of little gotchas and maybe you didn't tune this hyperparameter just right. Or maybe there's a detail that's a little bit obscured in the paper and maybe isn't as revealed as clearly as could have been. And so you need to go back and reference someone else's implementation to say, oh, that number needs to be 0.25 instead of 1.25 and suddenly it works now, oh, this is new. So make yourself go through all the stages basically from scratch to reproducing the learning curve that you see in the literature and then you'll know that you're doing the, the full stack.

Gotcha. And I'm going to ask for one other use case that's probably some commonality between them. But if you are a student maybe in high school now, and you're looking at trying to figure out what you're going to do as you go into your first university experience and you know, what would you advise that, you know, 16, 17, 18 year old kid who's ready to say, hey, I think this might be what I want to do. And they have the advantage of going through schooling and taking advantage. They're not out there in the world yet. What would you advise them to do? What kind of track should they be thinking about?

Yeah, definitely taking as many of the computer science and programming classes as you can. Being a good programmer is definitely a plus in this field. But if you're really trying to get into learning research, it's actually not always fully necessary. Like there's a lot of things to know about learning theory and such that you could also get involved in a lot of mathematics, a lot of statistics. So yeah, I would say try to take early on, take a broad approach because a different sub area within this whole field of learning might grab your attention and before you know it, you've gone a year or two deep into, into a certain subtopic and then you realize, oh, this is the part that I really want to do.

Okay, well this has been fascinating. Thank you so much for coming on. I guess as people want to reach out to you, make contact with you, where are you in your social media? How do you like to interact with people out there in the world?

Yeah, sure, that's easy. You can find me either on Facebook or old fashioned email can do the.

Trick as well and we'll include those as well. So thank you very much. This was a great conversation. I really enjoyed it. You really made the thought of getting into deep reinforcement learning accessible and thanks for doing that and sharing with us and appreciate the time.

Good. Yeah. Invite anyone who wants to to come in and jump on the field. It's exciting times.

Will do. Thank you very much.

Thank you.

All right, thank you for tuning into this episode of Practically I. If you enjoyed this show, do us a favor. Favor. Go on itunes and give us a rating. Go in your podcast app and favorite it. If you are on Twitter or social network, share a link with a friend. Whatever you gotta do. Share the show with a friend. If you enjoyed it and bandwidth for Changelog is provided by fastly. Learn more@fastly.com and we catch our errors before our users do here at Changelog because of rollbar. Check them out@robot.com changelog and we're hosted on Linode Cloud Servers. Head to linode.com changelog check them out. Support this show this episode is hosted by Daniel Whitenack and Chris Benson. The music is by Breakmaster Cylinder and you can find more shows just like this@changelow.com when you go there, pop in your email address. Get our weekly email keeping you up to date with the news and podcasts for developers in your inbox every single week. Thanks for tuning in. We'll see you next week.

Foreign.

Chicken dinner. You have won again. Yes. That's why you listen all the way to the end of the shows, because we give you previews of what's coming up. And as you may have guessed, we have another preview of our upcoming show called Brain Science. This podcast is for the curious. It explores the inner workings of the human brain to understand behavior change, habit formation, mental health, and the human condition. This show is hosted by myself, Adam Stowiak, and my good friend Mariel Reese, a doctor in clinical psychology. It's brain science applied not just how does the brain work, but how do we apply what we know about the brain to better our lives? Here we go. My wife and I, we've learned this. This concept of goodwill, right?

Yeah.

I can take your feedback or your criticisms in a different light if. If I know that you have goodwill for me, meaning that you're not trying to harm me, that you are for me, not against me. And sometimes change, as we all know, is painful and can be painful. So sometimes the necessary feedback and. Or criticism that can influence that change can also be painful. But I can accept it differently if I know that she or they or whomever is in the scenario with me has goodwill for me. Whereas if you know that they're not for you, then you obviously take it a whole different way. And that's. That's an okay thing. But we often are in relationship with people that are giving us crucial feedback. And we need to have that kind of that lens. It was significant in our marriage to understand, hey, I know there are times when you give me feedback. I am not happy about it, but I know you have goodwill for me. So therefore I calm down, I listen, I take that in and I process it, whatever. But I take it in a different way because I know that she's for me and not against me.

Yep. One of the key things when it comes to change is a sense of openness and even relationally like of going, I need to be able to see some how somebody else responds or how they're feeling as based on their perspective of what they're going through and not just my perspective of their perspective. And so this goodwill is like, I believe that we're on the same side and that you're not trying to make it harder for me. But so I can understand if I were sitting where you were sitting, had the background that you had, why you would have taken it in that way. And then I can provide an opportunity to clarify or create more connection even when it doesn't feel good. And I honestly think this is so much of what's missing in people's relationships. If I look at relational interactions through the notion of conditioning, wherein I get a sort of hit of dopamine, feel good feelings because I went to a person, I had a conversation that didn't necessarily feel good, but there was openness on both parties to hear one another's perspective, that it actually then reinforces like, oh, when I go and I have this exchange with people, I feel better. So now I'm going to go and engage with other people and get the feedback, even if I might not like the feedback because now I'm buffered and I'm not alone in this and I somebody else sees my world.

That's a preview of Brain Science. If you love where we're going with this, send us an email to get on the list to be notified the very moment this show gets released. Email us@enreviews change law.com in the subject line, put in all caps. Brain Science with a couple bangs. If you're really excited, you can also subscribe to our Master feed. To get all of our shows in one single feed, head to changelog.com master or search in your podcast app for Changelog Master. You'll find it. Subscribe. Get all of our shows and even those that only hit the master feed again. Changelove.com master.