Days 36-39 of a 100 Days of Code challenge: the speaker trains a Q-learning network to play CartPole using OpenAI's Gym, explaining reinforcement learning concepts like reward and exploration. He's also applied for a research assistant role at Oxford focused on existential risk and started a machine-learning course despite missing prerequisites.
Chapters
What's going on, y'all? Day 36 of the 100 Days of Code series. What do we got here? 16th of June, 2017. Hope you're all well. You know what? I want to tell you a story. When I started this course, Machine Learning, I just finished week 7, by the way, yesterday afternoon. I did not read this. Didn't read prerequisites. Calculus, never heard of partial derivatives until 2 months ago. Same with the chain rule. Linear algebra, hadn't done that since high school, 7 years ago. Octave and MATLAB, no idea what they were either. So, out of my own stupidity and sort of rushing into doing things, I signed up to this course 7 weeks ago without reading the prerequisites. And the fact that it's advanced I started programming like 2 and a half months ago. I'm doing 100 Days of Code. That's what beginners do. Nah, I'm kidding. Anyone can do that. Um, but if I had read that in the start, I probably wouldn't have started this at all. And now that I'm 7 weeks through, I'm really glad I didn't read that because I've learned so much over the past 7 weeks. And so I think that's relatable to so many different things, like, uh, job postings. If someone like was to read a job posting and thought they didn't meet the prerequisites and they don't apply Hey, they could have been a perfect candidate for the job. It's just like, you don't have to fit all the prerequisites. Same with this course. I mean, if, if I was to read all those prerequisites, I probably wouldn't have signed up. And I wonder how many other people have like read courses and things like that and not signed up because they hadn't met the prerequisites. Now, of course, I understand it could be a stupid move to sign up to something that you don't fully fit the prerequisites for, but with enough determination and a willingness to learn, you can learn it just like I did. And I'm not special. I just, I don't know, I just try to learn these things. I get excited about learning. P.S., I updated my 100 Days of Code series on Medium. I'll link that in the description if you haven't seen it. I write there, I write there every day with a little summary of what I'm doing, much like this video. It's like a dual wielding 100 Days of Code series. We've got the video series here and we've got the writing series on Medium. But I'll catch you in the next step. Oh, what am I working on today? I'm working on WordPress, trying to get the front end ready for my website. Still, still early days, still, still new, new to me, but it's all a learning curve, right? Catch you soon. What's up, y'all? Day 37 of the 100 Days of Code series. I just applied for a research assistant's job at the University of Oxford. Application successfully submitted. So what's it doing? Well, it'll be— they're trying to put together some research and study and create a book. For existential risk. What does that mean? So like large risk, let's just say a pandemic or let's say artificial intelligence taking over the world or whatnot. So, I'm really interested in that sort of stuff, particularly artificial intelligence, but there's also a lot of things that really do interest me as well, such as pandemics and genetics and things like that. How easily people can alter genetics these days is mind-blowing. So, 100 years ago, couldn't do it at all. Now, high school students can alter the genetics of small little organisms. Even, even you or I could. And we can get our DNA sequenced in a matter of hours. But yeah, exciting day. Um, I'm going to do some code later this evening, probably some Python, just a little bit of Anki and a little bit of reading the Python textbook that I'm reading. Otherwise Saturdays I like to have as a chill-out day, so as little technology as possible, let my brain recover and then have a big week of study ahead. But I'm sort of— I've narrowed it down over how I'm going to spend the next 6 months depending on if I get this position, unlikely, but if I do, I'd be happy. It'd be awesome. Uh, I'm going to do an open source data science master's degree and I'll keep you posted for that. Uh, you can go to datasciencemasters.org, I believe, or just search open source data science masters. I'm going to spend at least the rest of this year doing that and then hopefully use that knowledge to build something great or work for another company towards building something great. But I'll check back in later today if, if I get any good progress done. Otherwise, I'll see you tomorrow, and who knows what I'm doing then. Actually, tomorrow's writing. I do some writing on Sundays, but another day, another thing to learn. What you can see here is a deep learning transfer learning network, or Q-learning neural network if I've got that right, training this little cart here to keep this pole upright. It's a game. I'm still trying to understand it too. So I'm learning deep learning. It's day 39 of the 100 Days of Code, and I'm learning about transfer learning today. And what's happening now is I'm using a framework from OpenAI called the Gym framework, or the Gym environment, which OpenAI have created to sort of use an environment to train transfer learning neural networks on Atari games. And basic games like CartPole. CartPole is one of the easier ones because it's just a moving cart trying to keep the pole upright. And essentially what you're trying to do is, is take one network and train a game how to play the game by using things like reinforcement learning and stuff like that, and rewards and steps and actions and series and states. And just like we would learn a game, so when we play a game, we learn sort of what works and what doesn't work. and that's what we're trying to teach the neural network here in this example. This is a really simple example, but it's really cool actually. Like, there's a, uh, there's an article that's, that I've been linked here. It's about how OpenAI or DeepMind or something, one of the, one of the big companies in terms of AI, deep learning, whatnot, uh, was able to train a neural network to learn how to play different Atari games relatively quickly and better than a human level, well, a human level expert at these games. So really cool. Why start on the games? Well, because it's not going to hurt anyone. If we really want to create sort of artificial general intelligence, it's best to learn on games first rather than sort of deploying these things to the real world. That's what I'm going to be learning for the most of today is deep learning and Udacity. And then if I manage to finish the classes for today, I'll do some reading and whatnot, but I'll catch you up later this afternoon with what I've learned. As you can see here, it's going through a whole bunch of training iterations. So episode 543, 544, reward 199, training loss. The explore P is going down with each iteration. So see, 743, 730, 718. And as you can see, with each new iteration, the cart, which is this little black box at the bottom here, can gets better at balancing that wooden pole. See, look, that was really good. It's getting better and better. So each time the wooden pole sways too far to the left or the right, it'll reset itself. Or each part— each time the cart moves too far to the left or the right, it'll reset itself. But what it's doing is it's learning, and the longer it can hold the pole upright, it gets a bigger reward. So right now I think the reward's capped out at 199, but it is still slowly reducing the, the error here. Every time it gets a little bit smaller. That's really cool. So you take this approach, apply it to other, other games and stuff like that, apply it to real-life situations and get better and better, and there you go. All of a sudden computer can play a game that I would take a long time to master in about 5 minutes of training. The beautiful thing is, is this algorithm can be transferred to different games, different Atari games, because it's, it's based off the same principle, right? The reward and exploration and exploitation principle. But I'm still learning more about this. Hit you up more in the next clip. Whoa! If you haven't checked out Andrej Karpathy, well, if you're into machine learning and deep learning and computer science and whatnot, you have to check out Andrej Karpathy. His blog is for phenomenal. I just read an article called Deep Reinforcement Learning. I'll link it in the description. Pong from Pixels. So essentially he talks about how to train a reinforcement network to play Pong. So if you've probably heard before in the news and stuff like that, DeepMind and whatnot have trained neural networks to be able to play, uh, Atari games really well, but at a human level. And it sounds incredibly impressive, which it is, don't get me wrong, it is. But after reading Andrej's blog, it sort of breaks it down step by step and sort of teach you how to do it. So then you start to realize, oh wait, I can actually do this, and it's not sort of groundbreaking as, say, something like, I don't know, something else, something next level. But I think it's, it's awesome to, I don't know, have this resource available. So I'll link his blog in the description. Highly recommend reading if you're into machine learning, computer science, or just deep learning, or just amazing tech in general. So shout out to Andrej Karpathy. I'm gonna learn some stuff on Khan Academy now actually, because I'm finding that my linear algebra skills are not up to scratch. And this is one of the courses that's recommended on the open source computer, open source data science masters. So I'm gonna work through this for about the next hour or so. And that'll be it for today's day of study. So we'll catch you tomorrow.
