The long road to AGI

Hello, this is your podcast co host Daniel Whitenack. I wanted to give a quick disclaimer before we jump into the content for this week, and that is that Chris and I recorded this episode a few weeks ago prior to when we had both seen the George Floyd video and all of the protests and calls for justice have happened in our country and and really all around the world. The tone of this episode didn't really match what we thought would be appropriate for that particular week, and so we held off on releasing the content. We also shifted focus a little bit and recorded a new episode on explainability and bias in AI, and we released that last week. We're going to continue to try to bring a focus on explainability and bias and fairness in AI algorithms to the podcast, and we both think that that's incredibly important, especially as more governments and police forces start using things like facial recognition in some cases. But we also want to make sure that we keep getting AI content out there and so we're going to go ahead and release this episode. If you have questions about anything related to AI and policing and fairness and bias and all of those things, we'd love to have a discussion with you about that. You can reach out to us anytime on Twitter or on our Slack channel or on our LinkedIn page and we'd love to have those discussions. I'd love to hear what you are thinking about our content and about AI and fairness and bias in general. So please reach out and I hope that this episode is useful and beneficial for you.

Bandwidth for Change log is provided by Fastly. Learn more@fastly.com we move fast and fix things here at Change Law because of Rollbar. Check them out@rollbar.com and we're hosted on Linode Cloud Servers. Head to lino.com changelog this episode is brought to you by DigitalOcean. DigitalOcean's Developer Cloud makes it simple to launch in the cloud and scale up as you grow. They have an intuitive control panel, predictable pricing accounts, worldwide availability with a 99.99 uptime SLA and 24 7, 365 world class support to back that up. Digital Ocean makes it easy to deploy, scale, store, secure and monitor your cloud environments. Head to Do Co Changelog to get started with a $100 credit. Again, Do Co Changelog.

Welcome to Practical AI Weekly podcast that makes artificial intelligence practical, productive and accessible to everyone. This is where conversations around AI, machine learning and data science happen. Join the community and slack with us around various topics of the show@changehaw.com community and follow us on Twitter. We're at PracticalAI FM.

Welcome to another fully connected episode of Practical AI where Chris and I keep you fully connected with. With everything that's happening in the AI community. We'll take some time to discuss some of the things in the latest AI news and we'll be digging into some learning resources to help you level up your machine learning game. I'm Daniel Whitenack. I'm a data scientist with SIL International and I'm joined as always by my co host Chris Benson, who is a principal AI strategist at Lockheed Martin. How you doing, Chris? How was the long weekend?

I'm good. We're just coming out of Memorial Day weekend.

You get to get outside a little bit. I did, of course, socially distanced, but.

Outside I did and I was staying socially distanced and. And I did that despite the fact that officially Georgia was the first state to open up.

Yeah, that's true. And it was something weird. It was like pool halls and tattoo parlors or something.

Like everything about it was weird. And so. Yeah, I mean it really, really was. You know, you look at the things and go, why did they choose that? Yeah, once upon a time, pool halls. But no more. Not many a year. Yeah, I don't do tattoos, at least not yet. So maybe I should.

Yeah, it's been a while on the pool hall. I think my advisor, when I was at NCAR after my undergrad, I did a high performance computing internship at NCAR and Boulder and he was like all into pool and had he like, he carried his like case with his pool queue. I guess that's what it's called around at the lab and everything. So anyway, that's serious. Bringing back some memories.

We have a pool table in our basement, so usually that's really.

Oh, wow.

Sadly, you'd think that would make us quite good. We're not.

But recreational.

Yeah, it's strictly recreational and usually it is accompanied with plenty of lubrication in the alcoholic sense.

Oh, gotcha.

Yeah, it's kind of like we've had a couple of drinks, we're like, yeah, let's go get a game of pool. And so, yeah, that's what we do.

Yeah. Anything else exciting over the weekend?

I had kind of a weird thing happen to me before we get started. So, you know, I do all this animal rescue stuff, you know, we've talked about and so I get a call. I'm probably the only person, you know that has a snake hotline. So for our local area. So I go and help with. That's one of the things I do, is help out because people are afraid of venomous snake and other things. And I get a weird call about a garter snake which is non venomous. But it was, it turned out to be quite a big one, about three feet, which is very large for that. That was stuck in netting, like yard netting that would keep straw down. And it was going to die because it was thoroughly entangled in it. And so I went to rescue without any gear. It was non venomous. I wasn't too worried about it. But the snake, I forgot the fact the snake would be thinking, I'm a hundred foot giant, relatively speaking. So as I spent 15 minutes cutting it out of the thing it was trapped in, it constantly bit me over and over again. And so they have tiny little teeth, but just enough to get blood. It doesn't really hurt. But he chewed on me for 15 minutes. So by the time I got him free, I looked like I was like an axe murderer with blood running all down my hand. So it was really, I, you know, it didn't hurt, but I was just, I looked at it, I was kind of going, this is just kind of a bizarre moment. So that was my memorial Day, but here we are talking about AI. Now that I've dragged listeners through that.

Speaking of snakes, this is an AI podcast and this is a fully connected episode so we can, you know, talk about whatever is on our mind in the AI world. I know there's been a variety of things going on. You know, some Covid things, some not Covid things. There's been conferences. Iclear was was recently that around that was interesting that we just talked about the Nvidia stuff. I think in our last episode, which was interesting, my Nvidia Xavier NX did come to my house, which I ordered. I'm excited to play with that, which I haven't yet. So maybe I'll give an update in a following episode about my experiences with that. But what else is on your mind? What's crossed your path in the AI world?

Well, before we go there on your NX thing, we should ask listeners about things that they are doing, interesting projects at some point and maybe have an episode where we kind of talk about really interesting things people are doing. Not really work related necessarily, but just things that are cool that they're doing. They're very creative on how. I would love to hear what people out there are doing.

Yeah, if you have an interesting side thing or interesting unique project, let us know. Reach out to us. We'd love to talk about it and you give me some inspiration.

Yep. We're on Slack and LinkedIn and Twitter and everywhere you can reach us, just about anywhere you might be.

Yeah, for sure, for sure. So what else is on your mind, Chris?

Well, I'll tell you what I was thinking about when we knew that we were going to start planning this, I've had something on my mind for a while and that is that we've started to allude to it in recent episodes this year. And that is the fact that we're kind of getting to a turning point in the entire field of artificial intelligence. While we have been so very focused on deep learning for the last few years, and pretty much the entire time we've been doing this podcast, we're starting to get to a point where a lot of the big advancements seem to have come out and we're seeing a lot of incremental stuff. And that's not to say we won't see some big advancements continue going forward, but it's kind of becoming a little bit mature in terms of there are models of various types CNNs in the NLP space. We've talked know Bert and all the others, GPT2 and everything, but we're kind of finding ourselves focusing heavily on these kind of the next version of the same model to some degree, you know, and the variants that are there.

Yeah, more data, more performance, sort of.

Yeah, exactly. And I've been thinking, I've been hearing more and more about like truly advancing the state of the art on the research side. And that was really brought to the forefront a few months ago at the NEURIPS conference when I always butcher his name. But Yeshua Bengio, if I'm saying it correctly, and I apologize if I'm not, but he did a keynote at the open of the conference, kind of talking about advancing the state of the art from kind of where we are now to where we could go. And he had one way of looking at that. And then separately in the work that I do at my employer, you know, we often have interactions and stuff with darpa, and I've been really becoming very aware of the way DARPA is looking at the future. And I probably should, for those listeners who are not aware of it, I probably should real quick say what DARPA is. And that is DARPA is the Defense Advanced Research Projects Agency. And everyone calls it DARPA for short.

Just rolls off the tongue.

That's right. It's quite a name. But it's been around for over 60 years. It is the original organization that invented the Internet, which was the very first incarnation of the Internet as we know it today was four nodes that were connected by the earliest version of the Internet protocol. And the idea was distributed computing across a wide area. These were all happened in the test case. They were all in the same facility. But the idea was that could survive nuclear holocaust and all that by having nodes go down and still have the overall system, the network itself, operating. And so DARPA is always been incredibly cutting edge. And they're the ones that kind of lead at least the US Government, military interests, into the future, typically on a Horizon that's like 15 to 20 years. So, you know, you'll see something happen, it'll get developed over time, and eventually it gets out there. And the Internet's one really. That's the most famous example of it. So they are really looking forward from what we're. And we can describe all this later, but the idea of what they call wave two that we're in now to wave three, which is kind of the next incarnation. And between the Neurips talk and the DARPA perspective and a bunch of others that I've read from other sources, I just thought it was time for us to start thinking about what the future looks like. So I know we're practical AI in this context. It may not be the practical of pulling out your Pytorch and starting to work on a model on this episode, but I thought us talking about what we think it might mean to move into the future and where we might go and what we think of the current conversation might be a fun conversation to have. You up for it?

Yeah, definitely. And I guess I can bring the practical side in here. I will admit that, you know, with any sort of conversation about like AGI or general intelligence or like next things, I often come in with a good amount of cynicism and probably don't give it the respect that it's deserved in terms of the discussion. But I will say that some of the stuff that you forward me and also the Neurips a keynote which we'll link to in our show notes, but it was definitely interesting and there were elements of what he's talking about as far as system to AI and that sort of thing that are sort of rooted in things that are being explored and experimented with and. And that sort of thing. So it's not so much like Terminator scenario or like Singularity talk, but more kind of like you're saying, pushing beyond the sort of very limited task oriented models to maybe something slightly differently or at least things that operate in a slightly different way. So yeah, I'll try to get off of my limited imagination a little bit to discuss things. That's funny, people throw around this sort of term, AGI. So your AI to AGI or artificial intelligence to artificial general intelligence. I know, I was looking up some. While you were talking, I was looking up some definitions that people kind of have out there for AGI. Probably a lot of them are fairly ridiculous.

Not surprising.

OpenAI always talks about pursuing artificial general intelligence. And of course OpenAI has been criticized in various ways, but I think that they're generally having good intentions in terms of how they're going about trying to push the state of the art and that sort of things. I agree.

And we need that. We need that. Yeah, definitely that aspiration.

Exactly. So no organization is perfect. But I think they do a really good job at trying to get people thinking about new things and creative ways of going about things. And anyway, on their page, their sort of about page, they talk about artificial general intelligence and what they mean by it is highly autonomous systems that outperform humans at most economically valuable work. I've seen other. So I saw other definitions about how like. Or that sort of horizon of where we're shooting towards is intelligence that is able to learn across, you know, human tasks at human level, which right now we're at a very sort of specific task oriented model stage where like I have a really good machine translation model that translates, you know, English to French or something like that. Right. Not in all cases, but generally it's very specific. Like that model is very specifically limited to that specific task. And it's not going to do anything else. Right. It's not going to sort of easily generalize to any sort of other human task other than translating English to French. And you know, of course there's multilingual models and all that. I totally realize that now. But I think in general, like the things that we focused on as a community have been those sort of task oriented things. I don't know if you have a thought on AGI and what that means, but I do.

And another, just to throw a couple of other buzzwords we hear associated with these, as I answer this is the kind of narrow, modern deep learning is often called weak AI, whereas the idea of AGI is strong. And so I think both you and I have had a habit of kind of scoffing over time a little bit at the AGI, we'll admit it. Yeah, yeah, we do. And Part of it comes from working in this field and that is, you know, as working people in artificial intelligence, the, the sex appeal quickly, you know, drops away, that it appears to be from the outside and you are grinding through, working with data and you are trying to create models and testing them and trying to fix it. And it's no different from any other type of engineering. You know, essentially you're getting work done. And I know both of us are also software developers and there's a lot in common. You know, it's a very bread and butter kind of thing, which most people don't think of AI as about. But when you're working in the field, it really is.

Yeah, I tell people, like, it's not like you stick your laptop in a corner and sprinkle some fairy dust over it and it becomes sentient and like starts learning. Right. Essentially what we're doing is a very dumb thing. We're doing trial and error, right?

Absolutely.

We have a bunch of data, a bunch of examples for a specific task. And even though the models are very sophisticated in the definition, in the way that people have defined the model from sort of input data to output data, at the end of the day it's a bunch of parameters that we're training by. Trial and error in most cases. Right. Not all cases.

Totally agree with that. And I think it's easy to lose sight of that because it is a very practical, hands on scientific job that we're doing here in its various forms. And so, you know, historically we look at AGI, artificial Generalized Intelligence, and we kind of go, oh my gosh, you know, robots that are talking to us the way they do in Hollywood movies, that's so far away. We all know that because we know and we understand exactly what needs to happen and stuff to get there in the sense of I shouldn't even say what needs to happen. We know that there's a lot that we don't know, I should put it that way. There's so many things that we're just not there yet on. And so it looks like a very distant horizon. But what we sometimes lose sight of as engineers working with the here and now of what we're of our tool set is that it is coming and we are actually very rapidly moving in that direction. There may be a long road from here to there, but we are moving along that road quite rapidly. And it's an evolutionary process and there's a whole bunch of baby steps that get us from here to there. And if you lose sight of the baby steps you're like, oh my God, that's science fiction. We're never going to get there. But we are slowly working our way. And in the short time you and I have been doing this podcast, you know, we're almost to 100 episodes as we're having this one. We're now into the 90s and the field has changed dramatically in the time that you and I have been doing this, this show. And so as we look at that and try to figure out where we're going, and we're getting a call from luminaries now about turning that page into the next things. So that was really what I wanted to start talking a little bit about on this episode is just give us a little bit of context for the future on where we would go next.

Changelog News is the best way to keep up with the ever changing world of software. We track, log and contextualize the coolest projects, the best practices, and the biggest stories each and every week. Make changelow.com your daily destination or hit the snooze button and subscribe to our weekly newsletter that hits inboxes on Sunday mornings. Join more than 15,000 enthusiastic readers. It'll cost you exactly $0, and you can subscribe right now@c chainsaw.com weekly.

So one of the things that I enjoyed about some of the material that I was looking at when you pointed me to the Neurips talk, and then I followed some additional links after that and was exploring things, is this idea of System one versus System two thinking. I think this is an idea that was developed in a book by leading economists. Thinking fast and slow. This idea of System 1 thinking, which are those things that we as humans think about or the tasks that we do that don't really require any sort of slow thinking, or we don't have to slow down to figure it out, right? Like, I have my coffee cup by my desk and I want to take a drink. So I just pick up the cup and I. And I take a drink. I don't have to think about, like, spend, you know, take a moment, get out the chalkboard, like, write out how I'm gonna think, how I'm gonna pick up the coffee cup, you know, write some papers in academic journals about my unique method for doing it and then do it. It's just kind of something that I just pick up. I don't even have to expend. Like, I have to have the thought to pick it up and I have to do the motion, but I don't have to slow down my life to think about how to do it. Right. That's true.

Thank you.

System two. Yes. Well, in most cases, maybe in System two, thinking, some of the words that were used as related to that in Yeshua's talk and in other articles were slow, logical, sequential, conscious, linguistic, algorithmic planning, reasoning. So this idea that there are those times in our lives as humans where hopefully most of us do think about more complicated things than picking up a coffee cup, we have to reason through certain problems to come up with a solution that is a sort of unique solution, maybe something we haven't experienced before. It's expressed in terms of, you know, maybe linguistic elements or, you know, logical steps or a sequence of things. And of course, some of these words, like the sequential side of things, the logical side of things, of course, these are associated with some of these ideas that, you know, like I mentioned, OpenAI and others are exploring. Like if you think of a sequential series of steps that you have to put some logic into and execute with sort of not that much feedback. Well, we're starting to think about, like reinforcement learning, which is a sort of sequential decision making process where you get rewards from your environment and you actually can modify your environment in some cases. And so some of these words definitely get to those things that people are trying to push the boundaries of. Was that idea of the different ways of thinking, did that resonate with you as well?

It did. And it's funny, that book in particular, I've had it on my Kindle for a while and I need to dive into it knowing I keep hearing how good it is.

You need to think slow about it.

I need to think slow about the book. There you go. And I'll read slow through as well. But we definitely are at a moment where we, we've done amazing things in deep learning over the last few years, but I think we're all pretty aware that it is clearly nothing like we act as humans in terms of the way we process information. And you named a lot of that is the idea of something that requires your attention and your consciousness, both of which are core ingredients for System two, as we define what those mean. Like with attention, it's the ability to focus on one or just a few elements at a time. And we've seen that now crop up in a variety of deep learning algorithms at a kind of a basic level. It's definitely in some NLP areas and you're seeing it pop up in others as well based on those successes. And so, and the ability to kind of have, you know, what they're Referring to as soft attention, which essentially allows you to focus on the things that you need, but they also evolve over time. And it's very much kind of encompassed by the idea of short term memory. And Yeshua notes that attention is an internal action and it needs a learned attention policy. And so, you know, all this kind of starts also feeding into consciousness in terms of, if you think of consciousness not as a wishy washy thing. And I'm just horrified by the way, as an aside, at how poorly I'm doing explaining this after seeing his amazing keynote.

You're doing beautiful.

Oh my gosh, I'm taking a master's work and just killing it. But he talks about the need for consciousness to be defined computationally. And so not to be this kind of wishy washy, ethereal idea, you know, that we think of as typically in our daily life, but something that you can identify algorithmically. You know, the nature of consciousness is very short term in memory. It requires attention. You look at something. And the fact that we are going from moment to moment there, if you put that into a neuroscientific context, that that is what allows us to do what we're doing right now, you know, to have this conversation and to do all the things that make us human in our daily lives. But we are approaching a time when we can, with a very strict definition, potentially define what consciousness is and understand how attention and consciousness relate in a neuroscientific and human standpoint. And that is what is being codified as the idea of System two in a machine learning context where we're starting to track that. So as I listened to that keynote a few months ago when he gave it in December, I was really struck by the fact that, that the science that we've put our careers into is really shooting along and we're not that far from certain areas that we may be able to computationally understand in a strict manner. And so that's, like I said, one of those first evolutionary steps that might take us toward that longer path toward AGI.

Yeah, I was kind of struck when he started talking about consciousness and then brought in this idea of attention. And I think that it's certainly useful to think about sort of how to build a machine or intelligence that is conscious of the things it needs to be conscious of to do more complicated tasks like these system two sorts of learnings and that sort of thing. So personally, and you know, this is partly my own opinions and partly my faith and my own values is that, you know, I think that that sort of smaller scale consciousness, which can be quantified, is still vastly different than what, you know, makes a human a human. And that's another discussion that we can have. So I think that there is, you know, human value that's separate from that sort of small scale consciousness that allows you to do these more complicated tasks. So that that's my own sort of thought on that. But I do think it is really useful to define some measure, whatever we want to call that, whether consciousness is a loaded term for some people or not, to define some sort of entity, which is that sort of entity that allows us to push machine intelligence to these sort of larger scale problems. I think that, like you say, practitioners who work in this every day, there's such a long road to intelligence that would even generalize to many of these sort of system two tasks that I don't think we're in any danger of these sort of apocalyptic scenarios that people like to think about. But I think it is useful to think about what is that entity that helps us build up these more complicated tasks.

I agree. It's a matter of fact, just as a note, the time horizon to get to AGI has so many steps between here and there, and it will likely be long enough out, whatever that time length is. I can totally understand why people would doubt that we're ever going to get there. And a while back, you know, I do this Atlanta deep learning meetup, and a while back we had dueling sessions where I was kind of arguing toward why we would get to the concept of AGI. Another friend of mine there took a different session and argued why that was just fantasy. And I still think I'm right on that. No surprise. But I can totally get why he would not think that that is a realistic thing. And that is because as we do what we do as practitioners, it is so many steps out, it feels infinite. It feels like that thing that you're never going to reach. And so I'm sympathetic to people. I think that the work we're doing right now and all the work we've done in deep learning, as different as that is from the neuroscience of a human brain, that it's still incremental steps to get us there. We are learning from what is working and what is not working, and we are learning from the unexpected side effects and things like that, and every little bit of that pushes us farther down that road. So I do think we're eventually going to get there. And I think there's giant implications which we can talk about a little bit later. In this episode about what that means and legal aspects and all that other stuff around it.

Yeah, I think that one other example, which I think good to talk about examples here too and make things concrete is I work with our chief research officer, his name is Gary Simons and he's just a really fascinating guy. He was actually the first linguist to take a computer into the field to do linguistics, which was like basically a suitcase computer that he built himself and then there was no word processor.

Right.

So he programmed his own word processor to do linguistics, you know, in the field in Papua New guinea on this computer. Like he's been in the game a long time and has continually just innovated over, over time. But you know, I was talking to him and showing him some of these things when GPT2 came out from OpenAI and like the text generation capability of those large scale language models and you know, paragraphs of generated text that were logical and made sense and like connected certain points. And again this is using this attention mechanism which is popular in these sorts of models and was referenced in the keynote at Neurips and all of that. And you know, he made the comment to me like he thought this sort of thing was impossible. Like you know, this sort of generation of text by a machine in this sort of logical way was something beyond what he would see in his lifetime for sure. And so I, I definitely think you're right that these sort of mechanism and that comes about by thinking of okay, you know, what sort of mechanism do we need to push things further? Well attention was developed and this, you know, self attention and transformers and that pushed those systems to this new sort of level. And so I think that there's going to be, continue to be those things that are developed and it requires a different way of thinking. So yeah, it's a good point.

We deserve a better Internet and the Brave team has the recipe for bringing it to us. Start with Google Chrome. Keep the extensions, the dev tools and the rendering engine that make Chrome great. Rip out the Google bits, we don't need them. Mix in ad and tracker blocking by default, quick access to the Tor network for true private browsing and an opt in reward system so you can get paid to view privacy respecting ads. Then turn around and use those rewards to support your favorite web creators like us. Download Brave today using the link in the show notes and give tipping a try on changelog.com.

So I, I wanted to talk about the other thing that we introduced at the beginning and that was the DARPA perspective and I introduced DARPA for the purpose of making sure everybody was on board with what that was. But it's really interesting to see them kind of looking out. And so it's a great lens, you know, to read up on what DARPA's doing and what kinds of solicitations and announcements they're making out there in the public space. Because you may be able to infer kind of where things are going over the next decade or two, but in doing that, because that's their mission. And it's kind of funny, you know, kind of the Hollywoodish version of darpa, as though it was another, like a spy agency or something. And, you know, I was reading Men.

In Black sort of stuff.

Yeah, really, there was a. I won't name the book. I'm not sure if I can remember it, but a friend of mine asked me to read a book and I stopped at the first chapter because the whole premise of the book was like darpa, which was like this spy agency was sending agents out into the field to do nefarious things. And I was like, I just can't take that. It's just, it's so far from me. What DARPA is essentially. And this is almost. It probably will offend them to dumb it down this much, but it's essentially a giant project management office. And, you know, their mission is to look at what are the next great technologies that are going to lead us into the future. And many of those things, even though they may be looking with a military view, end up out in the general population just as the Internet did. And so it really does affect our daily life eventually. And so they have a three and a half billion dollar annual budget, which is a nice hefty little sum of cash to go try to figure out the future with. And another really cool thing that they do is that it's impossible to be a career DARPA manager. They do rotations, and I forget what the exact time limit is. I think it's four years, might be three. But you never can bet your career on your DARPA performance. And the reason they do that is they want people to take risks. They want people to be willing to make big bets without it being something that will destroy their career. So it's the safe place to do really revolutionary work. And that happens. And they essentially, the entire organization's a bunch of project management offices and stuff. So the reason I say all that is that they have kind of. It's been. It's. It's actually not terribly recent. It was back in 2018, they released a PDF that's open to the public. Anybody can look it up called the three waves of AI and you can Google it and find it instantly. And it basically segregates the history, as they see it, of what AI is and is going toward and the three waves. And if you think of the past one is what they call handcrafted knowledge. And I'll talk about that in a moment. The kind of the current one that we're in is statistical learning. And then the future that they're talking about, the third wave of AI is called contextual adaptation. And I think that the short quick version of each of those is that if you have a set of attributes about how sophisticated you're getting with your AI in terms of perceiving, learning, abstracting and reasoning, and how far each technology can go that we're at. And if you think like an example of first wave, which was previous, before we got to the current deep learning period, would be like an expert system. And that's where you eventually capture a whole bunch of rules into a system and you have a flow that goes through them, but it's a rule based system. So it's not inference, it's not prediction in the way that we think of it today. It's not statistical, it's following a set of rules. And then, you know, we eventually got to this point that we're at now, where we've been in this amazing deep learning revolution of recent years, which they refer to as the second wave of AI. And in particular, things like perception and learning have come a long way up that we haven't made so much progress in abstracting and reasoning. But you know, the idea is that we are learning and we have been, even in the short time we've been doing this over the last few years, we've come a very, very long way in terms of what we can do with those statistical capabilities. But as we started this episode with, we're starting to get a little bit mature about, you know, where we've gotten to, and we're starting to see many versions of the same models and technology coming out. So what DARPA is really looking for are organizations, academic industry, within the military itself, that are interested in this idea of contextual adaptation, where you're essentially pushing all of those characteristics as far as you can and you are able to perceive, form a contextual model and learn abstract reason. And really all of those concepts that really lead eventually to the AGI concept that we've been talking about are seeing at least the next stage of realization there May be many waves after this potentially, but it's kind of taking us from where we are now in 2020 as we record this into the next some odd years of what this is likely to be. And so that framework, I keep coming back to that in my own professional life. And I think it's really important that we start recognizing that maybe we're seeing a fairly mature statistical learning marketplace, if you will, and people are able to put into production all these great deep learning models. But a lot of the really cool research, as we saw with that Neurops keynote, are now focused on what. What in this context would be third wave and what in that context was System two.

Yeah, since we're talking a lot about generalization and also perceiving an environment and making sequential series of decisions, so solving these sort of longer or thinking slow type problems, it seems like a lot of the time when they're referring to these things, my mind seems to go to reinforcement learning. And I know that that's something obviously that OpenAI is working on, but it also strikes me that reinforcement learning is not a new thing. Right. It's been around since whatever, the 1950s, I think, even as an idea. So I wonder like, if that's a big piece of this sort of third wave or system two thinking or whatever. However it's framed, if that's so valuable, why has it not pushed forward more rapidly into kind of standard practice and practical implementations? And it's still like playing Atari sort of scenario, and maybe not as practical as a lot of data scientists using it and that sort of thing. You think that's because of the models that are being used within the reinforcement learning framework? Or is it because attention wasn't placed on it or.

Well, actually, before we go too far, and I'll answer that, but we probably should note, you want to real quick define what reinforcement learning is for anyone out there that doesn't know?

Yeah. So reinforcement learning is where you have an agent and an environment and your agent executes what's called a policy to make actions in an environment. So if you imagine like trying to play a game, or if you imagine like you're trying to, you know, route a car through traffic from destination A to B. This is a scenario where you have a kind of goal or there's something you're trying to do, but it's not clear, there might be multiple sort of routes that get you there with the same sort of reward. And also the actions that you take actually influence your environment around you. Right. Like if I change lanes to my right. Then, you know, other people respond in traffic. Right. So you've got this agent which is acting in an environment which it can actually influence. And so at each time step of reinforcement learning, when you're performing reinforcement learning, the agent makes an action based on a policy that tries to determine its future reward. And then the environment responds to that action with a reward and a next state of the environment, and you kind of loop through this cycle. And so this is used in robotics and other places. But it seems like most data scientists I talk to that are even applying some of these more advanced AI models are not yet really thinking about reinforcement learning. I guess my question is kind of getting towards, like, if that's to become a more pillar of this sort of new way of thinking, is that just because we haven't been trying to solve those types of problems or the models applied within reinforcement learning weren't good enough yet?

Well, it's only my opinion, but my sense of it is that we're still early days on reinforcement learning. And this current incarnation, by the way, that we talk, to take any confusion away from people, is called deep reinforcement learning, where we're applying, you know, the traditional deep learning ideas, such as having, you know, a bunch of nodes that are connected and you have a, you know, like back propagation that is doing error control and applying that within this learning model, which we call reinforcement learning, which wasn't always had. It originally had nothing to do with deep learning. So we took something that existed.

Yeah, it's independent of what type of model you use in it.

And we've retrofitted it with deep learning capabilities in that, which gives it this statistical graph of possibilities which basically extends what's possible with the model, makes it a lot more granular. And I think we're very early days. I think that you and I have been in a unique position to see as part of this podcast, even all of the different uses that you could use deep learning for. You know, we had guests from Google that were using it in ways that I had not thought about. Obviously, there's robotics we just heard about. I recall there was a Pac man example where Pac man had been observed for a while, and then kind of was it Pac man or that was essentially reinventing the game from just observing it for a period of hours. But we're. I think we're really early days, and I think it's a transitional technology, a transitional type of model that takes us a little bit forward. I don't think it lines up perfectly with, like, the deep learning versions of NLP and you know, CNNs, which are kind of the deep learning version of machine vision. So this is one of those evolutionary moments where we're having a technology that helps us take, you know, half a step forward from where we already were and we'll keep building on it. So I think we're going to see a lot more happening in reinforcement learning in the years ahead. And at least that's my expectation.

Yeah, yeah, I definitely am looking forward to it. I know it was at a event in Chicago before all everything got locked down and in that event a guy was presenting about application of reinforcement learning in a marketing context to manage like how they executed their marketing campaigns in terms of do we use this campaign combination with this campaign and how is that going to influence next week's sales and then this campaign with that campaign and there managing like seven different campaigns and over weeks and weeks. And this is a sort of sequential thing they have to manage and figure out and simulate. And I found it interesting. So I haven't heard a lot of that sort of filtering into those and that was a more, I guess, practical industry application event and I was kind of surprised to see it filtering in there, which was really interesting. So yeah, there definitely are a good number of those use cases out there that people are exploring and I think they are starting to filter into the real world.

You know, there's one other topic I know we're getting short on time here that I thought we should talk about and I know you mentioned earlier that I believe we're at the. Is it the two year anniversary of gdpr?

Two years. Happy birthday.

So happy birthday gdpr. And we're in this wave right now of AI. Ethics is a really, really big topic in the world. And what does that mean? And if you look at what we've just been talking about these past few minutes and the vast difference between kind of today's AI and tomorrow's AI and then the day after tomorrow's AGI and what are the context of each of those means. So we are still very much lagging in terms of applying legal and regulatory constraint around that because we're still trying to understand it ourselves. And it's a, a very fast moving target.

Yeah. Even for wave two or system one AI.

Absolutely. We are way behind even in system one or wave two, as you said. And so and as that starts to evolve toward way of system two and wave three, we are still trying to figure out what that means. And that fast change makes it even harder. So it's a really interesting, you know, and that has a lot to do with how these technologies are going to interact with us. You know, it's the human actors that are being used side by side with, you know, all this working together as a system. And so it's going to be interesting in the years ahead to see how the evolution of human culture, be it laws, be it regulations, be it ethics, starts to constrain or shape the future of these technologies as they evolve. It's not strictly a technical conversation which we have a habit as engineers to fall back into.

Yeah, for sure. And I think that there's still so much that we have to explore on the technical side and solving of course the complex problems that are associated with these things. But if they aren't able to be applied in a way in the, in the real world where things are regulated and people's data is their data and you know, there's business problems to solve and non technical people that are involved and you know, all of those things, then, you know, it's strictly an academic pursuit. So I think you are seeing a lot of that friction these days with regulation. I think as we close up I wanted to, you know, just share. We always try to share some learning resources here in these fully connected episodes. A couple that I just wanted to mention. Of course we'll link to the talks and such that we talked about, but that book that we mentioned was the title is Thinking Fast and Slow. So if you want to learn more about that kind of different ways of thinking system one and two, it's definitely something that I want to look into and read a bit. The other thing I was going to mention, we talked a bit about reinforcement learning and I think one kind of fun thing that you can do on the side and also learn about reinforcement Learning is using OpenAI's gym to do some sort of simple reinforcement learning problems like the cart and pole problem or you know, racing a car around a little track or something. It's really easy to use the OpenAI gym that creates these environments for reinforcement learning. And TensorFlow has a tutorial with their TF agents framework that's built in TensorFlow to use OpenAI gem. PyTorch also has an example using the gem. So I would recommend if you're wanting to learn more about that subject, you can get hands on in that way.

So I have a learning resource of a very different type from usual within a particular reason why I'm recommending it. So one of the things when you work in the defense industry and you also give you know, a lot of conference talks, like I do prior to the COVID 19 situation, you're often asked about, you know, the concern of AI and autonomy and their intersection with weapons and war and things like that. And it's a topic that scares probably most people that think about it, I would imagine. And so. So I get asked about that so often that I wanted to point everyone to a public document that is, I would argue, a pretty good news thing, that if that's really something that you're worried about, about the future, as we talk about AGI and all these things going forward and what happens in our world, there is a document, it's called the Department of Defense Directive 3009. That's 3000.09, autonomy and weapon systems. And what it does is actually quite an old document. It's from 2012. But some really smart thinkers, before the day of deep learning were kind of thinking through what are the implications and constraints that need to be applied to autonomy. And so you can go out and Google this. It's publicly available from the Department of Defense, and if you read through it, you realize they have handled so many of the use cases before we ever got to this modern day and age. You know, this is legally binding. This is what governs us in the United States, at least about how we do this stuff. And the first time I read that document, which was coming into this career I met, I had a sense of relief that smart people had come along before and been very good about thinking. So if this is a topic, it's a little bit of a dry document, I'll confess, but it has some really interesting things in it in terms of how we are keeping people safe. And it was one of the very first things I had to learn. And I didn't think it to recommend it a while back, but I keep getting asked about that over and over. So if you want to understand how we think about that, it's a really good document. And I suspect, despite the name of the document, you'll go to bed sleeping a little sounder, knowing that this is the reality. So I just thought I'd offer that. I know it's a little bit unusual selection.

Awesome. Good pick. And thanks for putting up with all my opinions and cynicism and optimism and, you know, all the emotions that went into today's episode.

No worries. Yeah. If we're going to really be practical, we have to keep our eye on the future.

Yeah.

And so sometimes we got to set TensorFlow and PyTorch aside and figure out what direction we're going. This was a fun talk today. Thanks for doing it.

Yeah, for sure. Talk to you later.

Take care.

Thank you for listening to Practical AI. We appreciate your time and your attention. Word of mouth is the number one way people find new podcasts. If Practical AI has helped you on your AI journey, please do tell a friend. Hey, they'll thank you later. Special thanks to Brakemaster Cylinder for the Beats and to our awesome partners for their support. Shout out to Fastly Linode and Rollbar. If you and your organization would benefit by speaking directly to the AI community, you should sponsor Practical AI. Podcast advertising is highly effective and we would love to work with you. Head to changelog.com sponsor to learn more. That's all for now. We'll talk to you again next week.