AI-driven studies of the ancient world and good GANs

Bandwidth for Changelog is provided by Fastly. Learn more@fastly.com we move fast and fix things here at Changelog because of Rollbar. Check them out@rollbar.com and we're hosted on Linode Cloud Servers. Head to linode.com changelog this episode is brought to you by DigitalOcean, the simplest cloud platform out there and we're excited to share. They now offer dedicated virtual droplets and unlike standard droplets which use shared virtual CPU threads, their two performance plans general purp CPU optimized They have dedicated virtual CPU threads. This translates to higher performance and increased consistency during CPU intensive processes. So if you have build boxes, cicd, video encoding, machine learning, ad serving, game servers, databases, batch processing, data mining, application servers or active front end web servers that need to be full duty CPU all day every day, then check out Digital Ocean's dedicated virtual CPU droplets. Pricing is very competitive starting at 40 bucks a month. Learn more and get started for free with a 100 of credit at.co changelog again.co changelog. Welcome to Practical AI, a weekly podcast about making artificial intelligence practical, productive and accessible to everyone. This is where conversations around AI, machine learning and data science happen. Join the community and slack with us around various topics of the show@changelog.com community. Follow us on Twitter. We're @practicalai FM. And now onto the show.

Welcome to another Fully Connected episode where Daniel and I keep you fully connected with everything that's happening in the AI community. We'll take some time to discuss the latest AI news and we'll dig into learning resources to help you level up on your machine learning game. My name is Chris Benson. I am a Chief AI Strategist at Lockheed Martin, focusing on artificial intelligence, high performance computing and AI ethics. And with me is my co host Daniel Whitenack, who is a data scientist working with various NGOs and nonprofits. How's it going today Daniel?

It's going great. Great to talk again after the the July 4th holiday and get back into all AI goodness.

Lots of good family stuff, right?

Lots of, lots of good family stuff. Lots of, you know, family stuff is always good and great and awkward and all of those things at the same time. But it was great to see family and have a couple days off. So what about you?

It's pretty much the same as you know, I have all these dogs, so for us, fireworks, you know, twice a year on the July 4th holiday and at New Year's Eve we try to have a little bit of fun early in the day and then hunker down with. With terrified dogs.

So, yeah, that's rough.

Yeah, it was our usual. It wasn't bad and I'm excited. As we are recording, I'm about to head to Boston to participate in the developer workshop for Alpha P, which is a venture that Lockheed Martin is partnering up with Nvidia and MIT and the Drone Racing League, where we have teams from major universities all over that are coming together and they're going to be starting the process of creating various neural network models that are going to drive some really serious drones around a race course at high speed. So I'm going to get a chance to meet a whole bunch of smart people this week. Really looking forward to that process. It should be a lot of fun.

That sounds really exciting. Is there a place people can follow up with that? Do you know if there'll be like videos or recordings of this stuff going on?

As far as this one, this is kind of the developer workshop and we'll see about what can come out. There's going to be a whole series of Alpha pilot drone races going on through the end of the year and I'll put a link in the show notes where people can kind of connect in if they're interested. But, you know, every time I think about this, I'm sure the people doing it will roll their eyes. But you know, in that Star Wars Episode 1, when the young Anakin's flying around through the desert, that's the thing that always teases my brain. I think it's a little bit more down to earth than that. But I'm looking forward to having fun with my version of Star wars this coming week.

Yeah, yeah, no, that sounds like a lot of fun. It sounds very exciting and it's always cool to see AI. You know, a lot of times we're just focused on like, I'm staring at my monitor on my computer and doing quote, unquote, AI. But it's cool to see like AI and machinery of some type interact, whether that be robots or drones or little cars or whatever it is. So that's. That's always fun and kind of connects it to the real world. I think it does.

I mean, it's going to be interesting to see this as AI that most definitely has a tangible, real world impact, you know, that people can enjoy as sport and as televised. So looking forward to the start of that process this week. And you know, you and I had not done a show where we're really focusing on current events in a while, and as we were talking, we decided, let's do that this time around because there's been so much news out that we have kind of gone right by in the weeks. And so we're hoping listeners will sit down, enjoy us as we kind of go through this process of talking about some of the more interesting things we found in AI news lately.

There's been a ton going on. I feel like we've done the topical shows which have been really good and I'm glad that we've kind of, you know, done that and got gotten deep in some of the subjects. But there's just been so much accumulating in the news. I think it'd be great to, to cover some of those things now for our listeners. There's been actually so much going on in the AI community that it's really hard to narrow down into a number of topics that we can discuss in this format only in 45 minutes or whatever we have. So we've done our best to highlight a few things. But if you find anything interesting that you're, you're interested in or involved with in the AI community, please share that with us on our Slack channel. You can find that@changelog.com community and, and, or on our LinkedIn page. And we'd happy to. We're happy to hear about what you're doing and also maybe feature some of that on the show in the future. We've done that in the past and hope to do it more in the future.

All right, well, you want to kick us off with first article that you had brought to our attention.

Yeah. So this is actually like a series of articles that I've seen recently. I tried to think about some trends that I've been seeing in the AI community. And one of the things kind of this spring, moving into like very recently a trend that I've seen starting to develop is various AI techniques that are impacting studies of the ancient world. So I'm thinking of like things like studying ancient languages or archaeology or preservation of artifacts or making discoveries about the ancient world. There's been various things that are happening where the AI community is kind of intersecting and worlds are colliding with archaeology and linguistics and other things. So I saw a couple of these things recently. One that seems to have gotten a lot of attention was a study. So I read about this in the MIT Technology Review, but there's an article on the archive as well, and the title of the article is Machine Learning has been used to Automatically Translate Long Lost Languages. Which seems pretty cool. Does it seem cool to you, Chris?

It seems Very cool to me. There's so much happening. We keep calling out how much is happening in NLP these days. And that is natural language processing for those not familiar. It's just been huge over the last year.

Yeah. And this in particular, it's like all of those recent discoveries kind of mixed with Indiana Jones a little bit. So it makes it even more awesome, I think. But there was this study. So apparently there are these ancient kind of lost languages and I guess there's a number of them. But archeologists have discovered kind of tablets and other things with these languages that date back to various, various times. In fact, there's this kind of language called Linear A apparently which dates from like 1800 to 1400 BC. And then there's another one called Linear B which dates, you know, along the same time period. And linear A actually hasn't, I guess hasn't been deciphered, but linear B has been deciphered. It was deciphered only like, I forget what the article said not that long ago. But the article basically shows how these researchers, which are from, let me see here, from MIT and Google, so Google Brain and mit, what they did was they basically showed how they could utilize similar words or similarities between various languages and how they were historically derived to kind of automatically decipher this Linear B language, which is actually a lost language. And they showed that they could basically do it in a way that didn't rely on parallel data. So it's a real deciphering of this language, which is pretty cool.

That sounds really neat. So they're not basically looking for common words for lack of a better description. They're almost common approach to how language would be formed at the time.

Yeah, so there's this idea of like cognates which I guess, and I'm not a linguist, so I apologize to all linguistic people out there where I'm butchering things. But these are kind of words that look similar in multiple languages. So you're probably familiar with this definitely in English and other languages. But they utilize these kind of similar looking words, I guess, or these corresponding cognates. And then they do a sort of character matching thing. And they have a pretty cool diagram in their paper where kind of lost language characters come in and they use like lstms and attention and softmax and kind of, you know, known characters come, come out. And so they do this kind of translation between linear B, which is this last language, and Greek based on these also utilizing these corresponding cognates and this cognate mapping sort of stuff. So they don't really use, you know, parallel, they don't need parallel data between linear B and Greek. They're really treating it as a lost language, but they're utilizing these similar looking things.

That seems really cool. I love the fact that there are so many applications that we're able to apply neural network technology of today too. It's impacting just about everything you can imagine. I can now going back to your Indiana Jones, I can imagine Indiana just sighing wistfully, putting down its whip and sitting down at his laptop to get the real work done.

Yeah, and there's this other article which I think very much fits within that framework. There's this other article which I actually also read on MIT Tech Review, which I guess I get a lot of things from there, it seems like, but this has to do not with language, but with games. So in a lot of. So I guess there's medieval manuscripts that include these representations of games. Whether that's like games played on horseback or games like played like table games or dice games or these other games. And up till now there wasn't really, I guess there wasn't really a field of like study around ancient games. And so there's this researcher in the Netherlands who's kind of proposing that there's this new field of study of ancient games which I probably going to mispronounce this archaeology. And anyway, he has these pictures of these ancient games that are represented in manuscripts. And he's proposing and has shown some evidence that using machine learning and AI, we could use things like computer vision, but also evolutionary algorithms and other things to actually learn about how these games evolved, about their rules and even like take the whole of games that have been, you know, developed through time and use their kind of basic elements to generate new games. So there's also this idea of using all of these basic elements of games through history and then using AI to take those components and generate new games that we might, I guess, that we might want to play. I guess that's the goal.

Yeah, I flipped up the webpage that you're referring to and like they have a picture of kind of an image of, of an old game with, you know, that looks like a plus sign sort of with various squares on it and images. And then they have what I assume is a modern version of the same game. So it seems. Let me ask you this, I'm kind of curious. There seems to be a little bit of commonality potentially between these two. I mean, in one case we're, we're talking about looking at Old games to try to derive, you know, how they came about, what their rules are. And yet we just talked about, you know, lost languages as well. And going back, would you consider that to be somewhat similar approach as NLP being used in both areas?

I think it, I think maybe the commonality is maybe even a little bit higher. Like, I know one thing that you're going to talk about later in our chat today is unsupervised learning. And I think the connection that's happening in a lot of different areas of AI is really the kind of transition to these semi supervised methods and methods that kind of start from some data, but not really labeled data or annotated data and try to learn something. So in the last language case, they're really starting with text or characters from these languages and learning and deciphering these lost languages. In the game's case, they're starting with these images and maybe rules and they're kind of doing some sort of evolutionary thing to learn about how those evolved or to generate new things. So I think we're definitely seeing a lot of that all over the AI community.

So I'm kind of curious, do you think that there's any particular work that is interesting to you at this point that this might be applied to in other areas? Like if, if you were, and I know you're deeply involved in doing work with languages in your day job, what other areas do you think may this be very applicable to in terms of going and rediscovering things from the past or taking something we're already doing and thinking about what's next?

Yeah, I mean, I was really excited when I saw the Lost Languages article. I think that I only have a certain perspective. So I'm sure there's, there's other things that are happening. But you know, in the, in the language space, I mean, there's a ton of languages where we don't have good machine translation technology, but there's also a ton of languages where, you know, these languages exist and they represent a certain view of the world and they're endangered. And you know, UNESCO has said like, we need to preserve these languages and other organizations have said that. But, but a lot of these languages don't. They may not even have written languages or written scripts or anything like that. They're just spoken languages. And so the fact that we could maybe use some of these techniques to document languages, to learn about languages or even sign languages using computer vision and other things, all of those are, are pretty exciting to me.

That sounds pretty cool. Foreign.

The Data Engineering Podcast is a weekly deep dive on modern data management with the engineers and entrepreneurs who are shaping the industry. Go behind the scenes on the tools, techniques and difficulties of data engineering so you can learn and keep up with the knowledge to make you and your business successful.

Can you give a bit of an outline about the motivation for choosing Jupyter.

Nopix in particular as the core interface.

For your data teams?

Yeah, and actually when I first joined Netflix, it was sort of tossed at me and I was definitely like, why are we crazy? And the answer was like, we might be a little crazy.

Go to dataengineeringpodcast.com to listen, subscribe and share it with your friends and colleagues.

So we were talking about a few minutes ago about kind of NLP techniques being applied in particular to deriving the rules from, you know, kind of ancient games and such, and I'm going to take that as a, as a rough segue into some of the things that are currently happening in games. There were two things in particular that I noticed. One has to do with DeepMind and there is an article that Ars technica has called DeepMind AI is secretly lurking on the public Starcraft 2.1v1 ladder. And you know, over the last few years talking about DeepMind and Starcraft and that link in terms of, you know, them having Starcraft being played by DeepMind a few years ago and now they're kind of doing a second version of it. That's, that's really interesting. In that article they're talking about the fact that they're having, that they're having AlphaStar playing it, play the game in an anonymous fashion and they've done a bunch of limitations on it to make it more like the way the human would play it. Some of the things that they had noted in it was the fact that the last time around DeepMind was able to utilize the entire game all at the same time. So it kind of had an unfair advantage over the humans in that it could do anything and see everything at the same time. And one of the things that they're talking about now is they're talking about limiting it on what it can do from an action standpoint in terms of acting. They're limiting it so that it essentially cannot out act humans. And they're also focusing really on the camera view for the game instead of having that kind of holistic view of the game. So I thought that was really interesting in that as they implement these characters, you may or may not, when you have a group of people in the game, you may be playing with obviously humans, but you also may be playing with non humans now and have no idea. What do you think of that, Daniel?

I think it's really interesting. I wonder in the game, so other than maybe like the screen name or something, I wonder if there's any way like if they're making it known that this is bot or anything. Do you know what they said in.

The article was explicitly. No, they were deliberately doing it anonymously and I don't know, I mean if you were to say, you know, if you were to communicate and say, you know, are you a human or are you a bot? If they're just going to lie if the, if it's a bot and says of course I'm human, you know, what a stupid question, that kind of thing. I don't know how they're handling that.

Yeah, I wonder what the perception. I don't know. I'd be interested to like read forums and different things around the perception of this because I definitely think there could be that. You know, I have brother in laws who are really into gaming. I'm not quite as much but I know, you know there's that like when you're playing especially like ranked games and that sort of thing like you know, if you're playing against someone that has an unfair advantage like you were saying, you know, I know that they've tried to limit that but that could not come off so well. Like especially if this player is moving up in the rankings or something like that. I don't know that that would be accepted very, very well.

Yeah, it explicitly was, was noting you know, the fact that they're really putting all those restrictions on and they refer to the previous time around a couple of years back as an unfair unrestricted view of the game. And so as they do that I find that fascinating first of all, just because gaming is obviously a great way to do that. But there's so many things I could see moving beyond the gaming world that are out there in terms of general use cases. We keep hearing that we are at this moment where we're going to start interacting with different types of, of AI models as just a standard course. And the first thing that comes to mind is medicine. And I know we've reported in the past about that you'll have bots as your primary care physician within the next few years, like less than five years from now. And so if they're able to insert these AlphaStar driven bot players and they're indistinguishable from the regular player in terms of how you're interacting and what they're capable of doing. Once again, it really brings us into that world world of that collaboration between the human and the AI that we see going forward for some period to come at least.

Yeah. The one interesting thing about a lot of these video games that they're getting into, and I know this was a topic of discussion, I forget with which video game they were talking about this, but the fact that these video games, there's multiple really interesting problems to solve from the AI perspective. So if we think about a game that has like character selection, that's like one thing that needs to be solved that like a human would normally do. There's also like the gameplay elements, but then there's even other things. Like if, if you really want this agent to behave like a human in a game, there's also like the chat and interaction element. Right. Like especially in a team play game, if this, if this, if an agent is playing as a member of a team, there's communication between the team often as well. So. So this intersects with a lot of different areas. And I would actually be pretty surprised given the state of conversational AI and other things. Like if, you know, really an AI covering all of those elements at the same time is really in our near term future. But maybe it is. I might be too skeptical.

Yep. I think I got to say I would love to have somebody from DeepMind who is directly working with this AlphaStar implementation onto the show and just talk through some of the details of this, but it feels like the implications of how this could be used. This is a great test case, but it really feels like we're turning a corner at this point. And I guess while we're mentioning turning a corner, there was another article that I ran across that has to do. The article is on polygon.com it was entitled An Unbeatable Poker Bot Offers Glimpses of video game AI's future.

And really I saw a lot about this on Twitter.

Oh good.

People were talking about this a lot.

What did you hear? Before I leap into my part, I'm kind of curious what the social media side said.

So I don't remember exactly what they said and I didn't look a ton into this, so you'll have to brief me. But I just remember seeing as I scroll down on my Twitter feed a bunch of gifs with this like flowchart of how this thing worked and like poker imagery and all of this stuff. So it seemed like something that really caught people's attention.

Yeah, the thing that the article really raises is that they kind of took a different approach with this bot. And so instead of the model kind of having the long term, you know, strategic game view, they limit it to thinking only, you know, two or three moves ahead. So once again, it's, in a sense, it's almost coming down to some human limitations that we have. But the thing that really jumped out about this was they trained this bot to be able to bluff effectively. And what I didn't know, and I'm.

Not generally, what does that mean in an electronic sense? Like it has no face, Right?

Yeah, actually it's a good question. I'm not sure. I think what they were kind of talking about in it though, was that I think it's just based on the action. So that even if you're not sitting around a table seeing faces, whatever someone out there is saying, whether they're holding or doing whatever, and I'm not a poker player, so I don't want to humiliate myself by trying to jump into that. But basically it's indicating that by only looking two or three moves ahead, it changes how bluffing would occur. And there was something in the article saying that if it had taken a longer view of the game, the way it would bluff would be different and it would be distinguishable. It's different from human action because it's taken a different view than what humans would do, which are typically looking just a few moves ahead. And so this could accommodate. It could bluff. It could accommodate what humans that are playing against it are doing, whether they're bluffing or not. And so I think that was what really made it stand out as a different type of model from some of these that we've seen before and reported on in some cases. But yeah, so we're teaching AI to bluff its way through, to lie.

Interesting. Is this like active in some type of, you know, some type of online poker play site or is it just kind of an experiment at this point?

I didn't see it. I'm looking back through the article right now. I didn't see it being deployed. So I'm assuming it's in a very controlled environment. But. But who knows? You know, sometimes I go out to Vegas for various conferences and such and you never know. You may walk into a casino at some point and be playing against the bot, or at least one of the one that'd be one of the players up there. That's the new gimmick.

Yeah. I'm also struck in the article I was kind of reading through as you were talking and they say that there was also like an advantage they saw. So they, they trained the model, I guess on a 64 core server with less than 512 gigabytes of RAM, saying that the cloud servers to train up the program would cost only $150. So I think that that's a really interesting point and consistent with some other things. So I've seen of course, fast AI and Rachel Thomas and other people talking. I think it was her. I saw it recently with a slide about, you know, how to do innovative AI things. It doesn't, you know, requiring a lot of compute isn't always necessary and we shouldn't have that really as a precursor in our mind. And I really like that both because, you know, there's a lot of people in the world that just don't have access to train a model on a TPU pod and spend like $7,000 on compute costs to train one model. So I think that that's encouraging that there's some cool stuff like this that's happening with very little cost as far as compute goes. But also of course, like, you know, these huge large scale NLP models and other things happening these days, like the compute cost is pretty ridiculous and also like the environmental impact of that is massive, which is kind of depressing. That like.

Yeah, I saw that as well.

Along with Bitcoin. We're just like, you know, destroying, you know, racking up all of these energy costs because of these models that we're training, which I don't know that that's like. I forget if there's like a study that shows the total percentage of our, of our energy usage or something like that, but it's definitely, you know, something that's contributing a lot and in some cases more than it really has to be because we don't always have to throw a TPU pod or a V100 or something at a model. I think there's a lot of interesting things that we can do with limited architecture, which is something that I'm really interested in exploring for sure.

Totally. And I'm going, I know we're sharing articles, so I'm going off script slightly, but I've seen the same kind of thing about the amount of compute being just kind of crazy levels in terms of environmental damage relative to the benefit we're getting. I was talking to, and I don't have any of my notes in front of me, but I was talking to a professor, I'm in the Atlanta area at Georgia State who had been Working on basically taking complex models and finding the key important nodes in them. He has a technique where it goes through that and it can kind of compress a model way down. So you can take a really sophisticated model and once it's trained at least to inference much more cheaply than you would have been able to before. And I know that there's that, that he's doing and there's a lot of other similar initiatives to try to bring the cost of compute down. And at some point it might even be worth doing a show on that is pulling some of these different techniques together or talking to some of the experts on this. But I do think that that is a giant area of research that the world at large is focusing on is being able to do effective of inference at least and maybe training without such a catastrophic cost.

Yeah, you make a great point that there's that cost. But also things are moving towards us running AI on more like in the browser and on edge devices, on mobile devices. And at least for the foreseeable future, those devices are limited hardware wise. So I think that privacy wise, it makes sense to run those things things in a more optimized way on edge devices in a lot of cases. But also, you know, we should, we should be careful that we're not using all of this compute as a, as a crutch in some ways. I was just listening to our partner podcast, the Changelog. They had the pragmatic programmers on the, on the show which there's this pragmatic programming book which is quite famous and they were having an interesting discussion about hardware limitations and some things like that that you know, just in some ways are, are good and some ways are, are bad. And I've talked to, you know, a lot of more experienced programmers that are really glad that they were forced to consider some of those things around like memory constraints and other things like that. Whereas now we kind of take it for granted that if we want more, if we want more power, it's just there in the cloud. And maybe that's not always a great thing.

No, we do tend to get very tunnel visioned in terms of, you know, we're, I know that we have this bias where we're so convinced that, you know, this is world changing stuff and amazing things can come of it and all and we tend to forget that, boy, we're using a lot of electricity there in the background. So yeah, we need to have a better, better, more holistic view of the world in that way.

Well, hello there listeners, how are you? This is Adam Stacoviak if you haven't heard yet, we're launching a new show called Brain Science. It's a podcast for the curious. Are you curious? Because if so, we're exploring the inner workings of the human brain to understand things like behavior change, habit formation, mental health, and what it means to be human. It's brain science applied not just how does the brain work, but how do we apply what we know about the brain that can transform our lives. Learn more about the show and subscribe@changelog.com BrainScience until then, here's a preview of episode number two. We're talking about how we're all designed for relationships.

I think about it like scaffolding that as our kids grow and anything and it doesn't matter, I just always want people to have this sense of hope and optimism around. Like, look, it's not over if you didn't get it in childhood or it didn't fully grow. Neuroplasticity is one of the most amazing and hope filled things because we can continue to build this and grow all throughout our lives. And so having another person participate in the development of our own mind, it's sort of helping build neural networks that say, hey, I totally understand that you're upset as a teenager because you did not get ice cream and you think your world is now ending. But to actually, you can still empathize, but that doesn't mean you necessarily give them that desire. Right. Because I don't want them to be conditioned, I. E. I don't want them to have the perpetual feedback that when they're upset that they just get to have the ice cream that they want.

Right. Let's also say we're using children as an example here because for the audience to empathize with us, that's our breeding ground for research. Basically. I can give an example where my son, I can't recall the exact scenario, but there was a moment where I was like, to my wife, I said, hey, it's not that he's misbehaving because we were both sort of in this crazy mode with him and he wasn't behaving. And I was like, you know what, it's not that he's misbehaving, it's just that he, he can't right now. He's just too far gone, he's too tired, he's too exhausted, he's overstimulated and his brain is just not developed enough to really get that we're asking him to behave and desiring and expecting him to, but he's just not capable. So that moment, we both sort of just curled into ourselves and just cuddled him and just was just, you know, loving to him rather than like, why can't you get this? Come on, three Nager do this.

You know what I mean?

Our breeding ground in research is our children.

Right? Exactly. And in my line of work, I mean, I will see the people where this sense of attachment and connection and feedback loop didn't go so well. And so they've learned. I always say it's sort of like they jerry rig things. Like they learned how to best function in their lives as well as they could. But we, we know this. Whenever we jerry rig something and don't actually fix it the way it was supposed to be, what happens? It breaks down.

Well, if you like what you hear, you should go to changelog.com brainscience the show is not out yet, so don't get too excited. But you can subscribe and be notified as soon as this show launches. Once again, changelog.com BrainScience.

I wanted to follow up on one thing that we talked about in a recent episode and a fully connected episode actually, which was deep fake. And I think in that episode, and of course in general around this topic, there's pretty much a negative view of this technology. You know, GANS are bad and like they're destroying everything that's real in the world. Sort of, sort of viewpoint. And in that show we talked about like, oh, well, what are some of the potentially beneficial things that this technology could do? And I remember talking through a couple things. We weren't like, totally sure on what those were, but I saw an article, a recent article that took a really good perspective on this. And I remember it. I forget what the article was titled, but I remember seeing it mentioned, like, good GANs. And for listeners that, you know, want more info on, on GANS or regenerative adversarial networks, you can listen to our episode on, on that. But this article is basically saying that a research group is using GANS and kind of the technology behind deep fakes to actually improve cancer detection. So the detection of tumors and abnormalities in X rays and CT scans and MRIs. So the basic idea is that you would take kind of existing imagery of cancerous tissue and you only have so much of that. Right? There's only so much data that exists in a nice form. And so your models are kind of limited to that data. And similar to what we were talking about at the beginning of the show, people are much more considering Kind of semi supervised and unsupervised methods intersecting various things, in this case, computer vision. So what this group did was to use GANS to actually generate new cancerous tissue imagery in the same sort of style to where it like, looked pretty much like what a different cancerous imagery would look like. And so they're kind of using this to create this new data set of simulated data that can help them in training models for what potentially cancerous things will look like. Which I think is really cool.

I do too. And, you know, I'm glad you mentioned that because I think going into that episode where we talked about deepfakes in particular, I know for me, I had just finished watching the congressional hearing on deepfakes and the danger from a national security and all that stuff, and I think I was definitely in a dark place mentally in terms of going, whoa. And then we sat down and recorded. So apologize to listeners if that was a very ominous show. GANS in particular, I mean, not only, you know, you're talking about the medical application there with cancer, and I think I actually remember running across that article. But, you know, in the past we've talked about lots of different use cases, including, you know, the various types of creativity that can arise from it. You know, we certainly talked about that. Christie's had auctioned.

You can get rich with new artwork.

Yeah, absolutely. You know, I don't know. I don't know why we're not focusing on that, considering that.

Yeah. What are we doing, Chris?

What are we doing, man? We could be making a ton of money with GAN based artwork.

So I guess maybe it's more meaningful to cure cancer. Although I guess, you know, there's no cure for cancer, but to help treat cancer, I.

You know what? Okay, I'm going to give up my greedy. My greedy fascination with AI based artwork.

Which one is better?

Yeah, so. But it's interesting as we're seeing these. These leaps ahead. So I think the GANS are going to be a big area of growth over the next few years. And I think that, you know, while some deep fake use cases are certainly on the negative side, I think we'll also see deepfakes that are being used in pretty creative ways. It was pointed out to me after that episode, I was talking to somebody at the Atlanta Deep Learning Meetup and they were pointing out that gaming and that there's lots of educational uses where you change up use cases to teach somebody kind of dynamically on the fly. And there are tons of good potential uses as well. So it may just Be that we've had our attention draw to some of the early nefarious things where bad actors have grabbed onto a new technology and done and done some naughty things with it. So I think we'll see some pretty good uses of the technology as well, aside from the kind of things that we covered in the previous show.

Yeah, I wasn't really thinking as much about like simulated data when we were talking about that, having that discussion. But now that like this, I read through this article and thought through it, that's kind of at the forefront of my mind around this topic because the idea of simulated data similar to things we've even already talked about in this podcast is really valuable as we kind of move into this phase of really focusing on semi supervised and unsupervised methods. And I think one of the things that you also found this week had to do with unsupervised learning. Is that right?

Yeah, so let me flip over to that one real fast there also an MIT Technology review. Wow, we've done a lot of stuff.

Yeah, it's an MIT Technology Review week. So plug for. Plug for the great work that they're doing, which is apparently giving us all the information that we're looking at. Maybe not all the information, but it's certainly they've been on a roll.

They have, we keep referring to them here. So good job to that group there. One of their articles was called the AI Technique that could Imbue Machines with the Ability to reason. And in it they had talked to Yann Lecun who is Facebook's chief AI scientist and he has for some time been talking about the fact that, you know, the field at large is going to find, you know, the future great, great evolutions of the field in unsupervised learning as opposed to supervised learning. And he, he mentions, I'm scrolling down to it, that you know, if we are going to think about, you know, kind of how the way humans learn and the fact that, you know, we, we start off being born with a very limited set of knowledge obviously coming out of the womb and that babies are really little of what a human learns in their process is through supervised learning. And he also pointed out that very little is done through reinforcement learning in the scheme of things and that it's really self supervised unsupervised learning, meaning it's self regulated. Unsupervised learning is how the vast majority of human knowledge comes about. And I gotta say that I agree with it. And I actually got interviewed myself a while back by Thomas Reuters. And I was saying essentially the same thing is that some of the really tough things that we have in this field, such as having the data that we need to train with and accumulating it, you know, are much harder than the actual AI world work itself that we're trying to do. Unsupervised learning presumably gives us a whole world to start exploring out there without these giant obstacles to overcome before we ever get started.

Yeah, I'm actually working with an intern this summer and we're working on a sort of language related graph based method. But anyway, we're kind of, what is this, like halfway through the summer? I guess at this point?

Roughly.

Yeah. And you know, we're still on the data issues of the problem and I'm sure everyone that's actually worked on these problems can sympathize. Those are really the bulk of the issues is often the data side of things. Thinking about another conversation I had with someone just yesterday where we were talking about, you know, how we basically just all like lost any confidence in any data that we ever receive or confidence in having good data because like, regardless of the process behind the data, like even if there's a good process in place, humans are generating data and they're like biased in how they do it. And so almost any data set that we get, even if it's like nicely formatted, I kind of just lost confidence in all of that, which is kind of, it's kind of depressing, but it's also kind of encouraging that people are really saying at this point, well, maybe, maybe we should focus on that less and focus more on this sort of semi supervised or unsupervised approach.

Yeah. Before turning to that, I'll agree with. I've been spending a lot of time focused on AI ethics lately at work. And I think it is a fair statement to say that an enormous amount of data out there is inherently biased in various ways. And so we're certainly going to need to address with tooling how we approach that. We can talk about more of that later. I know we're gonna have that as a topic in the future.

Yeah, biased or even just like bad, like in some cases data sets, I've got like it, and it looks good. Like there's a field labeled this and it, like it's a good name for the field and it's like, you know, language identifier or something like that. I'm like, oh, that's what this field means and that's how everybody used it. But then I find out like later on, oh, that didn't mean the same thing to everybody. And they input all sorts of. Of crap into that field. Right. So it's like, it's bias. It's like bad data. It's like everything's messy. So if everything's already messy, like unsupervised methods and these other things, maybe that's where we need to put more focus.

Yeah. And as Lacune points out, he notes, and I'll quote, he said, obviously we're missing something. A baby can develop an understanding of an elephant after seeing two photos, while deep learning algorithms need to see thousands, if not millions. And a teen can learn to drive safely by practicing for 20 hours and manage to avoid crashes without first experiencing one, while reinforcement learning algorithms must go through tens of millions of trials, including many egregious failures. And he has a great point there. We have these amazing biological models that have inspired us, but we're actually not very close to them currently in the vast majority of work we're doing. And maybe some of the work that comes out of that will lead to some pretty big breakthroughs. I'm rather hopeful there.

Yeah, it's good to be hopeful. Great point to end on. As we do in these fully connected episodes, we like to go through some of this new stuff and updates and all of that good stuff. But also part of what we want to do in these episodes is point you to some good learning resources that we run across. So. So as you're trying to learn about these methods that we're talking about, I'm always looking for new things, new ways to learn things, because there's so much information and so many methods to learn. So we'll spend the last couple minutes here just going through some good learning resources that we've found recently. Chris, have you found anything recently that's been interesting either that you've gone through or that you're interested in exploring a little bit more?

Yeah, so one that I saw was Google Cloud on their blog, had a blog that is deep learning containers. And then the subtitle says, build your deep learning project quickly on Google Cloud, obviously. And what they're really focusing on there is if you're in their ecosystem, and obviously there's the pool that each of the major providers has that they have these number of containers that are Docker containers that give you complete environments, and the purpose there is to be able to have a standardized environment. And I know the other providers are doing that as well, and I really wanted to call that out. Is that standardization on the same thing that the software development world is on Docker containers kind of becoming the de facto atom of work, if you will, about how people are getting things done.

Everything cloud native.

Yeah, really. And you can pull them onto your laptop and have the same environment. In this case, Google is just talking about the fact that they have a bunch of deep learning containers of various configurations with different frameworks and different configurations that they found that their data scientists and deep learning engineers are needing on a regular basis. The reason I wanted to talk about this today and draw it out as a learning resource is the fact that this is really how things are getting done. Most of the organizations I've been in have, have been container based over the last couple of years, last two or three years, and they're really focused on how do we want to do it and they develop a standard. And so if you're not doing that, if it's something that you need to ramp up on, maybe you're stronger on the data science side and containers are still feeling kind of newish to you. This is a good area to jump into. Read this article. There's similar ones out there from AWS and Microsoft and understand how you can accelerate what you're doing by using standardized containers that give you a full environment.

Well, in looking and thinking about learning resources, what stuck in my mind was a conversation we had with Joel Grus, which I think is the previous episode to this episode and how they'll be released. But he wrote this book, Data Science from Scratch, which we've referenced actually quite a few times on the show, I believe. So that's a great learning resource. But. But along with that, I was kind of looking at other things online that were from scratch. So kind of this idea of learning things by coding things from scratch. And I ran across this numpy ML repo on GitHub, which seems to have drawn quite a bit of attention, but I really hadn't seen since, you know, until I ran across it just in the last couple days. And there's a ton of stuff in there. So it's basically a bunch of things that are implemented from scratch with numpy. So if you're working in Python, there's this library called numpy and you can do like matrixy things and other sort of operations with it, mathematical operations, and there's just a ton of stuff implemented there. So hidden. Yeah, it's crazy.

I pulled it up. I'm just. Whoa. Okay.

Yeah, it kind of took. Took me off guard for sure. There's like, you know, LDA topic Modeling, hidden Markov models, neural networks of, of all types, pre processing methods, reinforcement learning. There's various utilities now I'm sure like some of this I surely there's multiple people contributing to this. It looks like there's six contributors, so I'm not sure I haven't gone through and looked at like the quality of all these things, but there's definitely a lot of, of attention that this has received. There's like 5,000 stars. So it seems like a reasonable place to go to just check out some things from scratch. So I'm, I'm definitely going to be checking this out and probably, you know, trying to learn from it and pull some of this into, into workshops and other things that I give.

Yeah, I am bookmarking it even as we speak right now.

Good stuff. I was commenced by Joel. The doing things from scratch is maybe not where you always want to do things, but it's a good exercise when trying to learn something. So I think this is a great resource. The other things I wanted to mention were a couple of new ish, so one newish, one new NLP related things. So as our listeners probably know by this point or I'm trying to learn more about NLP and work in that space a little bit, but there's a really great course that's free on advanced NLP with spacy from one of the developers there and it looks like a really, really great course and a really nice interface that's all integrated with nice notebooks and all of those things. Also Fast AI, which has just a ton of other great resources have introduced an NLP course. So a code code first introduction to NLP which covers a lot of different things, but including things like transfer learning, issues of bias, text generation, attention and the transformer. So all, all sorts of great stuff there. So check that out if you're, if you're interested in those topics as well.

Great finds, Daniel.

Yeah, it's, it's great that people are releasing such amazing quality material for, for free as well. And in a really, in really great interfaces, there's a lot you can do without having to pay for a college course or pay for expensive books and other things. There's a lot of good stuff out there.

Yeah, we had, and I wanted to call out that we had an episode, just a few episodes back where we talked about how to get into the field and we had a long list of educational resources, courses, books, videos at all, a lot of which were free. There were some really top tier things in there that cost nothing other than your time and attention and so we can add a link to that in the show notes as well. But yeah, great finds here. It's. There's never been a better time to jump into this field right now because you can, you know, even if you're tied down, you're working full time, you have a family anytime you can get on your, on your laptop to focus on these, you can, you know, log into a course that doesn't cost anything, keep working through it and stuff. So it's a good time to do it.

Well, I don't know a better way to end than with that pitch, so thank you for thanks so much for walking through some recent news with me, Chris. It was great to talk again, as always, and I wish you luck with your drones.

We're going to go have some fun flying them. I'll see you. I'll see you next week.

All right. Thank you for tuning into this episode of of Practical AI. If you enjoyed this show, do us a favor. Go on itunes, give us a rating. Go in your podcast app and favorite it if you are on Twitter or social network, share a link with a friend, whatever you got to do. Share the show with a friend if you enjoyed it. And bandwidth for Changelog is provided by Fastly. Learn more@fastly.com and we catch our errors before our users do here at changelog because of rollbar. Check them out at robot.com changelog and we're hosted on Linode Cloud servers and the linode.com changelog. Check them out. Support this show this episode is hosted by Daniel White Knack and Chris Benson. The music is by Break Master Cylinder and you can find more shows just like this@change law.com when you go there, pop in your email address. Get our weekly email keeping you up to date with the news and podcasts for developers in your inbox every single week. Thanks for tuning in. We'll see you next week.