2019's AI top 5

Bandwidth for Changelog is provided by Fastly. Learn more@fastly.com we move fast and fix things here at Changelog because of Rollbar. Check them out@rollbar.com and we're hosted on Linode Cloud Servers. Head to linode.com changelog.

This episode is brought to you by DigitalOcean. DigitalOcean provides worry free database hosting with their managed databases. If you need to get data in and out of Postgres, MySQL or Redis, call on the world class support teams at DigitalOcean and stop wasting time on setup, backup and maintenance. Get simple predictable pricing. Get detailed documentation. Get up and running in minutes so you can get on with your business. What are you waiting for? Head to.cochangelogged again. That's.cochangelogged.

Foreign.

Welcome to Practical AI, a weekly podcast about making artificial intelligence practical, productive and accessible to everyone. This is where conversations around AI, machine learning and data science happen. Join the community and slack with us around various topics of the show@changelog.com community. Follow us on Twitter. We're @practicalai FM. And now onto the show.

Welcome to another Fully Connected episode where Daniel and I keep you fully connected with everything that's happening in the AI community. We'll take some time to discuss the latest AI news and we'll dig into learning resources to help you level up on your machine learning game. My name is Chris Benson, I am Principal AI Strategist at Lockheed Martin and with me as always is Daniel Whitenack who is a Data scientist at SIL International. Hey, how's it going today Daniel?

It's going pretty good. It's 2020. Crazy man.

Happy New Year man. So we have, we have just put to bed our first full calendar year of podcast since we started mid 2018.

It's I think if I do the math right this is episode 71 unless we, unless we switch anything up. But yeah, so 70 plus it's pretty, pretty exciting. I don't know what we'll do when we hit 100 but we'll make sure something exciting for listeners when we hit 100 for sure.

And if any of the listeners out there have any suggestions for that, let us know. Join us in our Slack channel where we are on all the time every day talking to people or you can reach us on LinkedIn or Twitter. We are definitely out there having conversations with you guys.

Yeah, definitely. And pretty soon just as a final reminder for people I think might have mentioned this on other episodes, but both of us will be at the Project Voice Conference, which I think, as far as when this episode airs, will be the following week, so January 13th through the 16th. So if you're around at Project Voice, come find us. We'll be recording some episodes in the SIL International booth and as well, giving a keynote together, both Chris and I. So it'll be fun to be there and think a little bit about speech and voice and AI, what's going on in. In that world.

Absolutely. That'll be in Chattanooga, Tennessee, I think, on the. The week of the. Is it. I think It's Monday the 13th, if I'm. If I'm recalling correctly. So, you know, today we're going to do the same thing we did about this time last year as we got into 2019. We really wanted to kind of look back on a couple of notable points in the AI world in 2019, talk about kind of why, why we think they were notable, and also kind of assess the current state of AI where we are right now, and then look ahead to 2020. And of course, it would not be a start of your show if we didn't try to make a few predictions, each of us, on where things are going over the next year.

It is practical AI, so I feel like we have to make. Predicting has to be a part of it.

Of course. Absolutely.

And the predictions will be likely wrong. But. But maybe if after we do this for like 20 years, we can. We'll have a proper test set of predictions that we can really determine what our. What our accuracy was.

Oh, I'm not looking forward to that result. I'm not sure that's good. And I'm not sure if we should call them predictions or inferences, considering the field that we're in here.

Maybe that would be better.

One of the things that we were talking about before we started recording is just kind of. It's been an amazing ride so far, and that is entirely due to our listeners and our guests. And as you pointed out, you know, we just would not be. Be where we're at with the, with the. The show being as popular and so many people expressing how helpful it's been for them to get into this field and understand the detail.

Yeah. Thank you to our listeners and our guests. I mean, the, the guests for sure, of course, are. A lot of the great content comes directly from them. Chris and I are mostly, you know, I feel like a lot of times we're just kind of facilitators and there to listen to the great content that is there. So thank you to our guests. It has been great to get, you know, feedback on our Slack channel, talk to people on Twitter, talk to people at conferences who are aware of the podcast and are getting value out of it. And a lot of that is because we do get feedback. We hear it would be awesome if you did a show on this or I'd love to hear about this. And we definitely try to integrate those things in. So thank you to being part of the community. I hope you feel welcome and are excited for 2020 like we are.

Absolutely. I think a huge part of the show is the community aspect of it, even more so than the technical. It gives people an ability to connect. So thank you all for constantly talking to us over this past year and a half and making sure that we stay on track on how best to meet your needs. So I guess with that said, I know that on a couple of things.

Looking back, we definitely, yeah, top things of 2019.

Absolutely. We definitely are in agreement on quite a few of those. You want to kind of one of those big topics is Transformers and you want to kind of kind of jump into that and set that up.

So when we were thinking about, you know, the top AI milestones or notable things of 2019, we both kind of drew up our own list of things that we were interested in or thought were notable. There was a little bit of overlap, but the big piece that was overlap was Transformers. And we have an episode that talks about this in a lot more detail, specifically related to Bert. And we referenced GPT2 a couple times. But if you aren't aware of those episodes or haven't listened to them, 2019 has kind of been the year of the large language models, the year of the Transformers. So this kind of got kicked off with Bert and GPT2 and other models that were really large scale language models that in essence were able to learn a lot about language in general by being trained on many, many documents. So lots of data scraped from the web or other places and we're able to transfer to a lot of different NLP tasks. So whether that's machine translation or reading comprehension or named entity recogn, all sorts of things, text classification, these models have allowed us to sort of have a a zoo of really large pre trained models that know a lot about language and transfer those easily to these various tasks. And so we can kind of stand on the, on the shoulders of giants in a sense of OpenAI and Google and others who have trained these large models, have a lot of data and then allow us to kind of just level up our own nlp game by utilizing these, these pre trained models. And that's, that's been a huge boost to NLP this year in in particular.

I think the thing that really struck me about it is you and I actually come at this from different perspectives. You are a true NLP expert. Anyone who has listened to our episodes very much when we talk about this will know that it's what you do all the time. I observe it, but it's not my specialty. So I'm kind of coming from an outsider's perspective on that. And the thing that really struck me is these new large language models just impacted the entire world of deep learning and industry at large. Whether or not you were neck deep in it, the way you are, or whether you're really kind of watching this from outside and just being a user of these externally the way I am. And so it was like the big hits just kept on coming through 2019 as we did this, I know I was absolutely as was probably most people blown away when OpenAI did their first blog post on GPT2 early in the year. I think it was February, if I'm recalling correctly. And even as they, as they introduced it, they noted a couple of things, that it is, as we have specified, it's a transformer language model and it's used as a generative model of language where you can essentially give it a sentence to start with and it will generate a great deal of text, which in many cases is indistinguishable to the casual observer, you know, on whether or not it came from a computer or from a human. And it was pretty amazing when we saw that. And they did that initial release, which was a scaled down version, just to let the world try it. And they had to. They were kind of recognizing there could be security implications on that. They were slow to release and released in stages. But ultimately, if I'm recalling, the larger model they released later on in the year was trained on webtext, which contains over 8 million documents for a total of 40 gigabytes of text, which from, you know, if that was images, it wouldn't be so much, but for text, that's enormous. And they pulled that from URLs on the Internet in an unsupervised approach that was from Reddit submissions, in which case they had at least three upvotes. So they had a huge, huge corpus of text to pull from. And I just remember seeing those or those early examples of what was possible and thinking, okay, we're in a new place on the NLP front at this Point.

Yeah, definitely. I've talked to many colleagues who have expressed, specifically with that blog post that you mentioned, that prior to the blog post, if you were to ask them, hey, you know, what's the best that an AI model could do in generating text, regardless of architecture or kind of everything that's been done in the past, what's the best we could do? They would have guessed, you know, a much lower quality than what was published in that blog post.

Yeah.

And just kind of being blown away by that. And then of course, that fueled all sorts of things throughout the year. So I think these years, kind of 2018, 2019, have been referred to as NLP's ImageNet moment. So if you remember further back, when ImageNet came out, which is a. Was a challenge around object recognition and computer vision, there was a huge boost in computer vision and AI. And this I think, is kind of a parallel in what's gone on. And so there's just been an explosion in all sorts of things that build on this technology. So the technology itself, you know, these large language models, again, they're not. They're kind of building blocks in a way. We talked in the blog post about bert, about how these are structured often into sort of encoding layers and decoding layers, and how you can utilize BERT or these other models to create these sort of word embeddings or representations of text that can be used for a variety of tasks. And so that spurred not only innovation in sort of text generation, but innovation in all sorts of NLP tasks, like I mentioned in, in translation and all sorts of things. And I just saw one of the big indications of this, I think, is I saw that Google Search, which is arguably Google's kind of bread and butter, right? They just switched over to actually integrating bert, which is one of these transformer models, large language models, directly into Google Search in production, you know, live now. So I don't think Google would be taking that risk if they weren't convinced that this was a transformative technology. So that's pretty cool.

Is almost a meta issue around this. There was quite a bit of controversy in how GPT was released. And you know, we already talked about the staged release that they did in that original blog post. They note under release strategy, they say we're not releasing the data set, the training code, or the GP2 model weights. And they specify that we expect that safety and security concerns will reduce our traditional publishing in the future while increasing the importance of sharing safety policy and standards research. And that was really the first time that a major AI research organization had done. You know, everybody up until that moment was just publishing as fast as they could as new stuff came out. And that was the moment where they suddenly said, we have a greater concern. And there was quite a lot of debate about, you know, whether or not that was the right approach or not. I know we talked about it on the show a little bit, but it was just interesting to see how that policy debate shaked out over time.

Yeah, I would specifically, you know, like to note, and longtime listeners of the show will know that I. I like to mention this group quite a bit because I really think that they're a big part of what's happening. I specifically don't think that the momentum that's built up this year around Transformers would have been quite as much without Hugging Face's contribution.

Absolutely.

So we had Clem from Hugging Face on a while back. We'll reference that episode. That was actually before a lot of the stuff I'm about to talk about really built up. But after that episode, Hugging Face, they came out with a few things. One of those was this application called Write with Transformers, which is a really, you know, for non technical people. You can just go to this app and choose any of these language models you want and just try to generate some text with it. And it's kind of like a word document where you can kind of integrate these models. And I think that was just like a huge, big eye opener for people because not a bunch of non technical people could go in there and do this. It also forced Hugging Face to really deal with this. Well, how do we productionize these models? How do we integrate them practically? Which led them to release the Transformers library, which is one of the widest used NLP AI libraries. I think that's been mentioned a lot this year at top conferences, even all the research conferences, but industry conferences, even TensorFlow Dev Summit. Even though Hugging Faces traditionally work with Pytorch, I think. So this is really transformative. I even in my car often I listen to npr.

I do too.

I was listening to NPR and there was someone on there. I forget the exact topic. I don't remember the context, but they were talking about AI and they were like, well, I can use an AI model to generate some new NPR show titles for this show. And they used Right with Transformers, the app from Hugging Face to do that sort of on the show, which was pretty cool.

So, you know, I'm looking over what they do and you know, they've done such a good job of integrating their Transformers in with the Existing tooling As, you know, TensorFlow2, you know, was out this year, and Pytorch and those two still probably. I know there'll be some people disagreeing with me, but probably the dominant two frameworks and the tight integration, they've really made NLP not only powerful, but incredibly accessible to people. In your view. I know that you follow them very closely, you know, even beyond us having them on the episode. What do you think Hugging Face has done so well and so. Right. You know, they've kind of become, to some degree, it seems, kind of the darlings of the NLP world this year. That's at least how. That's my own feeling of it.

Yeah, for sure. I mean, I think that a couple of things that maybe can be extracted from that and we can learn for our own work is that they have focused on making things sort of giving people an immediate satisfaction with using these tools. So the right. With Transformers thing, you don't have to Even go to GitHub or download any models or anything. You just go and you try it out. Right. And so that, that I think is one thing is, is kind of, that we could extract from this is, you know, making AI consumable to all sorts of audiences is something that is incredibly valuable. But then also for developers, prior to the Transformers library, it was still rather difficult to integrate these large scale models into a normal workflow. And Transformers really gave a standardized API that people could use to pull in these models, utilize them for various tasks, or just utilize them for generating embeddings. And so I think that sort of standardization is something that we also saw with Spacy. So Spacy, which we had on the show recently. That's true. It has been and it still is extremely popular in the space, in the NLP space. And I think those are also characteristics that we've seen with Spacy, where they value good design, they value a good user experience, they have a nice way to standardize the sort of workflow around NLP to these sorts of pipelines. So I think those are really key ideas that we could take away. And just to give Hugging Face kind of a final congrats, they ended the year with an announcement of 15 million in funding to continue development of Transformers and what they're doing. So I think it's worth taking time to mention them and always happy to.

Absolutely. They've had such a profound impact on the industry this year. I've just been very, very impressed with them. It was a great episode. So if anyone out there hasn't listened to that episode. You definitely should.

If you like this show, I bet you'd enjoy listening to Brain Science. Join clinical psychologists Muriel Reese and Adam Stacowiak as they explore the inner workings of the human brain to understand behavior change, habit formation, mental health, and being human. Here's a quick taste of what you can expect. It's from episode four about coping skills and strategies. Take a listen.

I often use this acronym with people when they're trying to cope and it's halt H A L T Halt. Because if we are hungry, angry, lonely or tired, your coping will invariably look different. I don't care if you're 33373 right. If you are hungry or hangry, angry, lonely or tired, you just have less to be able to navigate it.

Brain Science is a great podcast. Check it out@changelog.com sl brain science or just search Brain Science in Apple Podcasts, Spotify or your favorite podcast directory. You'll find it. While you're at it, upgrade to our master feed@changelog.com master and let your podcast app download all the shows we produce. Then you can pick and choose the ones you're interested in the most and skip the rest. What have you got to lose? All right, back to the show.

So there were a couple of things this past year. I don't know that they were the most important things necessarily, but they were certainly events that really captured my imagination. And we did have actually episodes on both the things I'm about to mention. The first one people may recall a few months back, OpenAI did some work with robotic dexterity using a robotic hand where the hand was trying to solve a Rubik's Cube. And just to specify. And we had a whole episode talking about this. It wasn't the algorithm of the Rubik's Cube that the AI portions were solving because those are known solutions out there. So they just implemented one of those. But what they were doing was using reinforcement learning to really get the dexterity and sensitivity of the robotic hand to really a whole new level that had not been doing. And they shared some videos out there showing the robot manipulating the single robot hand manipulating the cube in all sorts of ways. And it really inspired me seeing that, to see the delicateness of it, the capability of being able to, to do very minute turns on the cube with, with digits and flip. It was interesting that it would make you hold your breath as you watch the video and at moments the Rubik's Cube would roll right up onto the fingertips of the robot and it would stop, balance there and Then spin. And it just made me realize that we were at the dawn of a new age for robotics in terms of what, what AI could do to really supercharge where robotics are right now, not only in more traditional movements and such, but also in these tiny little dexterity things with sensors that are able to capture delicate things. And after watching that video, you could easily think of robots as we've talked about medicine and things doing incredibly dynamic and precise forms of surgery on humans in that way that if you had all of the right sensors and everything, that you could take AI and robotics in medicine to a whole new level. And that really had a fairly profound. For just one story, it had a fairly profound impact on my perception of the state of the art. How about yourself?

Yeah, it was interesting. And this is a space, you're much more familiar with the space. But I think the things that stood out to me with that was their focus on making the models robust against perturbations and sort of new scenarios. So they develop these techniques around domain randomization and increasing the randomization during, during training, such that the, the hand was able to deal with all sorts of unexpected scenarios. And I don't know if it's accurate, maybe you can tell me, but it seems like maybe one of the things that's, that's held back AI and robotics a bit is, is this fact of generalizing to all sorts of different scenarios. Like if you're saying with medicine and surgery, you know, people are, come in all sorts of different shapes and sizes and ages, and so having, you know, a hand that would perform certain procedures would need to deal with all sorts of scenarios. And you can, you can't have all of those in your training set. So how do you make sure that your, your system is able to extend and generalize to different scenarios? I think that that focus in the project was, was really interesting to me. If that focus continues, maybe there's a way to, to kind of push the boundary there.

Yeah, it's really created a revolution in robotics in terms of, you know, we've had robotics for decades, you know, deployed in various industries, particularly industrial uses. And for a long time, everything about, you know, for instance, an assembly line had to be very precisely measured and there could not be variability, substantial variability in those workflows. And so we've really seen over the past two or three years at the moment, culminating in this robotic dexterity demonstration that we saw is the ability to accommodate some variability and have the ability to make change based on that variability dynamically in real time. And so when looking forward, and I know we're going to talk about kind of the world ahead, you know, time ahead later on in this episode, but it really starts creating new possibilities in terms of using these in scenarios that it just wasn't practical and realistic before. And so it was a neat demo. I don't think people should get too hung up on the Rubik's Cube aspect itself. I just think that was a tool to show what they were getting to. But it was, it was pretty cool. The other thing that had a very profound effect on me this last year, and we had an episode on it kind of mid year, I think it was in June, was on deepfakes, and more specifically, the very realistic types of deepfake videos where you're using a generative adversarial network again to generate those deepfake videos. And I think the thing that became obvious not just to us in this field, but to the public at large, you know, we had congressional hearings on it, was the fact that you were now entering a moment with this tool which could be used for both wonderful and nefarious purposes. It's not all bad, but you're really blurring the lines of what is real and what is not with this capability. And there can be fantastic, wonderful things. You go to an amusement park where they're able to implement deep fakes in rides, and it could be a lot of fun where it personalizes the ride ways and you could do some pretty cool stuff. Or obviously you could have things as bad as national security concerns about elections, the US elections in 2020, where we already have had the American FBI and the intelligence community warn us about that, it is highly likely that we'll have adversaries and strategic competitors trying to interfere in the elections. And then as I've heard some other people talk about, you know, what happens right now as we look at this technology and we have a little bit of time to assess it in some cases, what happens when we get to a situation where there is no time to figure out what is real and what is not. If, if you had a deep fake that showed the President of the United States describing that he had just launched nuclear weapons, you know, and you're, you're somebody out there who may be the target of that, you know, and that's not a real video, how do you assess in a responsible, appropriate, but expeditious manner to do that? So we're in a world that has changed in terms of our ability to know what's real and know it in essentially Real time. Any thoughts on that?

Yeah, I mean, I think you're definitely right. And along with that, we've seen an increase, I think, in research into detecting fakes, which is encouraging and I hope that continues. And then also, I know in a few of the episodes after we talked about those sorts of things, it's come up that there definitely are good uses of this technology. We've talked about, you know, generating medical imagery of tumors and that sort of thing, which is very expensive to annotate and generate manually. But, you know, we can kind of simulate that data and create simulated data using these methods that can improve, you know, tumor detection algorithms and that sort of thing. So with any technology, I think there's good and bad sides that you could draw from it. I think this one, the deep fake things and the videos that came out emphasize the bad ones first. And so it'll be interesting to as. As GANs are become more and more practical and integrated into different systems, what the positives are and how we deal with those other negatives.

Yeah, there was one website I came across and it used the deepfakes that were GAN powered to animate the Mona Lisa. So it took the, you know, what is the most famous painting in the world? And the Mona Lisa was busy gesturing and talking and stuff like that. So it was a cute thing. And I think we're going to see many good uses.

Yeah, some of the uses are just kind of interesting in that sense. There's nothing, I don't know who uses that sort of animated Mona Lisa for any sort of practical purpose, but it is still fun and it's pushing the boundaries.

One thing slight downer, not national security level downer. But I think I've heard, I've read some stuff that telecalls, you know, that you get from marketers and things like that, that that's supposed to be the next big wave is people scraping social media sites to get images of you and people, you know, and then trying to mimic voice or whatever on, on those. So as we go forward over the next year or two, that that kind of thing could be at a very personal level. It doesn't always have to be these giant end of the world scenarios. It can be something that is very immediate and known to you for sure. What about you? What are some of the things that you noticed in 2019 that were awesome?

I think one thing that we definitely have to note is TensorFlow 2.0. So I think the final official release of TensorFlow 2.2.0 was November 9th. If I if I search that right? I mean, I use Google Search. So if that's the wrong date, then I guess they can blame themselves.

I was going to say if anyone should know the date, it should be them.

But yeah, TensorFlow 2.0. So for those that aren't aware, with the release of TensorFlow 2.0, TensorFlow made quite a few significant changes, especially to the. Yeah, yeah. I mean, especially to the sort of default API to TensorFlow, which is now Keras, and also to the way in which computations happen instead of always generating this static graph that has to be executed later. And So I think TensorFlow 2.0 was just an amazing demonstration that the TensorFlow team. This is coming, I guess I should say this is coming from a Pytorch guy. I've used Pytorch way more than I've used TensorFlow and I really enjoy Pytorch and still really enjoy Pytorch and use it a lot. But for me, you know, I think it's a great demonstration that the TensorFlow team saw that, oh, we have this really powerful technology, but we've gathered feedback that we need to kind of shift some focus in some areas and make it more usable, make it more approachable and make it more practical. And so I think the sort of usability and practicality of TensorFlow 2.0 is just amazing and I think that they should be given congratulations for an amazing release and I can't wait to see more.

Yeah, as someone who used both the version one and version two, now I much prefer version two. And you can use it, you can use the Keras interface for the vast majority of use cases that most people are likely to see. Much more user friendly. And it was funny. This past year, a couple of conferences, I tend to keep my skills up. I'll go to TensorFlow classes and stuff. And I remember it was several months after the TensorFlow 2 beta had come out. We weren't a final to be full disclosure, so I can understand, but I remember going to a class, it was a TensorFlow class. The entire class on the beginning of that first day was immensely disappointed that we weren't using the TensorFlow 2 beta in the class instead of TensorFlow 1. I felt sorry for the instructor. I'm going to keep all the identities out of it, but it made that big of a difference in that community. And so Kudos to the TensorFlow team for listening to user feedback and turning out a great product that made great strides on the first one.

Yeah, I just tried this to See how easy it was to find. And I just searched for TensorFlow 2 CoLab Notebooks because that's probably where I would start if I was trying to find something. Or you could probably also search tensorflow2quick start, because the first two results are tensorflow2quick start for experts, tensorflow2quick start for beginners. And if you go in there, it walks you through the code itself. But also they have nice colab notebooks that you can open and try things out without even running anything locally. So it's super easy to get into and would recommend people to take a look. Probably the last thing that I want to mention, I mean, there was so much in 2019, I'm sure we're so sorry to all of you out there who are leaving, leaving out your favorite thing from 2019. But the other thing that I wanted to mention in 2019 is something that I detected throughout the year, and that was a sort of realization, which hadn't been there in 2018, that at least the way I felt it this year was that training AI models is super compute intensive. And this year I felt a little bit of pause from the community in saying, hey, how much energy are we, we expending to train these AI models and what can we do to make that more efficient and more responsible in terms of the environmental impact and all of that. So an article was released in 2019, I think, which caught a lot of people's attention, that, you know, training a single AI model, one of these larger language models, for example, can admit just training at once. A single model can emit as much carbon as five cars during their whole lifetime of use. And which is pretty staggering. And I personally felt like, you know, not everybody took this seriously this year in the community necessarily. And it's not like training large models has stopped. But I do think there is beginning to be a sense that we need to really pursue technologies that make AI more efficient and responsible in that sense.

Yeah, I remember us talking. I don't remember which episode it was, but I remember when it came out we talked a little bit about that. And I know that both of us are very environmentally focused in terms of being responsible. And so I was very happy to see people taking it seriously as well. And I heard a lot of conversations through the year about the topic. So I think it's a problem still to be solved. I think, you know, when you have very large scale model training, you have to do. There are currently not enough, we don't have enough solutions out there yet in terms of having the compute capability and yet still be able to, to be responsible because, you know, this technology is here to stay. We're going to be computing more and more. And so we need to be thinking about those responsible solutions just as we have at other aspects of AI that have come to pass that we'll be talking about in a few minutes.

Yeah, there's multiple facets to this. So I mean, there's the side of things which is of course making data centers more efficient and also running those off of sustainable energy sources. And I think that's been going on prior to this year and there's been a good amount of effort put into that. But also I think the pieces that I've seen developed this year, much more emphasis in sort of distilling and optimizing models to make them more efficient, make them run faster, which is partly driven by just practicality. Right. If you're using a model in production and it's smaller or you're wanting to port it to a mobile device or something like that, it needs to be smaller. So some of those things factor in as well. But also I've seen some efforts in sort of envisioning new more efficient architectures for, for modeling. So not always relying on, let's say the next larger transformer model. But are there other architectures, Maybe just regular RNNs that can do this task just as well as using or almost as well as using, you know, the full large size BERT model and are much smaller and can be trained in much less time? So I think, yeah, we, we need to approach this from various angles. But I think it's, it's something that people started hopefully taking seriously in 2019.

Yeah, I think we have the benefit of the fact that it doesn't require only a mindset in terms of responsibility toward the environment, but also just sheer performance. If you're able to find these other approaches that are allowing us to actually get there sooner, it's better for all concerned. So one of the things before we turn to what the future looks like is kind of let's take a moment and kind of assess where we are right now. We've just gotten to the end of 2019. We're at the beginning of 2020. Not only the beginning of a new year, but the beginning of a whole new decade. And so what are your thoughts toward where we are now as we hit this point any Daniel?

Yeah, sure. I think one super positive point of where I think we are and will continue to be in 2020 is really an amazing place in terms of the practical side of AI, which is what we're concerned about a lot here on the Practical AI podcast. And I say that because you have these things like we already talked about, like Transformers, but other libraries as well, and other toolkits or just code on GitHub, whatever it is. Infrastructure pieces, tooling. I feel like as compared to where we were at the end of 2018, there's just a lot more ways to be sort of robust and build system. AI systems with that have a lot of integrity in a much shorter period of time than we used to be. It kind of used to be very much the Wild west, and maybe we still are a little bit in the Wild west. But I think that a lot of the principles from software engineering have kind of come into the AI world and we're a lot more focused on versioning things, tracking things, monitoring things, whether that be with tools like tensorboard or other things, or it's like infrastructure pieces like Pachyderm and things like that. Kubeflow. We're just thinking a lot more about the AI systems that we're building rather than just AI models. And I think that's really encouraging and it helps people that are actually trying to build products and be practical and integrate AI. I think there's so much opportunity there and there's so many choices available in that regard.

Agreed. Just seeing you called out something a moment ago that really struck me and that is when we talked about this a year ago now, you know, going back to that episode, so much has changed. You know, we used to, when we first started this podcast, we were always searching around for good tutorials and examples, and sometimes we would struggle a little bit to find them in just that, that amount of time. And especially in the last year, there's so much available out there. The open source tools have really matured, great communities, the tutorials enabling people to do that. And we're finally seeing some of the surrounding infrastructure and tooling improving. I think there's still a struggle there as people really try to productize how they get models not only trained but deployed in the rest of their environment. But I think that's definitely something that's working hard. Now. Another thing that I've really noticed, I know at my job at Lockheed Martin, I'm very, very involved in our own AI ethics and responsibility initiatives. And so I spend a lot of time focusing on that. And over the past year we've seen pretty much all the major players out there, whether they be Google or Microsoft and many others, releasing ethical frameworks and their principles and such. And I think it's really gotten called out the difference between now and last year at this time where people were starting to talk about ethical AI. But the conversation has matured a great deal and the recognition that even with some of the limitations of where we are right now in terms of what deep learning can accomplish, that the dangers of abuse are very real. And we're seeing lots of the significant luminaries in our field kind of calling that out and expressing a need for standardization as we go forward on that. So that has been a fairly significant change in the last 12 months.

I think things like, for example, China's use of facial recognition, which we've talked about on the show before, and you know, Russia's use of behavioral modeling and that sort of thing to influence, for example, elections. Those have hit everybody. Right. And have been just kind of widespread or have been acknowledged in a sort of larger sense that AI isn't sort of something that is really cool and for sci fi, but there's like real uses of it that are going on, but not only real uses, but potentially really bad uses as well.

Yeah. And I know we've also talked about it in previous episodes, but as an example of something that it depends on where you are in the world and your values. But I know based on generalized Western values, China has their social credit system. And as we have been looking at that and talking about that for some time now, you know, they're using AI to not only surveil, but analyze and monitor their citizens and, and either reward or punish them accordingly. And so, you know, that's such a profound effect upon that particular country and the society that it's given us a lot to think about in terms of what do we want. If you live in a democracy where you have a say so in how things are implemented and you're one voice of many that can contribute, contribute to that voice. I certainly hope people are thinking about what is right for you and the community that you live in, no matter where you are and where does that make sense? And I think so that that has become. It's gone from being a fringe conversation to becoming a mainstream conversation in this past year. I'd say sure, sure.

So one thing that, that I'll bring up in terms of where we currently are in terms of the state of AI going into 2020 is I think that as we move forward, it's going to be more and more crucial that if we're really serious about using AI to tackle large scale problems like climate Change and the death of languages around the world. Access to good health care around the world. We're going to have to better involve researchers and developers from all over the world world. So we've had some really encouraging things this past year and, and things going into next year around that like you know, various workshops being held around the world in Southeast Asia and, and Africa. There have been conferences that have been placed in, in those areas. There's like the deep learning and DABA in, in, in Africa that's going on and offices of, of Google and, and others that are opening in those areas. But we're definitely not where we need to be. For example, you know, Neurips still this year there was a huge problem with researchers from around the world getting to Neurips and having their, their visas denied. You know, if you just look at publishing, we're still pretty driven by the US by, by Europe in certain areas. And so if there's been one thing that's been clear to me as I worked more with the NGO I'm a part of and also other NGOs is that if we really want to make an impact on these sorts of problems, we need to have representation from these local communities. You know, we can't just take for example if we want to extend translation like Google Translate to all sorts of languages. We can't not involve these communities because there's you know, we can't just publish research papers that say we're studying low resource languages and we just under sample English as our low resource language because that leaves out so much. It leaves out unique script problems and unique domain issues and cultural things. And so I think there's a lot of shifting that needs to happen in this area and I, and I certainly hope that that continues to happen as we move into 2020.

I think that's a really great point that you make there. Before we move on to predictions, the last thing I wanted to mention just about state of where we are right now is I also think there's a general consensus developing in the industry. We're seeing a lot of top luminaries. I know the VP of AI at Facebook recently said that we are very, very far from human intelligence. And that was in an article article I believe that Wired had and I think there was another article ironically that Wired had where there were some comments about the fact that with us hitting kind of some limitations on the types of problems that deep learning is likely to be able to solve and given the fact that it is a technique that is very narrow in terms of you get highly specialized results in a narrow scope. That one of the things that at NeurIPS that was talked about was the fact that we really need to get to biological roots of natural intelligence to understand what our next steps are going to be in the AI space. So what I think is that you may end up having people trying to reassess as they enter this new year about where they want to focus their research on and trying to do that. And I guess. Any thoughts on that before we move into predictions?

No, I think that's a great point and I've definitely seen, I think we'll put some links into the show, notes about various luminaries, statements on this sort of stuff. I've seen those as well. And I think that we can get into pattern that is, is kind of natural, but can be limiting in that like, for example, we, we're all about transformer models and we just do transformer models over and over and over and it breeds this sort of like NLP is transformers. But actually, you know, there's a lot of things that have happened historically in AI that we could pull from and there's new things that we could pull from maybe like you say, that are rooted in other sorts of, in other sorts of ideas related to biology or evolutionary algorithms or whatever it is. So I think we need to keep, keep our flexibility intact. I think maybe is a good way to put it.

I would agree, and I think the industry at large would agree with those sentiments based on the sentiment we saw at NeurIPS and that I think has been building over this past year. In general, let's look ahead to 2020.

All right. Inference time.

Inference time. Now figure out what we think might happen. I will start us off with a couple of them and then turn it over to you. I think as we talked about kind of AI ethics and responsibilities, I think we're now at an inflection point where we've had many organizations putting out their principles on what they think should be, but we don't have a very good way to execute on that. So not everybody is going to be an ethicist, especially in the, in the engineering field. And so I think that we're seeing a consensus that the next step now is to turn toward the creation of supporting tools or retrofitting existing tooling that enables nonethicists to appropriately implement the various aspects of ethical AI. Everything from eliminating bias from data sets to being able to think about where different types of AI should be applied to different types of solutioning. And so I think we're going to see, I'm predicting that we're going to see a surge over the next year and beyond in tooling to support ethical AI. Any, any comments on that?

I hope so.

You hope so?

Yeah, yeah, yeah, I'll be looking for it.

That sounds good. I think another thing that I think is that is happening already at starting to. I see a lot of conversation, I've been a part of a lot of conversations about this is the fact that we're getting to a point where instead of being a separate, you know, deep learning and neural network development being a separate little shiny object with dedicated people that only do the modeling, you get to the problem of how do you implement this in real life and you can build a great model but then people and organizations really struggle to get it deployed into production and getting a kind of a DevOps and feedback loop associated with what they're doing in those activities. And so I think you're going to see a lot of effort into moving neural network development into existing software development lifecycle and workflows that organizations already have in place and that they'll make adjustments to those workflows to accommodate these new technologies. And I think that's really important. Important for them to see a good return on investment for their efforts in this space.

Yeah, I'm thinking of. We've talked about that a little bit. Maybe Joel Grus episode on responsible AI development practices would come into play here. I'll link that in the show notes.

Sounds good. Another thing I'm seeing, and we actually already talked a little bit about it, I think TensorFlow2 is an example of this is we're seeing, I think we're going to continue to see simplification of neural network tooling and trying to make that learning curve more manageable. And I think you'll see different users and developers within this technology being able to buy into tool sets that are suitable for them and their own background. So I think that you'll see more tooling that is specific, that may cater to certain types of data scientists versus certain types of software developers. And you'll be able to kind to customize that tooling to match your level of knowledge, expertise and your background as well so that you can be productive quicker. And then I guess my final prediction is that I think that kind of given what we talked about, this acknowledgment at large that's developing within the deep learning field, that it's not well suited for certain problems, that it is taking a lot more data to learn than Maybe a human might use to learn something and is less flexible and stuff and overly focused on a particular solution and not able to move like by way of example from one game that you might have a deep learning algorithm that has learned be able to move to another game and be able to leverage what it learned from the first one. So we've seen many examples of that and I think as that message really permeates through the field that we will see people reassessing and maybe, and this is really the first time I have said this ever in our podcast, we will start looking at AI in the future as moving into a post deep learning world. I tend to, when I'm talking about the present, I tend to tell people I think of AI personally as equivalent to deep learning right now as we are at the beginning of 2020. But I think we also, we may get to the end of this year and that may not be a true statement anymore. And I may, I may have a different answer. So I think that is where we're going. What about you, Daniel? What are some of yours?

Well, I decided to go the safe route and I'm going to say that I'm going to. My, my prediction is that at least one of the following three things are going to be a huge, a huge player and a huge emphasis in 2020 and will really pick up Steam and maybe all of them, or maybe just two of them or one of them, I'm. I'm not sure. So I'm kind of covering my bases there that way, you know, my test scores are better when we look at things after time. But the three things that, that I was thinking of were first multi modal learning, then mobile AI or AI on mobile devices, and then federated learning. So multimodal learning is where for example, you, you make inferences off of multiple modalities of input data. Maybe you have an image and text that are input to a model and then you, you make some inference. This was I think already emphasized recently by our guests from Etsy in there sort of search technology where they have titles for their products and descriptions. But there's also more information in the uploaded pictures of the products. Right. And so you could take both of those input signals and do much more than you could with just the text or just the imagery alone. Right. And I think that this is going to be really revolutionary and pick up Steam and turn terms of a lot of things, whether it be chatbots or recommendation like in that, in that Etsy case or whatever it is. I think we're going to see a lot more of that. In fact, we saw that also with OpenAI's, a robot hand, Rubik's Cube thing where they were taking signals off of the hand itself, but also using the imagery from cameras and all of that.

I think you're right. I know that at my own company multimodal learning is a big deal. One kind of globally impacting use case that we're seeing it is in humanitarian assistance and disaster relief. And that is where you have, so you know, as you're trying to get data sets for a particular disaster scenario, maybe a wildfire, if you can get data from lots of different imagery, the various types of radio calls that are occurring and all that, that then then you can create a model that is much more robust and accurate and able to accommodate many more scenarios. So I, I, I totally think you're right on that. On multimodal learning. I think that is going to be huge going forward. That was a good call.

Good. Well, hopefully at least, at least that one comes true.

I have faith in you, man.

All right, cool. The other ones I think are really driven out of my sense that the privacy of course is all, it has been important, but is increasingly important and just the, you know, the scale of AI is extending to all parts of the globe. And so I think we're seeing a lot, I think we will see in 2020 a lot of deployments to mobile devices and a lot more tooling around that maybe along with deployment to things like browsers and that sort of thing where we're running, running models on user devices and fine tuning them on user devices. Along with that kind of goes federated learning, I think, which is the, the idea that we're not really centralizing data from all sorts of users and then running a centralized training and then porting the model back. But there is this sort of federated distributed training that's happening where a lot of the data from user devices doesn't have to leave user devices. And so there's advantages to that of course, because, because of privacy, but also data transfer and all of that. I've seen this talked about over the last years, but haven't really seen it, you know, really come, come about in a widespread way. And you know, possibly this is, this is the year.

I don't know, I think you're right, I think that's, I think that's a, you, you have stuck with safe, but they're very good bets. But I think their bet on all three of those actually. So yeah, I think you nailed it.

Well, I probably then have to learn a little bit of, I don't know, I need to learn a little bit of mobile development or something. Maybe we'll have an episode where we have some learning resources on that. But yeah, I've enjoyed this. Look back and look ahead, Chris. It'll be interesting to look back at this episode at the end of the day of 2020 and see, you know, see what, what, what came true and what didn't.

Yeah. So much has changed in the past year as we've, as we've called out and I suspect we'll have even more so this coming year. So it was a good conversation. Happy New Year again and looking forward to, to seeing you in Chattanooga at Project Voice and, and doing all sorts of cool stuff in the year ahead.

Awesome, Pat. Happy New Year.

All right, thank you for tuning into this episode of Practically I. If you enjoyed this show, do us a favor. Go on itunes, give us a rating, go in your podcast app and favorite it. If you are on Twitter or social network, share a link with a friend. Whatever you gotta do, share the show with a friend if you enjoyed it. And bandwidth for Changelog is provided by fastly. Learn more@fastly.com and we catch our errors before our users do here at Changelog because of rollbar. Check them out them out@robot.com changelog and we're hosted on Linode Cloud Servers. Head to linode.com changelog check them out. Support this show this episode is hosted by Daniel Whitenack and Chris Benson. The music is by Breakmaster Cylinder and you can find more shows just like this@changelow.com when you go there, pop in your email address. Get our weekly email keeping you up to date with the news and podcasts for developers in your inbox every single week. Thanks for tuning in. We'll see you next week.