Insights from the AI Index 2019 Annual Report

Bandwidth for Changelog is provided by Fastly. Learn more@fastly.com we move fast and fix things here at Changelog because of Rollbar. Check them out@robar.com and we're hosted on Linode Cloud Servers. Head to linode.com changelog this episode is brought to you by DigitalOcean. Digital Ocean's developer Cloud makes it simple to launch in the cloud and scale up as you grow. They have an intuitive control panel, predictable pricing, team accounts, worldwide availability with a 99.9 uptime SLA and 247365 world class support to back that up. Digital Ocean makes it easy to deploy, scale, store, secure and monitor your cloud environments. Head to do co Changelog to get started with a $100 credit again.co changelog. Welcome to Practical AI, a weekly podcast about making artificial intelligence pract productive and accessible to everyone. This is where conversations around AI, machine learning and data science happen. Join the community and slack with us around various topics of the show@changelog.com community follow us on Twitter. We're @practical aifm. And now onto the show.

Welcome to another fully connected episode of the Practical AI Podcast where Daniel and I keep you fully connected with everything that's happening in the AI community. We'll take some time to discuss the latest AI news and we'll dig into learning resources to help you level up on your machine learning game. My name is Chris Benson. I'm Principal AI Strategist at Lockheed Martin and with me, as always, is Daniel Whitenack, a data scientist with SIL International. How's it going today, Daniel?

It's going great. It was, you know, a cold and rainy slash snowy weekend. But on the good side, as some of our listeners will know, we had a season in our household of flu and sickness and that's kind of ending so very, very happy about that. You know, no coronavirus yet.

Oh yeah, I was about to make a joke. Welcome back to the land of the living. But you know. Yeah, with that in the news right now as we're recording this, that's been a big thing the last few days. Yeah.

And I know you pointed a link to me earlier which was pretty interesting for our listeners. You want to tell people about that?

I was just scanning across news articles and it was actually on Wired and it noted a couple of days ago it says the title is an AI epidemiologist sent the first warnings of the Wuhan virus. And so which is the virus that started in China and is spreading to some parts around the globe as a type of coronavirus. And I think the, the short part of the article is that there's a company called Blue Dot who has algorithms that take a lot of data sources, you know, from health and airline ticketing data and such to predict the spread of disease. And in this case they really kind of got there first actually on December 31st, before new, they actually sent out their first note that this outbreak was expected. And it was another, really another week before here in the U.S. the CDC, which is the U.S. centers for Disease Control and Prevention, got the word out. And so it was a case of, I don't know the detail of their algorithm, but they refer to it as an AI driven algorithm and they got those first reports out. So we like to talk about AI for good. That certainly seems like a good thing to get early warning of a major outbreak like this.

Yeah, definitely. It's, it's a super interesting thing and in some ways it seems very much science fiction to me. Yeah, like they're kind of, you know, like movie like over detecting all of these signals around the world and correlating them to say there's going to be this pandemic or whatever. Their article says that they scour foreign language news reports, animal and plant disease networks. I'm not quite sure what that network is, official proclamations and other things like that. So it's definitely pretty interesting. And they're really doing this. It's pretty cool. If anyone knows anyone at Bluedot out there, we'd love to have them on the podcast to discuss that. Maybe we can make that happen sometime soon.

That sounds fantastic. Yep. Timely news about AI having an effect on the world and possibly able to save lives, especially as they gain notoriety and others start really watching them. It might make a big difference. So yeah, I mean I think we're really over the next few years the way AI is really revolutionizing medicine at large. You know, this is one of, of many of those cases.

Yep. So speaking of how AI is revolutionizing things, you also found an interesting thing that we're going to talk about today on our fully connected episode. The goal of these episodes, again just being to keep in the news the AI news and keep ourselves updated, but also our listeners and kind of dive into the topics that people are currently looking at. And Stanford's Human Centered AI Institute in partnership with a variety of others, came out with an AI index report, 2019, which kind of provided a lot of. I mean we did a sort of top five things of 2019, I think, but that wasn't really based on rigorous research and data collecting and that sort of thing. This is more of actually, you know, who published articles, what's going on in the AI world as we move into 2020. And there were some interesting things in there I think that would be great for us to dive into. So that's going to do today.

Yeah, it's quite a lengthy document. I haven't counted, but I think, you.

Know, somewhere, what, 150ish, 291 pages.

I was far short. Okay. Yeah, it was quite long. I didn't look at the page line, but they had. Yeah, they broke things out into a number of sections and it was a really interesting kind of overall view of kind of the world of artificial intelligence in general and gave a lot of statistics and facts. The breakdowns were pretty interesting, actually.

Yeah. And they have, I'm just looking at the steering committee here From Stanford also McKinsey Global Institute, Partnership on AI, Harvard, OpenAI, MIT, SRI International. So it wasn't just Stanford that put this together. They also mentioned partners of Google, PwC and others. So yeah, it'll be interesting to see what they think were the, the noteworthy things that are happening now in, in AI, some of which are maybe different than the things that we talked about in that, you know, first episode of the new year. So, yeah, let's go ahead and dive in. It looks like the first section of what they talk about is research and development, which I know both of us aren't professors or anything like that, or involved in academic research in that context, but we both interact with research people. What, if anything, kind of surprised you by what, what they talked about, Chris?

Well, you know, they started off kind of really focusing on, on how much growth there had been in the space. And I don't think that was. Was surprising in itself, but we've really seen the rise of China in terms of just raw numbers of publications coming out. And as part of that, they finally passed Europe. They had previously passed the US already. And so, you know, we're still seeing that, I guess despite the fact that you're seeing those raw numbers, the fully weighted citation impact though, of US publications is still about 50% higher. China's. And that really rang true. I was actually in an internal artificial intelligence meeting last week with my employer and we actually were discussing that specifically. Just the fact that you're seeing more citations. The discussion was in terms of the quality and obviously we were speculating on why that is and when that might change going forward.

Yeah. And I think actually on our next Episode we're going to be talking to someone from the Semantic Scholar team at Allen AI and of course that project is, is really concerned with discoverability of, of research and kind of weeding through the noise and all of that sort of thing. So that'll be interesting to talk to them about how this, you know, this giant surge in research and archive papers and all of those things has increased and in some ways made it hard to find, you know, the real, you know, notable things that are happening in some cases. I thought, you know, another thing that was interesting was that they noted several small countries that were having a relatively high increase in deep learning papers in terms of. Of per capita. So these are countries like Singapore, Switzerland, Australia, Israel, Netherlands, Luxembourg. They're talking about these countries who, despite their small size, are really kind of, I guess, making AI research or encouraging AI research and believing in that as a future driver of economic prosperity and innovation and all of those things.

Yeah, it's interesting and we've talked about this before, you know, I know we've talked specifically about Singapore. It was very obvious to me about a year ago when I was in Switzerland for a conference. These countries have really kind of committed to some degree maybe their national identity a little bit, to saying, hey, this is something that we're going to do. You know, technology in general and specifically AI in a lot of these cases.

Do you think, I mean, obviously we're both pro AI, I think, I wonder just kind of off the top of my head if that's kind of making a dent in other research areas in terms of taking away funding or putting the focus less on other still very important areas. But now that everybody's kind of all in for AI, what effect that's having on sort of more traditional biology and medical research and those sorts of things.

That's an interesting idea. I'm not sure I would say that obviously there is a finite, finite amount of, you know, research and development dollars.

And time or whatever your currency is.

Yeah. Available and stuff. And I suspect as we look at different strategies over time, you know, AI is one of the great enablers of our time. And so I think they may be, you know, selecting certain specialties and then kind of going after AI as an enabler in this.

I mean, I know that that's why, at least partly why I'm in industry versus in academia, because I originally wanted to go this sort of professor route and that sort of thing, but I was in physics in grad school and I mean, physics as a discipline. I think it's exciting There's a lot of people that think it's exciting, but in terms of like a paradigm shifting things that have happened in recent years, there haven't been a lot in terms of the type of paradigm shifting things in physics of let's say the 20s and the 30s and that sort of time period. And so physics has become a sort of. I think it's plateaued in terms of its excitement to some degree. And that's made a lot of the jobs in physics research very competitive. Right. Because there's maybe not as many universities that are really filling their bench with physics people. Maybe they're now filling their bench with computer science, AI type of people. I think it's definitely will be interesting to see how that that plays out on those fronts.

Interesting. You know, another interesting note there was the fact that we're seeing, especially in Western European countries, but, but not limited to that, in countries such as Netherlands, Denmark, Argentina, Canada and even Iran, a relatively high presence of women that are involved in AI research.

Yeah. Where's the US on that list?

Yeah, I would ask the same question.

What's up with that?

Yeah, but it's nice to see this field, I know that that is since the very beginning of this podcast, that a big goal of ours is to see this field be a truly equal field in all respects. So that was a stat that caught my eye that I was really thrilled to see.

Yeah, I think we need to take note of some of these countries and see what they're doing to promote that and try to increase that more. I know there's been another thing in the next section of the index report is about conferences. And one of the things maybe that has driven that change is they talk about the Women in Machine Learning workshop. That happens throughout, I'm not sure how long it's happened. They talk about 2014, but they say that has 20 times more alumni than it had in 2015. And so I think that this is one of the contributing factors. I think it's a pretty big contributing factor that there's been intentional effort to have these sorts of workshops and kind of local chapters and all of these things focused on women in machine learning and AI.

Yep, I agree. It's a good sign. It's a good sign that we're moving in the right direction. And, and you know, just in general we're seeing conferences, their popularity and the number of people attending in AI related conferences just exploding.

It's insane.

Yeah, I mean, Neurips was, I think it upwards of well over 10,000 attendees, I believe at this point, yeah, I.

Think it was like almost 14,000, 13,000 something, which is just insane to me.

It is, Yeah.

I mean, 13,000 people, like, I don't know what we're gonna have to start doing, like rent out like football stadiums or something for AI conferences. Like what is the future? I don't know. I would say, I mean one personal note on that is I would love to see, and there was, I think some of this last year where there's a lot more effort to live stream things. And so I watched several conferences live streamed at least some of the content. I really appreciated that. And also from an environmental standpoint, if we have 13,000 people taking plane rides to go to a conference, I would love to see that many people involved. But you know, having live streaming resources and better live streaming and remote conference events would be something I would love to see.

I would too. I mean we're seeing this explosion and being able to participate as we see so many people wanting to get to conferences and you know, getting visas rejected, getting visas rejected in a lot of cases, just not able to get in. I mean Neurips is famous for its lottery and you know, so many people that would like to go, cannot go. And that's despite the massive number of indentities it already has. So yeah, live streaming would be, would be a fantastic way of being a little bit more inclusive for those who either can't travel or are wanting to be responsible and avoid the environmental impact by getting on a plane. Yep.

The next thing, which is kind of obvious that they would go into this. But the technical performance of AI models has shifted in several ways. Generally they talk about image classification as an example task and they talk about how it's, you know, the time required to train these sorts of models has drastically decreased and the cost to train them has drastically decreased. So there seems to be this sort of more general availability of architecture or of compute resources in the cloud that allow you to train these systems. So the availability of that, but also efforts to speed this up, maybe architecture or framework or language wise as well.

Yeah, you know, we're seeing, you know, I know that one of the topics you and I like to talk about a lot are transformers, for instance. And I've noticed just a few weeks ago I was talking to some folks and you'll see these large transformer models come out and then these follow ups that are huge performance enhancements and they may, you know, they may reduce the size of the model, but you end up getting a dramatically faster training, you know, based on these optimizations. And I think when you counter that with the fact that we're seeing gpu, TPU and other hardware architectures really, really accelerating, you got cloud options. You have options for being able to have one, you know, maybe have a GPU right on your desktop that you're working, whatever. And I think, you know, the combination of that has made a huge difference in accessibility for people to be able to actually. Do.

You like this show? So I bet you'd love listening to gotime. Not working with Go. Don't fast forward quite yet. Gotime covers a wide range of topics including cloud infrastructure, distributed systems, microservices, kubernetes and Docker. Here's a ridiculous clip from a recent episode about the defer keyword.

I think, I really think that Matt misses his calling as a stand up comedian. Totally. Yeah, funny. I mean he can still be one.

He just has to choose his audience very wisely. It's got to be a tech audience.

Well, he, he has got time fm I think the funniest low key podcast.

Out there thing is no one in tech, no one likes a stand up comedian. You just want them to get on, tell us what you did yesterday, tell us what you're doing today and if you've got any blockers and get off. You know what I mean? No one wants the. Yeah, there you go. See, that's why I didn't. I'll stick to programming.

I mean there's only three people here. You might have a whole audience that's.

Live listeners that's laughing right now.

Oh yeah, let's assume that. I'm pretty sure this could be edited to make me not sound like an idiot.

You heard Carmen, Go time is low key. Hilarious. Check it out@changelog.com GoTime or just search for GoTime in Apple Podcasts, Spotify or your favorite podcast directory. You'll find it. Once again, that's changelog.com gotime.

So Chris, I found one really interesting thing in the index report that is related to technical performance and if people are following along while they're listening, this is on page 68 of the report. So a good ways in there but they go through and talk about the milestones that have been achieved in terms of human level performance and AI reaching or beating human level performance in certain tasks. And I hadn't seen something like this. Probably are things like this compiled in other places. But I thought the compilation and timeline that they compiled here was really interesting. So this starts with Othello back in 1980 and it goes all the way to 2019. Detecting diabetic retinopathy with specialist level accuracy. This is really cool. I don't know how many of these things were familiar to you, Chris.

A few of them are that, you know, they talk about AlphaGo. There's a number of them, especially given back when I was doing gaming when I was younger. A lot of the older ones, ironically, are more familiar with me than some of the newer ones. The ones that have been in the news a lot in terms of AI in the last few years, I'm familiar with. But they had some that I had not noticed before. The prostate cancer grading, I had not seen.

Yeah. So pre2011, there's three milestones that they list, which are Othello, checkers and chess. And then post, you know, 1997, then we skip all the way to 2011 and you can kind of see this, you know, rapid advance. So there's like between 2011 and 2019, I'm not going to be able to count these, but there's probably at least like 15 or something or. Yeah, something like that to 20 after that, including things like, you know, Dota and video games, machine translation, all of those things.

And you know, something I noticed just as a side comment, since it leaps from 97 to 2011, those are the years, the new. Of the most recent AI winter. I almost said nuclear winter. It's not, not a nuclear winter. But yeah, the most recent AI winter was right there. And you saw zero progress as everyone turned away from neural networks for those years.

Yeah. Also kind of going back to the conference thing, they have this graph in the, in the report where they track AI conference attendance and you can see. So they track back to 1985 and I see. So this is one of those aside for a minute, this is one of those plots that, being slightly colorblind, I have no chance of reading because all of the lines are like colors that blend together for me. So just FYI, there's color palettes out there to help with that.

Okay.

But maybe you can help me know which one goes back to 1985. Uh, oh, there's a couple conferences back in 1985 that have at or over 5,000 attendees. And then you can kind of see as, you know, 1990-1995-2000-2005, it actually decreases all the way to about 2010. Some of them start to increase again and then, you know, 2015 and on, it's just like skyrocketing attendance in these conferences. So it is just historically interesting to see, you know, Back then there were these very high profile, you know, 5,000 attendees at a conference is no joke.

Yeah, that's quite a conference. And for those of you who haven't been to one that size, I mean, it's just you get lost in them in terms of trying to find your content and everything. So I actually prefer smaller conferences. They're a lot more intimate, a lot more fun from my standpoint.

Yep. Before we leave the technical performance side of things, one of the trends that I saw them point out, which was interesting to me, and they drew it out in terms of nlp, but I think it's true of computer vision in some ways as well is that in terms of benchmarks, like they had benchmarks for certain tasks in NLP or computer vision, like object recognition or like machine translation or entity recognition, reading comprehension, coreference, all of these different benchmarks. And as they've reached, you know, human level performance on these, they actually had to go like they, I'm meaning the research community or those in the research community. Many have had to go back and say, how can we make this more challenging? Because we're reaching, you know, we're reaching human level performance in so many of these tasks. So for example, in the AI world, there's this benchmark called glue, which I'm going to mess this up off the top of my head. I think it's general language understanding here.

Hold on. Come up with that.

Something like that. Let's see, there's glue and then there's super glue. So there's general language understanding evaluation. There we go. And then they had that, and then that wasn't enough, or that wasn't challenging enough. So they reached human level. And this graph is in the index report as well. So you can see kind of models surpassing human level performance in glue, this sort of task that combines a bunch of NLP tasks to make it harder itself. And so human level performance was beat there. And so then they developed this other one, which is super glue, which kind of ups it from there. And others like Allen NLP or Allen AI Institute and, and others are producing other benchmarks to kind of further challenge things. So as we've reached a lot of those milestones now, we're kind of in this season of, you know, what's next? How do we make this harder for computers? And a lot of those things are like common sense understanding and reasoning that are really hard for computers to do. We still have a lot of room for growth there. And of course, in languages other than English and multimodal settings, where we're combining video and imagery and text and all of those things. It'll be interesting to see what benchmarks come about there.

I agree. Just as a side thought there, we're having a couple of interesting conversations, meaning the community at large. Right now. We're seeing these things that we're calling out here in terms of how far, far we're coming and having to adjust for benchmarks. And then you see people in the kind of artificial generalized intelligence community saying, oh, we, we need to have completely new models and stuff. And I think people tend to get caught up in one or the other. It's interesting that in terms of kind of the deep learning basis where we're at right now, we're really still making pretty immense progress in terms of applicability and performance improvements. And while that may not be AGI, I think some people tend to get caught up in one or the other conversations. I think it's pretty remarkable that we're in an industry where you can have both of those conversations with the specifics of whether we are going fast or whether we are not making much progress at all in the larger scheme of things. But I really think it shows how vastly artificial intelligence has moved into culture and society and industry at large. That astounds me repeatedly. And I think this report on the applicability of these technologies is really amazing. Just as you just called out the fact that NLP is making such fast progress.

Yeah, sometimes I think we over complicate the sort of how much progress are we making? I mean, one indication of that is economic investment in AI and application within industry. That's another point that they call out in the report and they throw out, you know, really huge numbers like oh, AI investment was over 70 billion, with AI related startup investments over 37 billion. Honestly, I mean, I don't know about you, Chris, those numbers don't. It's hard for me to grasp those numbers because I've never seen a billion dollars.

Yeah.

But it's a lot of money. Right? And there's definitely. They also cite different percentage increases in jobs and AI investment. And I'm guessing, you know, some of that may be hype.

Right.

But there is actual proof that AI applications within industry are driving a lot of change and people are responding with investment.

There was a particular stat, it's a US centric stat that I noticed and that is the total number of jobs in the economy relative. And taking that total number and looking at the share of those that are kind of AI related jobs or at least, at least in terms of, you know, maybe titles or tangentially, it's approaching nearly 1% in terms of AI jobs to total jobs.

And that's AI jobs meaning like humans doing AI, correct, not AI doing jobs.

I'm glad you called that out just for, for clarity on that. But yeah, the, the number of jobs that we as humans are engaged in that are AI related compared to the total economy, it's approaching 1%. And that is remarkable because we're still at such an early stage in this industry. And so, so you can see, I believe back around 2015 it was just a fraction of 1%. I think it might have been 0.3%. So we are growing so fast in terms of how AI is impacting the economy, represented by the number of jobs being created to do just that.

And job wise, they did a bunch of analysis of LinkedIn which I found this interesting because they have a lot of, so they have asterisks next to like India, for example, because I think they said like 40% of workers in India are on, on LinkedIn. And so that numbers are likely not accurate in that sense. If, if anything, they're kind of diminished, I would say. And yet India was kind of at the top of a lot of the job statistics in terms of, of how many people are involved in AI and also the fastest growth in AI hiring. Again, these other countries that have invested heavily as part of their national strategy in AI, like Singapore, Singapore, Brazil, Australia, Canada, we're right at the top of those AI hiring stats. And it's interesting to, I mean it, it makes sense with what I've heard. So I've, I've got a few people that I work with in Singapore and from what I understand, you know, AI people in Singapore are pretty much snatched up instantly. So if you're trying to hire someone in AI in Singapore, there's just so much hiring going on and there's not enough AI people to go around, which is one of the reasons why they establish some of these things like AI Singapore, which is trying to feed AI expertise into industry. But there's just so much hiring going on, the demand is, is so high that, that AI people, you know, could, could get hired pretty much right, right away. So if you're interested in getting an AI job, you know, consider Singapore. It's a beautiful place.

Although I don't think it's, they may be the most extreme case, but I don't think it's a problem just for Singapore. We're seeing that really everywh know, all, all employers that are invested in AI, which is obviously an increasing number, steadily are contending with that same issue in terms of finding qualified people who can be productive quickly. And you know, so the university system, you know, they're snatching them straight out of universities that are oriented on AI. You know, it is just an explosive growth area. You mentioned all the billions of dollars a few minutes ago in terms of kind of global private AI investment. And, and along with that I was really astounded to see that year over year the annual growth rate being around 50% in terms of investment in startups is continuing to go. And that's despite various economic worries and things that people are concerned about life in general. So it's still quite staggering the explosiveness of the field in general.

This episode is brought to you by Brave. We deserve a better Internet. That's why the team behind Brave reimagined what a browser could be. Brave is like Chrome, the good parts. Even your extensions will just work. It has built in ad and tracker blocking, easy anonymization with the Tor network, earn tokens while you browse and use them to tip your favorite creators. And did I mention is lightning fast. Turns out the web is super fast when you remove all the cruft. Download Brave today using the link in the show notes and give tipping a try on changelog.com.

All right, so you kind of started into the conversation about education and me being in a university town. I also am kind of monitoring this and thinking about it, but more people are going to school to learn specifically AI related things than ever before. And you know, one of the things they talk about is international PhD students pursuing AI specialization in computer science. So that's up. But also there's this sort of interesting trend of PhDs that are graduating in AI, are not going the sort of academic route in general in terms of getting professorships and that sort of thing. But there's kind of this drain of AI talent where a lot of these people are going into industry to work at awesome cool places, whether that be like Google Brain or OpenAI or whatever it is so much so that, you know, AI faculty are leaving academia for industry and that's continuing to accelerate. And so that's, you know, exciting in some ways that there's kind of this infusion of expertise into industry. But it's also concerning in some ways because at least from my perspective, it seems like, you know, when I'm thinking about academia, this gap between how in touch academia is with industry is kind of separating more and more in certain ways and so, yeah, I would love to see industry and academia be closer and people coming out of university programs really ready practically to do jobs. But I'm not sure that that's really happening because the people, the AI professors in academia that are interested in industry stuff are just leaving to go to industry. And then what's left is kind of just pure academic research, which is interesting, but maybe in some ways less connected to industry problems.

Yeah, it's really changed both academia and industry, this trend, and it's changed the relationships we have. And one of the things that I'm involved with at my own employer at Lockheed Martin is I engage a lot of universities in terms of artificial intelligence with that. There's a number of us that do this and the nature of those collaborations are changing. Whereas once upon a time, you know, the, you, you might think of kind of the, the brain trust at these universities and that industry would access them to help them. But we're seeing so many people that, as you pointed out, might otherwise have been on an academic career path are going to industry because of the opportunities and because of the compensation that that has changed. And so it's interesting to see the partnerships that we're having between industry and academia where both sides are doing cutting edge research in different topics and collaborating across. And you're seeing, you're seeing, you know, a substantial amount of that in industry, whereas it used to be mostly in academic. And then industry would kind of take that and apply it to what they're doing. But you're, this field is moving so fast and the brain drain is happening from academia into industry for those reasons. Reasons that it's kind of rebalanced that. And I think both sides are trying to figure their way through that at this point. It's also driven compensation rates through the roof for AI specialties, obviously. And so that's. And that becomes another thing where different companies are competing for the talent.

Yep, it will be interesting. I know one of the other things we've talked about before on the podcast is sort of trend to more formalization around data science and AI programs within universities. So to where before a lot of universities took the strategy of, you know, AI as a graduate discipline within computer science, which it definitely is and should continue to be. There's efforts to kind of embed data science and AI across all organizations, with some universities even taking steps to establish, you know, the center for Data Science or Center for AI or whatever. And there's kind of cross discipline collaborations that happen within those centers. So that's interesting. I Think there's success stories within that and there's not so great success stories with trying to apply that. So yeah, I don't know what all of the solution is to this sort of balance between industry and academia and maybe it kind of just flows back and forth. But yeah, it'll be interesting to, to follow for sure.

Absolutely.

So other things that was emphasized in the economic point with the index, but also actually called out as a separate whole chapter in the index report were autonomous systems, specifically autonomous vehicles. So autonomous vehicles receive the largest global share of or the largest share of global investment over the last year, followed by things like drug cancer therapy, facial recognition, video content, fraud detection, other things. But, but autonomous vehicles were at the top. And I know this is something that you've been involved with personally and of course Lockheed Martin is interested in, but it kind of took me off guard a little bit because you hear a lot about self driving cars and that sort of thing, but it doesn't seem to me to have penetrated markets as much as something, let's say like facial recognition or computer vision. And yet it's at the top of investment.

I think you will though. You know, one of the things that was notable is that the state of California, you know, licensed testing for over 50 companies with an enormous number of autonomous vehicles. And they noted that they had already driven over 2 million miles. And when people hear autonomous vehicles, they're often thinking about cars on the road. But what we're really seeing here is a, is a transformation and they kind of call it out. In this report that you're seeing, I think we're right on the cusp. You know, now that California has done that, you're going to see other states and other countries as well engaging in the same thing as people recognize that these vehicles can be safely integrated into society, whether on the road or in the air or on the water, wherever we happen to be. And we're going to see that more and more. And I think we're kind of right. I think the stuff, this action by California demonstrates that we're kind of right at that tipping point as we're recording this, where we're going to see it everywhere. I know in the company that I work at, Autonomy is a big part of it, as it is in many different industries. And so I think you're going to see autonomy becoming fairly common over the next few years, whether it be on our nation's roads or those of other nations, or from the bottom of the ocean to the surface, all the way to outer space. I think that it's going to become.

In the Space Force.

Yeah, that's exactly right. That's another thing worth calling out is in the last few months, the US Space Force has been created out of what had been the Space Command in the United States Air Force. So we're now at a point where it made sense to separate those out as their own concerns. And so I think you're going to see autonomy in every facet of transportation sooner than most people might expect.

So I'm full of asides today. But just as an aside, did you see how close the kind of the Space Force logo and the Star Trek thing? So I saw people have them like side by side on Twitter and you know, it's kind of. Yeah, I don't know what to think about that, but it's kind of interesting.

We'll leave it to people to call out on Twitter and slack to us. We welcome your comments. I've seen some pretty funny ones so far.

What's your opinion about Star Trek and the Space Force anyway? I guess that really gets so see, I made a really good transition with that aside to talk about public perception and societal considerations, which were some of the last things that were talked about in the index along with national strategies. So they talked about public perception of AI, societal considerations around things like fairness and interpretability. One of the things that I thought was good in the report is they did specifically call out that there's these 17 United Nations Sustainable Development Goals, which cover a lot of things around education and climate and other things. Oh yeah, there's. So there's these 17 and then there's 169 targets. And they talk about how AI can contribute to each of these. And if you remember, we had another guest on the show with the AI for Good foundation who is directly working with the United nations to apply AI to the Sustainable Development Goals in really interesting and amazing ways. And so I definitely, if you're interested in that side of things, take a listen to that other episode. But I think that's really worth calling out is now more than ever because we have reached so many milestones in terms of AI. We're at a point where we can really apply AI to all of these different problems that really matter and make a difference for the quality of life for people, to give them a better life. So if you're interested, you know, that's really a great effort to be a part of. And in terms of if you're looking for side projects or just to learn about AI, why not, you know, take on, you know, some side, projects related to the Sustainable Development Goals, related to AI for good. Yeah, I think it's a really great time to be part of that sort of thing.

I agree completely. And as we talk about AI for good and societal impact, I think maybe finishing up with one last point that they note in this document is they really point out kind of the rise within the context of AI fairness, interpretability, explainability, what we tend to call ethics. And they identified that really those topics in terms of references to AI ethical principles have become an enormous, enormous conversation that we're having globally at this point. And we're recognizing that we have these powerful tools and before unintended consequences could arise, that we need to be thoughtful. I love the fact that people are engaging on this and trying to say how can we think about fairness before we have problem? We've had some bumps in the road over the last few years, obviously, but I'm very optimistic as we kind of go into the 2020s here, about people at least engaging on these topics, on these kind of ethical AI principles on the front end of the decade as we, as we surge forward. So I just wanted to end on that note of optimism and ask people to continue to do that. Don't just do the engineering side and the data science side of AI, but think about the world that you want. And AI for Good, as you mentioned, is a great place to be thinking whether it's in your primary job or whether it's what you're doing for a side project when you go home at night.

Yep, definitely. And we always like to share learning resources as part of these fully connected episodes. And one one related to what Chris was just talking about, which we could share, which I poked around a bit with, is the AI Fairness360 toolkit from IBM. I think we've mentioned it maybe once on the show, but if you just go to aif360mybluemix.net There's a toolkit there where you can experiment with their tools for fairness and analyzing data sets and, and modifying models and all of those sorts of things. They have a web demo, but also as a resource they have links to read more about bias mitigation concepts, terminology. They have a Slack channel where you can ask questions related to that. They also have tutorials that show kind of examples of code that checks bias and in different industries and different applications and, and I think I'm scrolling down, it seems like there's even more here than what I remember the last time I checked it. So they're talking about all sorts of things. So disparate impact, Manhattan distance, average odds difference, equal opportunity difference, all sorts of different methods. Then also talking about adversarial, debiasing, reweighting. Really cool stuff. So I would suggest to check it out. And of course they have notebooks where you can try things and it's easy these days to spin up a notebook on Colab or other resources to try out a toolkit like this.

Absolutely. Sounds good.

Yeah. Awesome. Well, great to go through this with you Chris. Interested to see what the index looks like next year, but it was great to talk through it with you and looking forward to a great year of.

AI, as am I. Sounds good. Talk to you later. Daniel, thanks.

All right, thank you for tuning into this episode of Practical AI. If you enjoyed this show, do us a favor. Go on itunes, give us a rating, go in your podcast app and favorite it. If you are on Twitter or a social network, share a link with a friend. Whatever you gotta do, share the show with a friend. If you enjoyed it and bandwidth for Changelog is provided by Fastly. Learn more@fastly.com and we catch our errors before our users do here at Changelog because of Rollbar. Check them out@robot.com changelog and we're hosting on Linode cloud servers. Head to linode.com changelog check them out. Support this show this episode is hosted by Daniel Whitenack and Chris Benson. The music is by Breakmaster Cylinder and you can find more shows just like this@changelog.com when you go there, pop in your email address. Get our weekly email keeping you up to date with the news and podcast for developers in your inbox every single week. Thanks for tuning in. We'll see you next week.

Sam.