Stanford HAI 2019 Fall Conference - AI and the Economy

Thanks, Ken. Silicon Valley, we have a problem. We've taken advantage of some amazing technologies, met some grand challenges that built self-driving cars, robotics, a whole host of cognitive and perception challenges. But there's no economic law that everyone is going to benefit from all of these challenges. It's possible for many people to be left behind, and indeed many people have. And that's why I think the challenge that is most urgent now is not simply more and better technology, although I'm all for that, but creating shared prosperity. And I'm glad we have this conference here to help address some of these sorts of challenges. The data are stark. If you look, for instance, at the contrast between what's happened with productivity versus what's happened to median income, you see that there's been what Andrew McAfee and I call the great decoupling. And for most of the 20th century, those rose in tandem. More production, more wealth, more productivity went hand in hand with the typical person being better off. But recently those lines have diverged. Median income has hardly moved at all since the year 2000. How is that possible? Well, the pie is getting bigger. We're creating more wealth, but it's going to a smaller and smaller subset of people, and that means that a lot of people are being left behind. Not just in statistics like economic growth and income, but ominously now we're even seeing it in life expectancy. For the first time in recorded American history, life expectancy started to fall 2 or 3 years ago, especially for those who are less educated and have lower incomes. Now the problem is not with the technology. Fei-Fei Li and what The team, the folks have done with ImageNet has been remarkable. We see increasingly rapid progress as neural nets are being introduced to solve a whole set of problems in vision systems, things that we thought were very difficult previously, or in voice recognition, or really in a whole set of different categories, anything mapping a set of inputs into a set of outputs, X. While we're very far from artificial general intelligence, we are able to solve more and more problems, and I'm sure many people in this room are working on some of these specific applications here and making progress in those areas. But when you look at how that's affecting the workforce, there's a number of different possibilities. These technologies can be used to augment or complement human labor, or they can be used to substitute or replace human labor. And in many cases, it's been used primarily to replace labor. And working with Tom Mitchell and Daniel Rock at MIT, We've looked at the kinds of tasks that are most suitable for machine learning systems, this particular set of technologies. And we find that they're not at all evenly distributed throughout the economy or across different tasks. For instance, if you look at all the occupations in the US economy, each of these red dots is an occupation, you can see that if you array them from the lowest paid ones to the highest paid ones, there's a clear pattern there. Lower paid ones tend to have more tasks that are suitable for machine learning. For instance, cashiers. There's a lot of those tasks that are already being automated. But of course, there are also some high-paid occupations. One of the highest-paid occupations in our economy is airline pilot. And a lot of that work is increasingly automatable as well. I couldn't help resist— I couldn't resist peeking at where the economists are. Down there. So maybe not as high-paid as they should be, some say. But so far, not as suitable for machine learning as some of the other ones. But you can go through it, and you can see a bit of a pattern and some changes. You can also look geographically. And it's hard to tell. Silicon Valley is very concentrated there, but it's doing pretty well in terms of suitability for machine learning. But you go down the coast a little bit, or you go to parts of the interior, and there are big chunks of the economy where there are many jobs, many tasks that have a high quotient of suitability for machine learning. And if you look at the economy overall, there's a tidal wave coming that's barely hit yet. If you look at just the one category of the most suitable for machine learning tasks, those tasks that have the most characteristics that make machines likely to be able to do them, that accounts for about— was it $713 billion worth of economic activity? And that's just that one category. Yeah. high category. Looking at different firms, each of these dots now is a firm in the economy. There's actually not much of a correlation with market value. I don't think that the market has priced this in yet. But my prediction is if I come back in a year or 2 and do this chart again, that line will be much more tilted and we will see a bigger and bigger effect. So I'm hoping that today we'll have a chance to talk a bit about this grand challenge of how we can create shared prosperity before this wave of new machine learning technologies affects as many jobs in the economy as it's likely to affect over the next 5 or 10 years. Yes. And have the human, the organizational, and the economic side of our research and our policies catch up with the remarkable progress that we're seeing on the technology side. Thanks.

Okay. So, let's dive in. A broad observation from the marketplace that I live in, enterprise software largely, one of the things that I'm seeing is that there is a sort of a malaise, a little bit of a disillusionment with the perception of the ROI that enterprises are getting from AI in general. Mm-hmm. There's a sense that we expected so much more, so much sooner and it's not happening. And so, you know, this, and I think this supports some of your work on productivity, Eric, but I'd love to throw this to you, Susan. How do you think about this? Do you agree? Do you disagree?

Yeah, I think it's kind of interesting that we're seeing these paradoxical outcomes. We're seeing sort of very fast adoption in some places and very slow adoption in other places. When I try to break that down, I see that when you have something that you can bring in as a service or that's light touch or that isn't really deeply integrated in the processes of a firm, then adoption can go very quickly. So things like, you know, image recognition, optimal character— optical character recognition, trying to analyze text, these types of things are often layers you can put on top or things that companies can buy as services also. Things like in digital marketing, just over the, at the edge of that, rather than the part where you've integrated into understanding your customer. Mm-hm. But then where I see firms, even sophisticated firms, taking much longer is anything that really involves getting into the guts of the firm and sort of reworking their systems. So we can also see that on that path to using machine learning and AI, you have to have a lot of infrastructure, have the data set up, and companies can take, you know, 5 or 6 years to transition something to the cloud. So you see a lot of the companies that I work with, a lot of the applications that are put into place, there's this huge investment phase up front. But then actually, either for a newly born firm or for the parts of what you're doing that are in the cloud or that are set up in a more flexible way that have the data available, then they go very, very fast. And they innovate very quickly. There's a lot of A/B testing, a lot of data-driven innovation that moves very quickly. So I think that's one of the reasons that, you know, when you go to these conferences, you know, some people put up like story after story after story, and they're real and they're impressive, and you see like, wow, I saved all this money, this is so much more efficient. But then when you look out in the data, you don't necessarily see it coming up in the measurement, is that the bulk of people are still in the slogging it through phase. And then I think the second part is then, you know, what is really going to move the needle the most, and how many tasks can you really fully outsource to automation? And that gets into how you organize work and how you think about the future of work. And although there's many specific tasks that are easy to automate, and Eric talked about this, Actually, giving, like, full autonomy to an agent is actually often quite hard, and that comes down to issues of measurability, that you often can't measure the thing you really care about in the right timeframe, and because you can't measure the thing you care about in the right timeframe, you can't incorporate it into the AI and have it really optimized for the things you care about, and so there's still this role for all the things that were hard to write down, that were hard to digitize, that were hard to incorporate in the objective. And so in the end, you, you end up actually needing to provide sort of a coach or a copilot or an augmentation to the human rather than a replacement. But then once you realize that, you realize this is actually quite a hard problem. Mm-hm. Because working, you know, getting the, the humans are much more complicated to work with than just, you know, outsourcing a task. And actually, you know, humans have opinions. They're not sure whether to trust the machine. They might, you know, care about outcomes in ways that conflict with the objectives of the firm or that actually just incorporate things that, again, are hard to codify. And so this whole process of really understanding what is the domain, who are the decision makers, what has to happen for something to be adopted, what has to happen for it to be accepted, That becomes cross-disciplinary within a firm. It's cross-disciplinary within a university. And it's just harder and slower. So this is actually, I think, points into some of the research that we want to do here at Haas as well. I think both in the business world and in academia, the real benefits will come when we really dive into the applications and understand the sort of the entire vertical, everything through implementation, including the ethics, and the feelings of the people involved who are adopting. But that's slower, takes longer, requires more specialization and interdisciplinary work. And I think that's one of the reasons that it's gonna go slow for the big benefits.

Yeah, I would observe it seems like you covered a lot of ground there. 80% of the time, talent and money is often— involves the front end around data infrastructure. And establishing that. And that's almost always underwhelmed as companies move into, you know, trying to benefit from AI. So that's one of the challenges that I see. But then, as you rightly point out, there's this whole organizational issue where, you know, working between IT, traditional IT, which is changing rapidly and becoming a little bit of the water company around focusing on security and privacy, versus the line of business people and the developers. who are trying to get product to market, and there's a kerfuffle that goes on there. So, as you rightly point out, this whole organizational issue is part of the delay. But Eric, why don't we move to this question of big and small? Mm-hmm. I'm interested in, you know, who benefits. We've seen these cycles where roughly every, pick a number, 10 years, we see sort of the Fortune 50 or so, a lot of disruption. Companies come in, companies come out, and I think being able to execute in this space is probably going to be one of the drivers. So who benefits? Do you think it's the small companies who are smaller entrepreneurial companies who are working from a whole clean sheet of paper or whole cloth, or the larger companies who have the benefit of scale?

Well, I think that the framework that Susan laid out is a great way to think about this, and Just as we need to reinvent our organizations to create shared prosperity and help things at the economy level, companies need to do that reinvention as well. And you hit on exactly the trade-off, which is the bigger companies do have the benefits of scale. And if you can get a system working— we were doing some— took a look at what eBay was doing when they rolled out a new machine translation system. It didn't require a lot of organizational change, so they were able to quickly scale it to hundreds of millions of And so as a bigger company, they got more benefit from that system than a smaller company would. But more often, you do need to make significant process change, significant organizational change, and then being a big company can be a hindrance because you've got inertia. You've got things that are working well that were optimized for a set of technologies, and it's not just that you understand how they all fit together. Having served on the boards of some big companies, companies don't even know what they know. And there are so many subtle interactions, complementarities we call them, between different business processes, between the skills the workforce has, between the product line, between what customers are expecting, the sales channels. And if you try to change one of those to take advantage of technology, it becomes out of sync. Think of a Swiss watch. You can't just take one of the gears out and put in a digital transistor and expect the watch to be a little more digital. It doesn't work that way. You have to rethink the whole mechanism, how it fits together. That, as you suggested, can be a lot harder for bigger companies. And one of the reasons I think we've seen a productivity paradox, remarkable eye-popping technologies, yet relatively slow productivity growth, is that there hasn't been enough effort on that other 80%. Actually, I would up it to 90%. Mm-hmm. I use a metaphor sometimes of an iceberg, that the cool technologies is just the most visible part of it, but the organizational business process, even cultural and skill change, is the bulk of the transition. And I should say, by the way, this is not the first time this has ever happened. We went back and we looked at the introduction of electricity, internal combustion engines, steam engines. Each time there were literally decades-long delays as the economy and as business was reinvented. And so I think one of the important things that Hai is doing is focusing not just on eye-popping technologies, but also on this organizational and human side. That increasingly is where the bottleneck is. And if we want to unleash productivity and hopefully shared prosperity, we need to focus more on that.

Right. So the question of measurement, you mentioned it, Susan, going in. There's— so you have measures of productivity. I think about measures of convenience. and maybe even measurements of entertainment. Those 3 are probably interchangeable at some level. But how do you think about measuring utility relative to those kinds of things? As it regards benefit to people.

Yeah, absolutely. So I think I'll take this, and Eric's also done a bunch of research on this as well. But one of the things that I'm working with a lot of social impact organizations now who are really, their main goal is to improve outcomes, social impact outcomes. But trying to map between the things that you observe quickly, like the time you spend in the app or the number of clicks, and the things you actually care about is quite tricky. Yeah. So one of the kind of big challenges that we're embracing is to improve that science, to really get to understand how you can detect as quickly as possible good signals of the things you care about. Trying to do that actually raises a whole host of interesting issues because those metrics can be manipulable. They can be unstable. Like, you can change something in your app or service or educational product that actually changes the information content of the signal you're measuring. So really trying to understand that. And it's also can be quite expensive because you need to map out to measurements of people in the wild, not just people who are interacting with you digitally, to relate their outcomes. That also raises another challenge in the US that we are kind of short on government data. Mm-hm. And so a lot of the infrastructure for governments to understand the efficacy of their services is also lacking. Cities have been building that, and I'm partnering with a couple of organizations including Ripple in Rhode Island, which basically has put together a lot of government data in Rhode Island to prove the efficacy of their services. But this— getting this mapping right is really tough. One of the— an example of a project we're working on there from the workforce perspective is we're trying to measure the ROI of— the return on investment for different worker training programs. And historically, workers just had to navigate these programs just by looking at a list or choosing what's closest to them. But by merging the information about the training with their government— the government administrative data on outcomes, we can actually understand which program works best for which person, and then also build an adaptive application that allows people to navigate that and get the information about what's good for them most easily. And so I think these types of problems, it's not that the idea of giving people this information is novel, but we just didn't have all of the pieces in place to do the machine learning to get the personalized treatment effects, to produce the application and have the data. But really, the biggest enabler was the data, being able to actually measure the impact of these things and help people.

Well, we have more data than ever before. We live in a digital age. People say we live in an information age. The great irony of this information age is that in many ways we have worse measures of the economy, worse measures of what's happening on than we did 30 or 50 years ago. And just as we have to reinvent our processes, we need to reinvent our whole measurement infrastructure. We've got some opportunities there. I know you've done some amazing work on detecting emotions, for instance. So there's a lot of data, the IoT, that allow us to, in principle, measure things much more accurately. But at the same time, the economy is changing. And to give you one concrete example, GDP, which is the foundation of our whole measurement edifice. Productivity is just GDP divided by hours and so forth. GDP, what does it measure? It measures all the things that we buy and sell in the economy. Well, that means that if you don't pay for something, if something has zero price, it's likely to be missed by GDP. Of course, a lot of the digital economy is not paid for. A lot of it, Wikipedia is free. So all the things on my apps, the GPS I had to use to walk over here, all free. And that means that we are missing a bigger and bigger chunk of where we get value. As Susan mentioned, I'm doing some work on trying to measure not just the things we pay for, but also the digital and other free goods that are increasing part of our lives. And eventually, we can broaden it to household production, pollution, and other things. We're looking at the value people are getting from those, not just what they're paying. And when you do that, you see that there are hundreds of billions of dollars worth of value being created that aren't recorded anywhere in our national statistics. Yeah. And if you don't have— if you're not measuring the right things, you're not going to be managing them. You're flying blind. Companies need to reinvent their measurement systems. The economy— the national statistics need to be reinvented. And we, in principle, have a lot of the tools available We have much more detailed data. We haven't made that connection to reinventing our measures.

It's really interesting. One of my observations in those comments is that there's a distinct difference between the progress being made in terms of the impact of AI on humans between consumer-facing technology versus more sort of traditional enterprise businesses. And it seems like the consumer-facing It

is, but I think that's where a lot of the measurement problems are as well. I mean, like Wikipedia, all these goods, people spend upwards of an hour on Facebook and other apps per day, and those are not measured in our GDP. So they're creating value and they're creating millionaires and billionaires, but they are not being counted in our productivity statistics. It's as if they don't exist. In fact, I In think that's a very good point. fact, to give you a concrete number there, there's a section of GDP that's assigned to the information sector. It includes software, data, movies, books, newspapers. They lump it all together and they call it information. In the early '80s, it was 4.6% of the US economy. Then we had this explosion, and now so many people in this room and elsewhere are working and benefiting from all the digital goods. What was it in 2015? 4.6%. Wow. It was like nothing happened. It was literally invisible in the statistics. So I agree that it's happening faster there. On the enterprise side, there's a different problem, or an additional problem maybe I should say, which is that there's a lot of intangible capital that's needed, organizational capital, human capital, reinvention of business processes, and that is much harder and much more difficult. We're not measuring that either, by the way, but also we're not creating it as much as as we need to be doing. We're working on some other research to try to get at that. So the enterprise side, there's a real structural problem of even creating the value in the first place. On the consumer side, it's more of a matter of it being created but not yet measured.

Yeah. So if we think about our broad economy and policies and we move to some of the other parts of the economy, I'm thinking of healthcare and government. And those seem to be moving even slower. And I'm interested in your thoughts about kind of how do we have an impact there? How do we move the ball forward faster?

Yeah, I guess this, I think, relates to thinking about workers as well as the economy most broadly. But if I think about the lower part of the income distribution, what actually contributes to well-being? It's easy to focus on wages and wages alone, but actually your cost of living is very important. And certainly in the Bay Area, we see people commuting 2 hours each way to come here and, you know, clean our floors and work in our cafeterias. And if you could take that 2 hours and make it 1 hour, that would be a huge benefit to people, and it would be, you know, comparable to a large pay raise. And in fact, you know, people might value that more. One of my colleagues in the business school is doing some research now that's using data about people taking sort of gig work and showing that especially women value their commuting time enormously. And that's presumably because people have, you know, family commitments and being away from caregiving responsibilities, it's not just money. You know, you actually sometimes just need somebody there. So if we think about improving transportation and directing some of this innovation and also policy towards transportation. Of course, transportation and housing are two sides of the same coin, because if you can get places faster, that makes a bigger supply of land and thus housing cheaper. So when we think about the direction of autonomy, for example, for vehicles, if we could more quickly get to cheaper public transportation that would move people longer distances, or semi-public transportation or last mile that helps people people get from places with good housing, that's super important. So housing and transportation are the big buckets, then education, healthcare, food. All of these things in principle can be— have technology make them more efficient. We can make healthcare much more efficient through technology. Right. Historically, the technology's actually made it more expensive, and that has to do with competition and regulatory failures. in principle possible to get healthcare to people more efficiently. Again, meet people where they are, value people's time, whether it's telemedicine, getting people just that answer they need in 5 minutes at 10 o'clock at night rather than making them schlep to the doctor. And then education is another thing. Again, we have very big inequality of access to education. We've had a lot of failures in terms of just seeing, oh, throwing technology at education hasn't moved the needle. But perhaps the next generation of education technology will, in fact, be more beneficial, partly by realizing that the technology needs to be combined with humans. Some of the partnerships I'm working on now involve apps that help parents, even less literate parents, read with their children, and that facilitate the process, get the right reading material, interesting, exciting reading material, and the right difficulty doses that keep people involved. Helping teachers manage their classrooms and differentiate better among their students, teach executive function. So these types of— I say the second generation of bringing technology to students has potential to solve some of the problems of the first round. And of course there's also educating adults and training. And again, through the mobile phone and through more personalization and customization, we can meet people where they are. But there's still a lot of gaps that have to be solved. How do people know what's the right way to invest their time for their children, for themselves as workers and adults, for their families? How do people— how does the labor market recognize different forms of training and education? So again, the technology is just a piece of it. You have to understand the context into which you're putting the technology in order to get the biggest gains. A final thing is competition policy. So getting back to Eric's point about what's measured where, in principle, in competitive markets, a lot of the gains from this technology should go to consumers. We get more efficient, we get more productive, that leads to lower-priced goods. People can get the things they want more cheaply. It's produced more cheaply, it's delivered more cheaply, that's a lower cost of living. But if we have too much market power in any part of the supply chain, whether it's the robotics or, you know, the platforms through which people find goods and information, too much market power there can lead to high prices. And so then you can get low wages or less employment but higher prices. And that's the disaster we really, I think, need to avoid. If we get more efficient, in general, a lot of those gains can naturally be spread. But those gains need to occur in the right areas, and they need to get passed on to consumers.

Great, thank you. So those of us that are chasing returns, that are investing, we— my observation is, if you look at the data, AI investing is gonna reach another, goal or target, and we're going to exceed what we've done in investing over the last 2 years. So despite the fact that we're not seeing the ROI that we expected, at least at the enterprises who are investing, we're still pouring money in. We're still driving ahead. And the implication there is that this is inevitable. We're going to push through, and 7, 10 years down the road, we're all going to see nice returns from this. this investing. Challenge, we don't have— we have difficulty finding enough talent. And what we're seeing is that the talent that we need, the interdisciplinary talent, tends to be at the university, at least the talent you can afford. Because the folks that we haven't seen— you can't afford the 2,500 PhDs they have at Google. Those people are too well compensated. They got our— They're not here. So if you're going to go build teams, you're going to have to go. So universities are a critical part of this journey. And when you work with academicians, as I learned, my investment partner and I found this intact team. And the interdisciplinary part is really important. We needed behavioral cog science, computer vision, machine learning, deep learning talent. And we did our research. We found an intact team. with those 3 disciplines, fortunately, and pulled them out. But with that team, we found they gave us a perspective on the work that we were doing and the businesses that we were driving towards that, you know, was different. They worried about a lot of other things that typical business people don't just worry about. Yeah. I had to explain often why we were going to go down a certain path, how we were going to go down, and they would push me on, well, is that fair? Is that the right thing to do? How are we going to use that data? How do we protect it? So these were much longer conversations than I've ever had in, in my business world as a leader, and they were very valuable. I'm interested in kind of how you see the universities contributing to the work that people like me do.

Well, financial rewards are incredibly powerful incentive and motivator. We're economists, so we spend a lot of time studying the power of those kinds of incentives. But they're not the only motivator, and those of us at universities are often motivated by a lot of other things as well. Everybody is motivated by lots of other things as well. And one of the reasons you mentioned the progress in the consumer internet, part of that is that it's measurable, it's easy to translate that into financial quantitative rewards. But there are so many other issues like the ones Susan, you know, made a great description of what's going on in healthcare and education elsewhere. where it's not as easy to map onto a financial reward. And that's why I think we have to understand that, you know, I'm an optimist about using these technologies to transform the economy and create human-centered AI. But we can't assume it's just going to happen automatically if people just, you know, as you said, chase financial returns. Yeah. That's part of it. But we also have to think harder about what our values are and what it is that we want to create in our society. And I'm an optimist who believes that if we first understand what our values are, we're gonna be able to direct these technologies in a better way. We have right now, I think, the most powerful technologies that humans have ever had. More powerful technologies by definition means we have more power to change the world. And I think a direct implication of that is that when you have more power to change the world, Your values matter more than before. Now you have to rethink what is it that we want to do. We actually can reshape it. We can do a lot to create shared prosperity. We can address those challenges in healthcare and education and elsewhere. So it's time to have a really hard conversation about what we want to use these awesome, powerful tools for and ultimately what kind of values, what kind of world we want to create.

Well, we reached the end of our time. One of my joys in life is hanging out with really smart people like Susan and Eric. So help me Thank them for their time this morning. Thank you.