Finding success with AI in the enterprise

Bandwidth for Changelog is provided by Fastly. Learn more@fastly.com we move fast and fix things here at Changelog because of Rollbar. Check them out@robar.com and we're hosted on Linode servers. Head to linode.com changelog. Welcome to Practical AI, a weekly podcast about making artificial intelligence practical, productive and accessible to everyone. This is where conversations around AI, machine learning and lines happen. Join the community and slack with us around various topics of the show@changelog.com community. Follow us on Twitter. We're @practical aifm. And now onto the show.

Welcome to another episode of Practical AI. This is the podcast where we try to make AI practical, productive and accessible to everyone. I am Chris Benson, one of your co hosts. I am the Chief AI Strategist at Lockheed Martin, RMS API Innovations, and with me is my co host Daniel Whitenack, a data scientist with SIL International. How's it going today, Daniel?

It's going great. How about with you, Chris?

Doing good. What you been up to lately?

Well, I finished out my course that I was teaching at Purdue University, so I'm enjoying one grading and then throwing some eggnog in there when I can. When I compare the two, that's working out well.

Sounds great. As I mentioned as we opened up, I actually started this new job at Lockheed Martin. Very excited about it, been ramping up on that and I've never worked for a defense contractor before, so I'm learning all sorts of new things about how to apply AI and it has been absolutely fascinating the last couple of weeks doing that.

Yeah, it's exciting. Don't share too much or you'll have to kill us, I'm sure.

Yeah, I'll have to kill myself. So there we go. So I wanted to introduce our guest today. Guest is become a good friend of mine in recent months. Susan Ettlinger is an industry analyst with Altimeter, which is a profit company, and Susan and I met at the Adobe AI think tank earlier this year in New York City where she moderated a 90 minute broadcast on Facebook and I was privileged enough to be one of the people on the panel. How's it going today, Susan?

I'm great, thank you. It made it sound like we spent the entire 90 minutes talking about Facebook.

But we actually talked about AI very tr.

I'm glad you said that. Very, very true. Yeah, we had a great panel and talked about AI with a lot of really smart people. They were able to contribute to that conversation and so it was a great, great time to meet and I've enjoyed talking to you ever since. And it became obvious really, really early on that I had to try to twist your arm to get to see if you would come onto our podcast because there is so much about the world of AI that you know, and fortunately for us, you have just. I know that you have been working on a report that is fascinating called the MA Model for AI and Enterprise, where you're talking about AI in, in enterprise, in the industry. And I was wondering if we could start off with you just telling us a bit about that.

Yeah, absolutely. Actually, it's just gone live as we're recording this. So by the time this podcast airs, everybody's going to be able to see it. So what I've been trying to do over the course of the past, you know, depends on how you count it, several months to several years, is, is understand a little bit about the way that artificial intelligence is evolving not just as a technology or as a, as a kind of societal impact, but also just in terms of the impact on business because the impact on business is so different in so many ways, the kind of enterprise impact versus the consumer impact that I wanted to try to get a handle on it. So this report is about two major things. One is kind of what are the four trends that are really affecting the way that enterprises implement AI? And those four trends have to do with how we interact, moving from rules screens to senses, so moving from URLs to kind of speech and images and that sort of thing. The next is around how we decide and that is moving from the old way of programming if then statements from business rules to probabilities, because AI is of course inherently probabilistic. The third is around how we innovate and in the past or actually in the current, in the future, we're going to go to more of a kind of a data engineering world where we're actually incorporating data into the engineering process in a much more fluid way than we can do today. And that's something, Chris, that your insights really helped me shape today. We're kind of in many places anyway, we're in a sort of reporting on the past kind of world and we need to be able to use data in a much more forward thinking way. And then the last is around how we lead because we live in a world that's very hierarchical, that's very expertise driven. And of course data and the ability to get clean data is going to help us make decisions based on data.

I've had the ability to go ahead and read this. I know, you had quoted me as one of a number of people in the article and let me see a preview. And the big thing that I really was thinking through this process was how much I wish I had had this over the last couple of years as I worked for previous employers and trying to put together the business case and the operational aspects of AI teams. So, you know, we're starting to see organizations like Google and Amazon and stuff offering up some of their internals. But this, the AI maturity playbook that you've put out is a huge, huge tool to get people started in this and wish I'd had it all along. And Daniel, were you going to say something?

I was just going to say it's. It's interesting that the thing that stood out to me when you were talking through that is, is kind of the emphasis on engineering that you were talking about an integration within a company's infrastructure. And I don't know if you've seen this, maybe you can comment on this, but it seems like we've seen a trend, at least when I'm looking at like job postings and people's titles and such. There was kind of a time when we were talking about, oh, everybody needs to be a data scientist and we're all going to use data for stuff. And then like, it kind of moved into everybody needs to be doing like AI and be an AI person or machine learning person, scientist. And then like now it's kind of drifted into. I see a lot of job titles looking for like machine learning engineers or AI engineers or data science engineers, whatever that is. But I think it's like people are gradually coming to the realization that they actually have to do some type of integration of this stuff in their infrastructure. Yeah. And I don't know, are people feeling that pain? What's pushing that side of things forward?

Yeah. Well, it's funny because this topic, this issue of sort of data analytics to data science to data engineering really popped up in my interview with Chris. Not to do too much log rolling here, but I mean, it's really where I started to think about it. And so in the subsequent interviews when I spoke to other people, I asked them, and even just people that I know in the industry who weren't necessarily formally interviewed for the report. And I'd say, so what, you know, how is this working for you? If it's a startup, what are you seeing with your clients? Or if it's a big enterprise platform, kind of enterprise platform, what are you seeing? And then in enterprise companies, I was asking them, like, Kind of, where are you on this spectrum? And they're like, oh, yeah, yeah. Because we've brought in all these data scientists who have a very particular way of working. And the challenge is getting to scale, you know, and getting to be able to build not just models, but products that we can scale across the organization. And that's a whole, you know, not only a technical challenge, but a cultural one as well. And also a recruiting challenge in terms of trying to figure out what are the qualities we should be hiring for in order to be able to build scalable infrastructure. So that's been, you know, that's been kind of a scalable products, and that's been kind of a big theme really, that I wasn't, I didn't really know to expect.

When we had that conversation, Susan, and we're discussing that, it was. I found it in my own experience as I went into a previous employer and was creating a full AI operation within that organization, that a big surprise for me had been that I was hiring on some new people and I was pulling people from other parts of the organization and I had a mixture of skills there. And some of our team members were just straight data scientists in a lot of cases, fresh out of school. And that had been their exclusive focus. And being this new field of neural network model creation and such, I think myself and others on the team really expected that to be the strongest skill set. And what we were surprised to find was some of the other members of the team had already been in industry and had created products and services for other companies or previously for the same company. They had been programmers in various other roles, and they had moved in and maybe gone back to school in some cases for data science and to learn this. And I was surprised that those people were able to apply the after model creation, that they were able to apply that better after the fact. And so in some ways, potentially the people who had focused exclusively on this had had a leg up. But as soon as some of the others caught up with them, the fact that they knew how to deploy and how to meet a business need from, in terms of products and services was a huge advantage for that crowd. And that was something that surprised us all.

Yeah. And you know, I think what's interesting is that this seems like just part of the evolution. You know, if we think back on other technologies, how they became kind of enterprise ready, you see similar trajectories where you're hiring for a skill, and that skill may or may not come with another particular set of skills. That's a challenge with every technology, but I think particularly with AI, because there is so much hiring that comes directly out of the academic setting and it's such a different set of expectations.

So I'm curious on your opinion on, on the following in light of that and in light of the other things that you mentioned that are changing around how we will be interacting with systems, for example, and how systems will be more dynamic and reactive. Do you think for the software engineers out there that are listening to this podcast that are maybe interested in AI, I know that there's some concern amongst software engineers that they're kind of being like their job will need to drastically change and that sort of thing as AI is more integrated into the products that we're building. Do you see that software engineering as a whole is going to see a very dramatic shift, or will it more be like AI is just going to be something they interact with, but it'll be another layer in the stack or something like that?

Yeah, well, it's funny, Daniel, I mean, I can answer as a non software engineer, just in terms of what I've observed. And what I've observed is I don't think I've ever seen a software engineer who hasn't had to change, you know, who hasn't had to evolve their skills, who hasn't had to figure out something that they weren't expecting. You know, if you think back to the beginnings of the Internet, like that was a massive, massive change in the mid-90s and the early 2000s, you know, and the development of even like social technologies and so, and mobile technologies and all of that, you know, every single time there's a massive shift, there's a massive set of changes that reverberate through the industry. And I just don't ever see that changing, you know. And then in terms of kind of the long view, I do think that learning, you know, intelligent systems, the ability to learn from data, autonomous systems, that's going to be table stakes. I don't know how many years, you know, I can't, I don't have a crystal ball. But it's going to be what we're, what we're thinking of as sort of exotic now is going to be table stakes. And that's really a lot of the thrust of the report too.

So I know that having had the advantage of seeing it ahead of time, you started off the report kind of talking about some of the macro, macro trends that would affect AI and stuff. And you were really thoughtful in how you were approaching how kind of the real world would affect this. You talked to, I remember about kind of the interactions that we're having with computing. I remember one of the sections was talking about as we move from screens to different senses that we may not have used historically. And then I believe you went on to kind of how we decide, how we innovate, how we lead. And I was just kind of wondering what some of those insights were that we could share with our listeners.

So, you know, the screen thing, you know, how we interact is really interesting because we're just so used to, you know, if you're older than like, you know, 30, you know, you're used to interacting with a laptop computer or even a desktop computer and a phone. You know, if you're younger than 30, more of your life has been spent, you know, talking to your phone and talking to that weird little cylinder on your dining room table or, or your thermostat or whatever it is that you're talking to. We're certainly becoming much more accepting of things like facial recognition and image recognition, although obviously that comes with issues. And there are even people who are working on sensory based interactions based on smell and taste. So none of our senses is actually, none of our senses is going to be left behind. And of course, touch, using haptics and pinch and zoom are all very normal to us now. You know, and you go back 10 or 15 years and like that was just. That was Minority Report. That was something that lived in science fiction. The biggest shift to me though, of all of these shifts is around how we make decisions because we are so used to living in a world that is based on if then statements. If my balance drops below $500, send me an alert. If I make a transaction more than $300, send me an alert. If I do a transaction. If I try to buy something in an airport in Berlin, decline my credit card. And now what we're seeing is that the world is a lot more probabilistic. And sometimes that's fantastic and it's really easy to understand and it's intuitive and sometimes it actually creates a lot of stress for organizations because you could say something with an 85 or 87% confidence level is fine for one industry and completely off the table for something else.

I mean, I imagine that that creates, I don't know, in my mind, I'm thinking for a lot of people, maybe including myself in certain scenarios, that creates a lot of trust issues. Right. It might be harder for me to understand naturally, the probabilistic way of dealing with all of these complicated scenarios. But I kind of have to put my trust in the modeling at that point. Right. And not just kind of in an easily understandable if then statement.

Yeah, absolutely. The thing is too, that it's not just about putting your trust in the model. Right. It's the engineering and user interface and other kinds of communicative decisions that are made to let you know whether you should trust the data. So I'll give you an example. And this sort of almost kind of is a nice segue into the conversation about ethics in Turkish. And as in many other languages, there are no gendered pronouns. So the word for he and the word for she are the same. It's actually the word O with the letter o. And if you take the sentence she is a doctor on Google Translate, you can do this yourself and you translate it into Turkish, it will come back with o Bir. Doctor. Sorry about the pronunciation, Turkish speakers. And then if you take ober doktor and you translate that back into English, Google will assume and write, he is a doctor. Now, this is probabilistic because if you look at the Word2VEC data set, we know already that the word doctor, as many other professions, is biased toward male humans because there are more instances in that data of men being doctors than women being doctors. And Even if it's 50.5%, you know it's going to be a man. And so there. So here's the thing. The language, the Turkish language has been around a bit longer than Google and yet and is not likely to change, you know, for Google's sake. And yet there is no indication when you do Google Translate that what you're looking at when it says doctor, you've probably got a 97, 98% probability that it's correct. But when you're looking at the O that signifies the gender of the human being discussed, that, you know, it's way, way, way lower. And so what I'm saying is that sometimes we actually need to incorporate into engineering and into user interface design some indication for people that what they're looking at may or may not require further analysis.

Yeah. And I do think that this leads right into a great discussion on ethics, which I'm eager to get into. But before we kind of jump into those details, I'm wondering if, kind of based on what you were just saying, those are kind of real problems, real biases, real kind of dangers, if you want to put it that way, that exists right now in machine learning and AI. I'm wondering like, like so much of the conversation around the danger of AI and other things, people kind of naturally go to the scenario of the Terminator scenario or consciousness or something, right? Do you think that that distracts from these real kind of dangers and biases that we're experiencing now? And should we even be having that conversation? Or should we as practitioners kind of, how can we help bring a more balanced view into what we should really be talking about in terms of ethics, I guess is my question.

Any of us who work in this field, somebody like me, who's an analyst, really, with a humanities background, versus you guys who have much deeper technological chops than I could ever hope to have. Maybe you hang out with your family at the holidays and they ask you what you're working on, you say AI. And they're like, when are the robots coming to get us? That's the conversation really that much of the world is having, right? That the trolley problem, you know, if the car's driving down the road and it has the, you know, it's going to kill one person or five people, or it's going to kill you or a woman with a stroller, like all that kind of stuff is where people, people's minds naturally go to. And I'm not saying that those are trivial issues. Obviously they're not. And when you get into things like autonomous weaponry, I mean, that's a whole other topic. But AI isn't a monolith. And so when we think about the, both the benefit, you know, the innovation benefit and the risks of AI, we have to think about it in particular context. And that context could be something like a financial services context in which you're trying to manage risk, or it could be a diagnostic context in the healthcare industry. And so what I really think is important is for us to understand some of these nearer term issues, some of these very pragmatic practical issues around what happens when we use algorithms to kind of abstract humanity. Just not that that's bad per se, it's just that it has implications that we then have to deal with on the other end. And so this is part of responsibly learning to use the technology, just as we would responsibly learn to use any other technology that is extremely powerful.

So, Susan, as we are kind of talking about how about what ethics are in AI and how to apply them, which is very personal for me as I've come into a new job in a new company, new industry, the defense industry, where we're looking at AI use cases, I think this is the first time in my life where I'm Almost leading with ethics. And I think, you know, there are many other people that will be in similar situations because AI has such tremendous capabilities. What types of advice do you have for people who are moving into jobs or are now having to face how does AI affect our products and services at our company? What kinds of things would you advise them to do in terms of their thinking that maybe they haven't had to consider in the past?

Yeah, I mean, I think there are a few very straightforward things. The first is to understand that algorithms are as good as the data. You know, this is like the classic garbage in, garbage out. Right. The algorithms are only as good as the data and the way the data is modeled. And, you know, the data that we have in many cases is simply it's just absorbed from society. Right. You know, in the case of the Google Word 2 vec, or in the case of the Word 2 vec data set that includes all that language stuff that I mentioned earlier, you know, it just absorbs the reality that we live in. And sometimes you want to perpetuate and amplify that reality, and sometimes you maybe don't. So, for example, if you're creating a segment, if you're a marketer and you want to do audience segmentation to doctors, you don't want anything. Assuming that all doctors are male. Right. You're going to alienate all those female doctors out there and potentially even stifle the potential of younger female, you know, students who maybe want to get into the medical profession. So we just need to know these things and we need to actually have processes in place to ensure that when we can and can fix and catch bias, that we do. We can't change society by changing technology, obviously, but we can be mindful about it. The second thing is around explainability. So there's a woman, Rachel Bellamy from IBM, I heard her speak in London not too long ago, and she said explainability is the new user interface interface for AI. And I thought that was a really interesting point because one of the things we're not used to in probabilistic systems is the idea that you put data in and then there's this sort of black box and then there's the output. And so in many cases, we do need to understand what some of these, you know, what some of these decision criteria were. In some cases, it's fairly straightforward. You know, maybe there are a few kind of keywords that were determining the outcome or suggesting the outcome. And in some cases, for example, maybe with disease diagnosis or with pharmacological types of use cases, it might be very, very complex, you know, or weather, you know, these very complex systems. So this idea of trying to understand the, you know, what happened between the input and the output is very important so that people do have a sense of trust. You don't simply say, well, Chris, I'm not giving you a mortgage loan, even though you have financially pretty much the same profile as Daniel. And then three months later you give Daniel the mortgage loan, even though he pretty much matched where you match. You have to be able to go back and understand what happened. You have to understand a little bit about what caused that action to be taken. So explainability is interesting and it's also become kind of a huge issue in the industry, and I think there's a lot of controversy around it. And then the third piece, piece is there needs to be an understanding that ethics and AI are simply just norms of behavior. And we don't really have norms of behavior in the digital world the way that we do in the physical world. You know, not to push in front of somebody getting on a bus. You may do it anyway, but, you know, not to do that. We don't have those same norms in the digital world. And so having internal controls making explicit the decision criteria, all those things are really important.

I'm glad that you addressed that, because that was actually going to be. My next question is kind of what do you need in place around it in terms of what you're calling internal controls so that the burden isn't entirely on the individual that is trying to figure their way through this and apply ethics, you know, as they do that from an internal control. Do you need systems in kind of AI implementation that you might not have needed in other environments? And if so, you know, know, what might they need to be thinking about? What, what might those systems need to be addressing?

Yeah, we do need systems. In some cases it's like grandfathering existing, existing processes and controls, you know, grandfathering AI into that. In other cases it's entirely new, entirely new types of controls. So, for example, some industry examples out there, AI now, which is a really phenomenal organization focused on ethics and AI, they've issued what they're calling an algorithm, algorithmic impact assessment, very similar to like an environmental impact assessment, that when you're going to build something or excavate something, that you need to understand the environmental impact. So this is built on that same premise that, you know, if you're going to introduce algorithms and algorithmic decision making into, in this case, it's, it's meant for Governments and for cities. If you're going to introduce that into kind of a civic environment, that you need to think through some of those potential impacts to vulnerable people, to systems and processes and all those things. And so that document lays out kind of a template for assessing the impact of your algorithmic system. I think something like that can and should be customized for industry. That's one example. IBM has built a couple things that are quite interesting. One is called a supplier declaration of conformity. So imagine, you know, as a defense contractor, or as a retail bank, or as a healthcare provider. You're not only using your data, but you're using data and systems from other organizations, other companies. You want to make sure that you've gone through the process of understanding and holding your systems up to the highest scrutiny, but you also want to make sure that your suppliers and vendors and partners have done the same thing. So that's another example they've also built. And this is again something that's a bit, I wouldn't say controversial, but it's open to scrutiny. This idea of a dashboard that shows kind of a bias quotient, right? And a confidence quotient. As a simple example, if you're trying to settle car insurance claims, you should know that the data that you have for 19 year olds is very, very scant, whereas the data that you have for like 42 year olds is very, very rich. And so if you're settling a car insurance claim on a 19 year old, you need to dig down into some other things and really probably use much more human intervention to understand what the situation was, simply because those recommendations are based on just, you know, less rich data. These are just some examples of things that people are doing. Microsoft is rolling bias. Check into word and PowerPoint. So if you use a word that you're maybe not aware has some kind of connotation that is hurtful or unpleasant, it will let you know. The same way to let you know if you misspelled a word.

Yeah, that's really interesting. And piggybacking off of that for my own selfish reasons. I want to ask the next question because I've taught a few corporate workshops recently and we kind of, of course, talk about, you know, oh, you want to make your, you know, maybe you want to make your training set as representative of reality as you can. And then you try to optimize for accuracy or whatever it is. And then, you know, I bring up bias and these issues that we're talking about. And, and we, in the midst of those discussions, I think every time I've done this, someone somewhere in the audience asks the question about like, well, if we include, you know, gender or zip code or income or whatever it is in our model, makes it more accurate, why wouldn't we want to do that? Isn't that just the accurate representation of reality, even though it produces a bias model? And I know kind of how I've tried to answer that question, you know, but I was curious your thoughts on how you would help that sort of person understand why they should care maybe about bias in their predictions and why they might want to consider that a little bit more seriously and not just talk about accuracy.

See? Yeah, well, Daniel, you've hit on, I think, one of the most crucial issues around algorithmic bias we're going to see in 2019. And that is there's a little bit of a storm brewing between some data scientists and engineers and sort of people. I'll just be brave here and say people like me who run around talking about AI ethics and here's why. It's really complicated. It's not, you know, and there is a tendency, and I've had this conversation with some data scientists who work at very well known companies, off the record, there's a tendency, I think, for some folks to kind of do a little social justice virtue signaling around these darn data scientists. They don't understand people and they don't understand humanity and they're going to ruin the world by allowing bias to creep in. And then on the other.

No biggie.

Pardon me?

No biggie.

Yeah, no big. Right. And then on the other side, we have data scientists saying, well, okay, so who elected you the arbiter of all that is good and just in the world? And these are both completely valid points of view. So here's where I stand on it. We do have to have this conversation with precisely the group of people that you're talking about in a productive way. These industry conversations need to happen because as somebody who I'm not allowed to quote, said to me not too long ago, who gets to choose who's the person who puts their finger on the scale? And that is really critically important because what we may ameliorate in terms of, of bias for one group, we may actually impact for other people or have unintended consequences that we're not even able to forecast. And I'll give you one simple example. Okay, so if you think about what happened with Amazon's recognition system where it incorrectly identified John Lewis and six members of the Congressional Black Caucus as criminals, as matching faces in Their criminal facial recognition data database, you know, okay, that's like arguably. It's not even arguably. That's unarguably bad. Like bad, bad, bad. Right. You know, we've got John Lewis, who's one of the greatest civil rights activists ever known to man, who is now basically, along with six members of the Congressional black, been matched to a criminal. If this happens to John Lewis, you know, you can only imagine what's happening to other people.

Yeah. And similar with like the recidivism models and other things that I've seen.

Okay, so here's the other side. And this is. Why is this. Because image recognition and facial recog really much less accurate at recognizing and understanding people of color than it is in recognizing and understanding Caucasians. Okay, so how do you fix that? Do you make facial recognition better so that it better identifies people of color? How are you going to get that data? Do you start encouraging people of color? No, no, no, really, it'll be great for you. Just give us your face data, let us analyze your face data and put you in our system. We promise that will just help in terms of accuracy. It won't have any, you know, bad impact on you. Like, this is a really, you know, who's going to say yes to that? Right. And so this is, you know, some people will say, you know, we're, we're perfectly happy, you know, that the false positive rate is so high. Like, just let it stay high because we don't want to be included in those systems. And, you know, there are absolutely valid reasons for that. So, you know, this stuff is not easy. And one thing I would say is I don't stand on a soapbox, you know, trying to say I'm more ethical than anyone else. I am cowed every single day by how complicated this stuff is. I just feel like we have to have these conversations.

Yeah, I appreciate your perspective there. I agree that the discussions are complicated because oftentimes immediately after I have that conversation and people are like, oh, well, we'll remove the gender column in our data set or whatever. But if there's 1200 other features, who's to say that the model still can't infer gender from those other features? So it's not just take all the sensitive data out sort of thing.

Yeah. And zip code, my God. I mean, there's no better predictor of your race than your zip code, and there's no better predictor of your health outcome than your zip code, not even your genome. So there is a way in which people could say Very disingenuously. Oh, well, we just, we didn't, you know, we, we didn't include race. Race isn't a fact. We can't do that. It's a protected class. And, you know, and then. But we just chose zip code, you know, like you. So this is why we all need to be educated about these things. Right. The business people need to be educated about proxy data, and data scientists need to kind of game out and scenario plan some of this stuff or at least be part of that conversation. And we have to get past, you know, virtue signaling and actually into some real methodologies that people can get behind.

Yeah. At least monitoring for bias, at the least.

Yeah. And that, then that's hard too, actually, because, you know, who wants to be liable for that?

Yeah.

So as if this isn't complicated enough trying to take all this into consideration, we now have the reality, regulation and stuff coming into it. Obviously in Europe you have the General Data Protection Regulation, which we call GDPR for short. And when you throw that in the mix with all the other complications of trying to be ethical in your use of AI, how does regulation impact that? You know, there's, it seems like there's quite a balancing act that you, that, that a practitioner is, is trying to manage through this process.

Yeah, I mean, you know, GDPR is really interesting. Interesting is probably a diplomatic word. I mean, I will say I am a huge fan of GDPR as a philosophy. Right. Because. And yesterday, as a matter of fact, was the 70th anniversary of the UN Declaration of Human Rights that came out of World War II, and Eleanor Roosevelt was involved in crafting that. And the whole point really was to protect the civil rights of individuals, protect their rights from unreasonable search and seizure and from discrimination and disenfranchisement and actually physical harm. All these things coming out of the Second World War. And GDPR is really built on the UN Declaration of Human Rights from a digital standpoint, so that we should be in control of our own data. We should know when algorithms make decisions about us, why those decisions were made, and be able to contest them. And so from a philosophical and historical viewpoint, it's critically important. However, most of us experience GDPR in the weeks and months leading up to May 25th of this year as an onslaught of horrific opt in emails and then not being able to get to a couple of websites that we usually frequently and not a whole lot more than that. So, you know, there's theory and practice, there's the fact that GDPR and its enforceability is a bit of a gray area for global company. I mean, if you're global, of course you have to comply just in case people do wander into the eu. But you know, fundamentally, with regulation around technology, it is always so far behind the reality of the technology. You know, we're still literally, in the wake of the 2016 election, we're still literally grappling with, you know, is Facebook a magazine or a magazine stand? I mean, that's, that's the law this is based on. And so when you think about it in those terms, I mean, yes, there does need to be protection. What protection? I am not an expert on that.

So I guess as we start to come to the end, I want to pose it, and I'm trying to not scope this final question too big. I know in this paper that you've just put out, you kind of finish up by kind of, kind of taking practitioners through how to build up their playbook. With that in mind, maybe if you could just kind of give us some pointers or some starting tips on how you might start that process, recognizing that our listeners should definitely go download the playbook that you're offering on how to build their own playbook. But what are some good finishing points where you can leave them with to get started on that process?

Yeah, I mean, what I've published is really a meta playbook, right? It's a playbook for a playbook, as you just said. And fair enough. Part of that is that as a, you know, as an analyst firm, we publish our research for free as a service to the industry. So this is really intended to help people think through the issues that they need to think through in order to do what they need to do. And of course, you know, I'd probably be beaten around the head and shoulders if I didn't say that. I'm more than happy to help with that if people need that. But there are five areas that I think are really critically important. The first is looking at your business strategy. You know, moving from kind of optimizing existing processes to actually business model innovation, customer experience, and using intelligent systems to enable those things. In data science, we're moving from kind of a specialty or an exotic specialty within organizations to the ability to scale with product and service development. We're moving from kind of reactive, taking in all the signals about what's happened in the past, to anticipatory, trying to anticipate what's happening. We're finally getting to what we were promising for the last 20 years, or an agile enterprise from an organization and culture perspective. We don't talk about this enough, but we're moving from a hierarchical to much more dynamic organizational culture. And when you have agile development in an organization, an agile mindset, it really changes the way people work together. And some people don't like that very much. And some people are highly empowered and that makes a lot of difference. And terms of how successful AI can be. So one major piece of that is you have to have the willingness to fail and fail fast. And that doesn't mean move fast and break things, because that's probably a relic of the last 10 years. But it does mean actually the ability to move in tandem very quickly, learn from mistakes and keep moving, because that's just the essence of these systems. And then finally it's around ethics and governments. We're not in the anything goes era anymore. We've seen in the last year tremendous stories about what happens when we don't pay attention to these issues. We do have to start thinking about the ethics and the customer experience of AI in a much more rigorous way. And as we talked about earlier, that's not the easiest thing to do, but at least there's some early thinking in here about how to start to frame those conversations internally.

I really appreciate it. I love what you've done with this. For our listeners, we will have a link to the AI Maturity playbook with Pillars of Enterprise, five Pillars of Enterprise Success in the show Notes. And Susan, if people read through that and they want to engage you so that you can come in and help their organization, how would they do that? How would you like people to reach out to you?

Yeah, I'd love to hear from people. So most directly you can email me@Susan AltimeterGroup. You can connect with me on LinkedIn. I'm Singer on LinkedIn and. Or I'm sorry, I'm Singer on Twitter and obviously Susan Ellinger on LinkedIn. I'm easy to find.

Fine, Great. Thank you very much for coming on the show. This was a great conversation. I so wish I had heard a conversation like this before I was getting started in industry. So I think you're really helping some people that are still trying to get in and get their organizations in involved in this and thinking about it the right way. So thank you so much for coming on the show.

It's my pleasure. Thank you both so much for having me.

All right, well, thank you very much. And Daniel, I will see you in the next show. Bye bye. Bye.

All right, thank you for tuning into this episode of Practically AI. If you enjoyed this show, do us a favor. Go on itunes and give us a rating. Go in your podcast app and favorite it if you are on Twitter or social network. Share a link with a friend. Whatever can you gotta do. Share the show with a friend if you enjoyed it and bandwidth for Changelog is provided by fastly. Learn more@fastly.com and we catch our errors before our users do here at Changelog because of Rollbar. Check them out@robot.com changelog and we're hosted on Linode Cloud Servers. Head to linode.com changelog check them out. Support this show this episode is hosted by Daniel White Knack and Chris Benson. Editing is done by Tim Smith. The music is by Breakmaster Cylinder Calendar and you can find more shows just like this@changelow.com when you go there. Pop in your email address. Get our weekly email keeping you up to date with the news and podcasts for developers in your inbox every single week. Thanks for tuning in. We'll see you next week.