Behavioral economics and AI-driven decision making

Bandwidth for Changelog is provided by Fastly. Learn more@fastly.com we move fast and fix things here at Changelog because of Rollbar. Check them out@rollbar.com and we're hosted on Linode servers. Head to linode.com changelog this episode of Practical AI is brought to you by Hired. One thing people hate doing is searching for a new job. It's so painful to search through open positions on every job board under the sun. The process to find a new job is such a mess. If only there was an eas. Well, I'm here to tell you there is. Our friends at Hired have made it so the companies send you offers with salary, benefits and even equity up front. All you have to do is answer a few questions to showcase who you are and what type of job you're looking for. They work with more than 6,000 companies, from startups to large publicly traded companies in 14 major tech hubs in North America and Europe. You get to see all of your interview requests. You can accept, reject, or make changes to their offer even before you talk with anyone. And it's totally free. This isn't going to cost you anything. It's not like you have to go there and spend money. Get this opportunity. And if you get a job through hire, they're even going to give you a bonus. Normally it's 300, but because you're a listener of practically I, it's $600 instead. Even if you're not looking for a job, you can refer a friend and Hire will send you a check for $1,337 when they accept the job. As you can see, Hire makes it too easy. Get started@hire.com PracticalAI.

Foreign.

Welcome to Practical AI, a weekly podcast about making artificial intelligence practical, productive, and accessible to everyone. This is where conversations around AI, machine learning and data science happen. Join the community and slack with us around various topics of the show@changelog.com community. Follow us on Twitter. We're @practical aifm. And now onto the show.

Well, this is Daniel, data scientist creating AI for good. And I'm joined here by Chris Benson, my co host, who's a digital transformation strategist, which is very exciting. How you doing, Chris?

Doing great today. How's it going, Daniel?

It's going well. And speaking of strategy, I think you're going to really like today's guest. We have Mike Bugembe with us today, and when I met Mike, I was really intrigued by his story. First of all, some of the things that he's worked on in the past, but also just the experience in kind of guiding a company all the way through the process of kind of defining and implementing data analytics and AI strategy within the company. And so today we're going to talk to him about all of those things. So the culture of AI and the operations of AI and, and strategy around that and when and how you can find AI use cases. So I think it's going to be really great. Welcome, Mike.

Hello.

Yeah, so why don't you tell us a little bit about yourself.

So hello, I'm Mike Bugembe. I'm an all round data evangelist and a consultant working with organizations trying to help them find value in this wonderful asset that everybody's so excited about working with them with, identifying AI solutions, working machine lear problems and so forth. And all of them seem to be really interested in building some really interesting algorithm for them and the whole spectrum of AI. And formerly before I started working as a consultant, I was the chief analytics officer for JustGiving.com a UK based company that also happens to be the world's largest online social giving platform.

Oh wow, that's, that's amazing. So I know I wasn't super familiar with just giving, but it's, it's really large. Right.

I think we're, we're bigger than a lot of people sort of initially think we are a for profit organization that operates in the nonprofit space. And so to date they've raised more than 3 billion for great causes out there. I suppose I should explain a little bit about how it works for the audience because it might not be so familiar. So just getting started in around the year 2000, 2001, essentially what they are is a digitized form of the paper form that we used to take around when we were doing an event for charity. So let's say we were running the marathon for charity. We would take a paper form to our colleagues and our friends and say, would you sponsor me for this particular charity that I'm, I'm running for? I may be running for Cancer Research UK for example. I've picked that because someone in my family has been affected by cancer and I'm really passionate about them eradicating the disease. So I take this paper form and I go around to different people and they sponsor me. Some people would say they'll give you a pound a mile. Some people just throw in 50 quid which is 50 pounds or even, or even more than that. And then companies tend to match that as well. So if you raise £500, companies will also add £500 to that form. And what Justgiving did is quite ingeniously, actually, is they took that paper form and they digitized it so that they could reach effectively more people. As we were becoming a much more connected world, digitally at least, we were able to take that form and spread it around to lots of other people who would then come onto that new page, which we call the fundraising page, and they'd be able to. To donate directly to your page, and then you'd raise the money for charity and then do the event. And that was, if you like, the mainstay of the organization. As we progressed into, you know, late 2014, 2013, around that time, we also started building a sort of individual giving platform where people could raise money for whatever course they wanted. So it didn't have to be a formal charity. It would be things like, let's say someone needed to travel to South America to see their grandmother before she passed and needed to raise funds for the plane ticket, for example, and they'd create the same page, reach out to their friends, their friends would come and donate their friends or people they knew, and then that individual would be able to take those funds and carry out the for good mission that they had initially raised the money for.

That's awesome. It's great to hear about how just giving has really empowered that sort of giving. I know. Like, you know, you can only. Only reach so many people giving them a physical form. Right? Yeah, but. But our, our digital networks are so much larger now, so that's. That's so great.

It is. It is also one of those ideas that you think, well, why didn't I think of that? It's so simple in the concept, but it works so well. And as I said, they've raised, you know, more than 3 billion for. For really good causes today. Yeah, very exciting.

That's awesome. That's amazing. I've seen you post some stuff related to animal charities and stuff. Chris, you'll have to look into this.

Oh, you just. Yeah, you just piqued my interest. When I'm not talking AI and data science and such, I'm all about animal advocacy, so.

Oh, great, great.

That's awesome. Well, now you have just giving.

Exactly, exactly. And I think one of the great things about that is it really begins to democratize the whole idea of giving, you know, moving on from the traditional ways that haven't changed very much, to be honest, where charities would sort of solicit, ask requests from individuals, or you have the. What we call chuggers on the street who are walking around with those tins asking for money here. It's really connecting people to the causes that they care about and then also connecting people to other people. So it's some really interesting AI concept. AI use cases, if you like, that come out of that as well.

Looking forward to talking about those. Yeah, that's awesome. I'd be interested to hear just a few more details on that front as far as. Because this will I think give some context for the rest of our conversation in terms of when you came on with just giving and how it's grown. If it's international at this point. I know you've been. I think it was acquired, right?

Yes.

So kind of how. When you came on and what the state was then versus kind of now and how it's grown versus in terms of like the markets that it's in and size and all of that.

Yeah. So I think I joined in 2010 and at that point I would say the organization was very much in its infancy with regards to how they would work with data. They had a lot of the traditional elements in place in terms of having a data warehouse and collecting some of the information. But culturally there was a long way that they needed to go. It was a team of one when I joined, which was essentially myself with the founders main objective saying they've got this valuable data and it is a valuable data set. If you think about it, you've got in excess of 20 million individuals from more than 100 or so different countries that have very clearly stated that they're interested in a particular cause. And more importantly, they've also said why. So for example, I'm doing this for prostate cancer because of what my grandfather went through or something like that, or I'm from a particular country where there's been an earthquake. So you can see there's a clear relationship. So millions of people connecting to thousands, hundreds of thousands of causes, telling us why they've connected to those causes and then you have an additional million people coming to support those causes and leaving all those individuals leaving comments that give us an even greater indication of other people that might be interested in those causes or why they're supporting you. It's just a breadth of information that was just sat there in that database.

Yeah, that's crazy. It seems like that is like as a data person, that really excites me. And then also I definitely see how you could leverage it for good purposes and also for somewhat nefarious purposes as well.

Yeah, sadly that's true with almost any organization. Or any data set. It's that ethical boundary of what do you actually use it for?

Yeah. So that begs the question, especially given the fact that you've been there as long as you have. So you know when you're coming into the organization and you're trying to drive the decision making process based on data at the organization and build that up, how you go about doing that, what is the best way to proceed? Can you kind of share some of your experience there?

Sure, yeah, absolutely. And I will start by saying with difficulty. And that's not just good disclaimer. And that's not just for.

You mean there's not a secret sauce, actually.

So I put together what I believe to be a secret formula that helps us. Helps organizations understand how they can generate value from that asset.

Is this. You're writing a book, right?

Indeed, yes.

Before you jump into Chris's question, tell us a little bit about your book. I'd love to hear it.

Sure.

I actually think the book is aligned to Chris's question. So sort of put them all together and bunch them all in. My response just now. But the book is called Solving the Data Puzzle primarily because it is. It's different for every organization, but it has an end state picture, an end state game. But it has many pieces that need to be put down in a specific order sometimes in order for you to eventually get the picture that you're after. And I think the nuances are very, very important, like the shapes of the. The jigsaw puzzle pieces and so forth. So that's what the title of the book is called. And essentially what it does is it highlights the. Simplifies the whole problem that we have in the data space and simplifies it to five keys that you need in order to truly begin to see value from data. And this is built from both my experience at just giving and the organizations I worked in prior to that as well as fortunately just giving. Just as an aside, I had the opportunity to consult with a lot of other organizations at the same time at a really good relations. And the founders are just giving that enabled me to do that. And as part of that, got to test these five keys to see what was required to be successful. And also working with some of the larger organizations out there, such as the likes of Facebook and Google and so forth, working with individuals at those organizations to really begin to just see what's working for them. And it all comes down to those five ingredients, essentially. So I think the question, if you. Chris, do you mind reminding me what that question was? That you had asked earlier?

Sure. Just kind of as you are coming into the organization, you're trying to figure out the best way to, to build and drive a culture of good decision making that's based on the data that you're collecting. How do you go about that process?

Great. So I think where you started is a real indication of some of the problems that we suffer commercially. Let's say you immediately said, how do we get into, I'll sort of paraphrase, how do we get into the habit of good decision making within an organization? The first task that I had was to get the organization to understand that decision making was an area that we needed to focus on. So the things that happen with data is people get very excited about what it can do. And we hear a lot of, you know, like we hear about Facebook's Edge Rank and Google's PageRank, get very excited about those sort of things. And you're. You begin looking for, okay, what's, what's our big algorithm that we're going to talk about? And just giving, we ended up coining it, Give Rank, which is, you know, not very creative. But, you know, you just get a lot of people just excited about what algorithm can we build. Forgetting, just completely forgetting that the main use case for data, and it'll be useful to see if you can challenge this main use case for data. Whether you're playing with data science or AI is for decision making. Right. And decision making beyond just your organization. It's internal decision making and external decision making. Take for example, the two algorithms that I talked about. Facebook's EdgeRank. Its purpose is to decide what content to show in order to maximize the chances of somebody coming back or enjoying their time on Facebook. Or Google's PageRank. That addresses the decision of what content should I serve or return based on the query that was sent. So these are decisions that we could do manually, but they've been automated and had some mathematical algorithm placed on top of them. So with Data, Data's main use case being decisions, you can now begin to see that organizations then need to take some time to try and understand, well, what are the key decisions we have within our product, within our operations, internally, or even with the users that use the product. Am I making any sense there?

Total sense?

Yeah, definitely.

And not only that, but when you talk about whether or not that could be challenged, I don't think it can. I think it really comes down to. To decision making can occur in different types of contexts, whether, whether it's, you know, supporting humans or whether it's automating a decision or whatever. But I don't think that. I think that your.

Your.

Your basis is. Is really firm there.

Yeah. And I think that part of my struggle in the past at a. At a couple of places where I've worked is maybe, you know, I can get people to the point of thinking about the decisions that they make on a daily basis, maybe in sales or, or in operations or whatever it is. But I have a harder time motivating them to understand that those decisions can be augmented or even leveled up in some way by data. They kind of have this kind of mindset that they have the knowledge in their head to make the decisions, but they have a harder time crossing into that realm of understanding how. How data or algorithms actually interface with those decisions. Could you kind of comment on that? And if that's something you've seen as well, or.

I think.

Am I totally.

No, no, no. You've hit the nail on the head there. I think that's. See, the most interesting thing about data is that the individuals that tend to be the experts at it are also the individuals that are poorest at interacting with people. Right. And the whole problem really here is it's a people problem, not a data problem. Right. What you've described there is the n. Natural human tendency to overestimate or give more credibility to their own opinions. Right. Which is fraught with biases. I mean, there are hundreds of books that are written on that right now. You can look at Daniel Kahneman or Dan Ariely, people who talk about the frailties in human decision making, Right. Where we're just surrounded by so many natural biases. I think there's over 120 different cognitive biases that influence our decision making. And I play a lot of tricks on some of the audiences when I'm speaking to them. I do. Like there's one that I do. For example, I would ask people, does the population of Rwanda, small African country, is that bigger or smaller than 80 million? Right. And you see, naturally there, what I'm playing with is anchoring. And most people would give an answer that isn't too far away from 80 million either side. Right. But the population may actually be less than 10. But because I've said 80, they're anchored on that. So we have a lot of those things that humans do making correct decisions with. So what we did at just giving, interestingly, was we spent a lot of time on behavioral economics, spent a lot of time in just bringing the organization up to speed and understanding the frailties in Human decision making.

So that's a pretty new term to me. Could you explain kind of what behavioral economics kind of entails?

Yeah, it's a field. I'm not too sure when it began, but the sort of key names when you type in behavioral economics are people like Dan Ariely and Daniel Kahneman. That looks at human beings as irrational beings. So whereas if you look at economics and the decision making from an economic standpoint, it looked as human beings as very rational beings who were always seeking to maximize payoff with each of the decisions that they made. But I think as research continued to carry on, they found that we weren't always behaving that way and we were actually behaving almost irrationally. And it was because of these sort of biases that came into play. Just like with the example that I gave you, because I said 18 million, most people were anchored at that number. Right. Which is a bit irrational because a rational econ, if you like, would have thought through a different process. They wouldn't have been nudged by that number. But we are human beings and this is how this just works. Right. And so the field of behavioral economics, I believe, came into play to try and understand research and explain how we are nudged or how those biases affect our decision making.

Yeah, it's been really popular lately. I've read a bit on it recently, and I know it's very much on the rise and trying to kind of fix some of the fallibility in traditional economic thought, but I think it's pretty fascinating that you had the foresight to bring it into the organization and train people in the field enough so that they could get the benefit of it.

I think one of the things that I would say it did, and this wasn't deliberate, but it seems to have worked, is I find in a lot of organizations, whilst people get very excited about data, there are a lot of people who get very nervous about it as well. Because you'll be looking at. As a data scientist, you look at data in a way that no one else has looked at before. You're able to find things that people haven't seen before and therefore you're challenging things. An example is, imagine going to a CRM manager whose job it is to send emails to an audience to try and get them to come back to your platform. Let's say, for example, and the traditional methods for that are using things like recency, frequency and monetary value as a simple equation. And then you come up to them and you say, well, I have an algorithm that can has a 90% improvement on your approach in bringing people back. And they'd be like, well, I've been 20 years in the game, so why would I listen to some mathematical formula that, you know, it becomes a very personal thing for some people. So you have that human challenge of them being threatened by data as well as being excited by data.

Yeah. So that actually leads right into what I was just thinking, which was what kind of strategy can you put into place to express to people their kind of built in irrational thought while at the same time not negating their background and their expertise and, and giving them some light at the end of the tunnel that this is actually going to improve what you're doing, you're going to be able to make more sales or whatever it is versus just kind of telling them that they're irrational, which maybe they are, but probably no one likes to hear.

That's a high wire act right there.

I think a lot of the examples we were giving them was the interplay between gut driven or a gut informed decision, experience informed decision and a data informed decision. And what we did is not only did we show the frailties of human gut making decisions, but we also showed the frailties of data because data itself is also limited. It's limited by the data that's captured. You can add 10 more data sources to an algorithm and suddenly it'll give you a very different answer. You can just add more data of the same labels that you had previously and you will also get a different answer. So data as well has its problems. So what we were trying to do was look for the balance, essentially the balance that is not fully leaning on either side. So it's not fully data driven and it's not fully gut based driven, but it's really a combination of the two. It's how we understand and are aware of our, let's say, the things that don't work very well in human decision making and where data can supplement and improve that, but at the same time not neglecting their gut and experience. Because we as human beings, interestingly enough are one of the best data machines out there. We take data from a range of sources, not just our five senses. I think they're, they talk, there's people who talk about a whole range of sensors I think in excess of, in excess of 10, I think I've read where, you know, take for example how you can close your eyes and still point at your nose. That's a sense that it's not touch, it's not taste, it's not any of those but it's some sort of directional sense that we have automatically. So we were really good data machines, so we shouldn't ignore our gut at all. And that was a big message that we were saying that this. And there's something in that because we probably captured more data than the machines have for now. So it was getting that balance. Right?

Yeah. So would you say that in that light? A good guiding principle is to frame things for people in the sense of augmenting their valuable capabilities rather than, you know, replacing everything that they've done. Right. But kind of utilizing their amazing capabilities, their skill, their background, but augmenting them to, you know, make them more effective or bring new things to light or whatever that might be. Is that an okay way to frame it?

Yeah. I wish we had your language to begin with. It might have made things, the journey smoother. Yeah, augmenting, I think, is the way you should say it because there you're. You're giving credit to, to their own decision making rather than dismissing it. And all you're saying is you, you bring something to the table, but what we want to do is enhance it.

So how do you, when you're considering that and you're kind of educating them and getting them into the right mode of thinking about, about this and, and bringing those practices into the organization's culture and accounting for the fact that you have behavioral econ and then you have more of the traditional data strategy. How do you build a strategy out of all of these desperate parts into a coherent message that everybody can understand and follow with.

Yeah, that's a fantastic question. And you'll see there that one of the things about data is that a good friend of mine describes it as a team sport. The data team alone cannot do that. The strategy needs buy in from the entire organization. It's one of the reasons why I think I always push for the lead of the data team to be part of the exec team because they need to span across the entire organization. But in order to develop the strategy, there's several things you need. Firstly, you need to make sure you understand the organizational strategy. And by that I mean you need to know the, the objectives of the organization, the boundaries of scope and the approach. And usually those sort of organizational strategies are hidden in reams and reams of documentation. And you've got to just try and simplify it because data also needs an objective to work towards. So when building the strategy, the first thing that we did was make sure that we could disseminate the organizational strategy into a sentence that Everybody could understand. And also that, you know, could be easy, could. We could work very easily with data. So it was a little more discreet in its numbers, it had very clear numeric objectives, a timeline for it, which it was working with a bounded scope. So it wasn't just any free idea. And also a clear advantage that we were using. So for example, at just giving. The advantage that we had is that we had millions of causes on the site, you know, so. And nobody else had that. So we had to work with that piece of content, you know, rather than just coming up with something arbitrary at the time. So getting the business strategy right was one of the most critical things we needed to do in order to get the data strategy. It was then from there that we went on to start looking at the possible use cases. And those use cases were really disseminating those decisions, as I said, to try and understand what are the decisions we're making operationally. For example, sending an email about a new campaign, let's say there was an earthquake. And like, for example, we had the earthquake in Haiti a while ago, you know, so who exactly are we going to send that email to? Because every time you send an email, as with any decisions there are trade offs. So something happens. You send the email, there are people who will unsubscribe. That means there are less people available for us to email for the next cause. So we need it to be personal. So we needed. That was a decision that we had to make. Who do we send that email to? And that's where we could apply AI. So that helps with the use cases and then also looking at the decisions that are being made externally by our audience. So when someone comes onto the site, are they deciding how much they want to give? Are they deciding who they want to give to? Are they making a decision on whether they want to just read or absorb content? So trying to understand those and support those decisions. And the last thing actually, so I said the strategy, the use cases. So the last thing in the data strategy was understanding where you are as an organization. So looking at it on almost two spectrums, the first spectrum is what capabilities do we have to develop any of these data solutions? Are we at the stage where we can only say what's happened and why it's happened? Or can we build algorithms that can predict what's happening and even prescribe? And the second spectrum we were looking at was how well do we know our decisions? And once we get an indication of where we are, you almost have a game plan, a roadmap of how you're going to get to the desired destination. There was a lot there. So let me know if I need to go through.

No, appreciate that. That gives a ton of great, great context. Okay, Mike, So I definitely have learned a ton about implementing data driven strategies and a strategy towards decision making in a company culture. But after all, this is practical AI. So if we get to that point of kind of scoping out certain decisions that we want to tackle within a company in terms of data driven decisions, how then do we make the leap to considering AI and machine learning efforts? Was that something that you considered right off the bat as you were implementing this strategy or did that come somewhere down the line?

Yeah, you know what, it's something we had in our back pocket the whole time. We knew we did have to take the organization on a journey. But there were some of these that were. I've mentioned this a few times. That email example was one that was so obvious to us very early on. We needed a machine driven way to very quickly identify who was interested in what cause, because no matter how intelligent we thought we were as humans, we just couldn't figure that out. Whereas there were a lot of clues hidden within the data. But we needed to go through that journey of getting the strategy understood and the use cases of which this was one of those. The other thing that we needed to do quite practically, and it depends on where the organization is, but you needed to get an indication of what the payback was, right? So as a result of doing this, what's the cost versus the corresponding payback if it works? And so there's some calculations that we had to do there, which makes the, the case for investment much easier. And once you've got to that stage, this is where data becomes, continues to be a bit of a puzzle you're not done yet. So you can have your game plan, you can have the areas for which you, you know exactly where you're going to apply some algorithms. You can get a team deployed on starting to think about, you know, are we going to use a collaborative filter or a genetic algorithm and get really excited about that. But there is nothing worse than spending all that time building, you know, your really sophisticated algorithm and it sits on the shelf and it never gets used. Ultimately this has to be deployed either to some level, on some level of production or given to a team for them to actually use it. So your biggest challenge after that is getting the culture of the organization, getting them into value and to have new behaviors for where these algorithms actually get used.

You're Speaking our language here at Practical AI. So you're helping us make it practical in terms of that. Is that something that you just, you know, in building up this strategy, is that something you instilled in people from the start of a project that like the real value comes, you know, once things are operationalized and deployed, not, you know, when things are conceptualized or when a model is trained in a jupyter notebook or something? Is that something that you instilled in early in that mindset or is that something you learned? I mean, I think where I learned that is by lots of painful scenarios where it wasn't the case.

We had a lot of. I don't know how to get away from the painful scenarios because culture change, you know, behavior change is very difficult to sort of instill. Now imagine being a data type individual for which we are where, you know, our interests aren't in how do we change people's behavior? You know, personally, I mean, they're psychologists that have that as a profession. So, you know, for us to try and jump into that, it's always going to be difficult.

I'm definitely not a psychologist. Maybe Chris is.

No, my wife can verify.

I'm not. So I think. But this is why there's lots of other pieces that need to come into play. So for example, if you have the buy in of the exec team, that helps because then working with each of the individual teams, you can begin to make sure that they get into the. It changed the reward of value base for using some of these. What I found, I found a really simple equation to help sort of summarize behavior change from the cultural side of things. It was by a champ called BJ Fogg. He does a lot of product behavioral design stuff and he has a formula called which is B equals mat and that's B is for behavior is equal to motivation, ability and triggers. And it's the multiplication, the product of each of those three things, which means if any of those don't exist, you don't get a change in behavior. So by motivation, you've got to get the organization in a sort of a behavioral approach where they are motivated to use the algorithm. So there's something. They've got skin in the game. There's something interesting there for them as well. And you can't do that without working collaboratively with the teams. And that's a key thing that the leadership team has a huge amount of involvement in. You cannot do that alone as a data team. The motivational structures for doing that, if they don't exist. And I've seen this, and I have experience of this where you can build something really amazing. We'll show an uplift. All the maths adds up. It's definitely something worth doing and you've built it, but it just doesn't get used because there is no motivation to do so, you know, and if individuals are rewarded by doing something else, they would really struggle to sort of go off piste. So motivation is a big one. Ability that's massive. One of the things we forget as data individuals is we don't speak English to the rest of the organization, so we almost limit their ability or create this. This environment where we just look like really brainy eggheads and we're not so accessible. So a good example is how you can build that algorithm, but it's not designed in a way that your team can use. Your CRM team or your customer service team have to be able to have the ability to use it and understand it. So we as data individuals have a lot of work to make what we do more accessible and the output of what we do to be way more accessible. And that's there in the ability. If it's difficult and there's so many brain cycles required to even process, you know, the output ability goes down and then that breaks that equation straight away. And the last one is triggers. The other thing that I've seen and also been privy to is we can build some amazing things and never shout about it. And I remember being a frustrated analyst many, many years ago, sitting there thinking, how come all the marketing teams get all the accolades and we've got all this amazing stuff here. And then you find out, you know, your CEO would come and say, well, you never told me about it, right? So that's what the triggers is. Just we need to work on our communication and in fact just giving. We took the whole data team and taught them how to communicate, how to share some of the stuff that they're doing in English in a language that the rest of the organization can understand. And we began to slowly see some real changes in behavior when we took this equation and intentionally addressed every single one of those aspects.

You mean the rest of the world doesn't communicate in Python and data frames?

You mean there's another way?

There is.

I apparently need to learn a few more things.

I mean, the fact that you guys are running a podcast is pretty amazing. I've got to tell you. I've got a great story about an analyst who, when we took the team off site and one of the Things we did first was to do some sort of one of those psychological evaluation tests to see, you know, whether you're a blue type individual, red type individual, or some of those, you know, Myers Briggs type things. And the whole team came out exactly the same. We were all very blue introverted type individuals. And you could see immediately where the problem was because we had no one who is like yellow or green to communicate with the rest of the organization. And a classic example was we had one analyst who used to start a sentence when he was talking to the business by. With the following words, let me explain to you how stupid you are. And then he'd carry on his sentence.

Oh, you know, that's not effective.

No, Daniel, that's not effective.

I am learning so much.

Oh, okay. So I have been learning so much as well, especially as we've dived into behavioral economics because we. I don't think we've ever touched on that in any of these episodes that we've had to date. And I'm pretty fascinated by how you've. How you've taken strategy and added that in and kind of all the various facets that you've been talking about. What I'm wondering is, could you tell us what you think makes a good AI use case to dive into? And if you have a specific example that you're able to share and kind of talk about how you made it real at a nuts and bolts level, I would love to hear that kind of like, you know, pedal to the metal kind of kind of thought process.

Yeah, I think I'm now reaching my head. I actually had an approach that helped you. Okay. So I think what we were thinking about was when there's lots of decisions that take place within an organization, right. You have some operational, you have some strategic. A good AI use cases is a decision that is typically quite. Is repeatable. So it happens more than once. And whenever it happens, it's relatively the same questions asked. So I keep deferring to that email example was the one that's on top of my head. Well, let's take Amazon, for example. Whenever somebody puts something in their shopping basket, there is always the decision of what else could we. Could we serve them in order to increase the basket size? Right. So you get that recommendation at the bottom. That decision is a repeatable decision. It's one we're making every. And it's a repeatable decision in a specific point in time. And that point in time is very clear. So every time someone puts something in their basket, you know, you have the Opportunity to upsell them so that you can increase their basket size. That decision happens all the time. And so the questions you'd ask is, can the organization identify when that decision will be made? Yes. Can they decide or have an indication of what information is considered every time that decision takes place and it's the same information and are the possible actions that they could take consistent? So it's relatively the same ones. And then next you say, can the outcome be measured? One of the most valuable things about working in AI and machines is, is how measurable those the outcomes of the algorithm are. Because ultimately it's an investment. Data scientists are not cheap cloud computing costs, although most say they are. They should be cheaper. They, I always feel to some extent they can be more expensive than on prem. But then also your costs can go through the roof with the amount of compute and the amount of data that's available. So looking at some repeatable decisions that are relatively complex, but they are measurable, that's the sort of lens that we would look for a good AR use case.

Yeah, you already mentioned kind of data scientists are expensive. One big piece of this that's really tough is actually building up your team for AI and the operations around that and the building of the models, but the operationalizing of them and the monitoring of them. In terms of your experience at just giving or maybe other places, have you taken a strategy of kind of building up software engineers that are existing within a company into kind of AI engineers or machine learning engineers, or kind of just brought in fresh AI and data science people, or has it been a mix of both and do you see advantages or disadvantages to one or the other?

All right, so I think it's a mix of both. I'll answer this by going through the roles that I think were really critical for our team and we were building the team and they were essentially four key roles that we were looking for. So the first was your traditional business analyst. So this is the individual that is perhaps more communicative. They've come from a consulting background. They're almost the front facing ones. They're the ones you take out of the basement. Right. And allow them to speak to users. These are quite important because, excuse me, data always needs a face and this is a representative representation of that. But we want them to be able to speak some of the languages of the rest of the team. The rest of the business go native if they need to, but really understand the decisions and the key needs, then we've got the engineers. So one of the things we Stumbled across, which was very interesting, is when wrangling with large amounts of data and making it available for data scientists, what we found was, you know that rule, that Pareto rule, that 80% of your time is spent on getting the data ready and only 20% on building the algorithm. We thought we would try and turn that on its head and reduce the time that a data scientist spent on data preparation. So we built a team of data engineers whose sole purpose was to make data go really smoothly from the source all the way to being available for whatever algorithm was going to be built so that the data scientist time was not spent on getting the data correct and majority of the time was spent on just making sure it was in the usable format for whatever approach they were going to use. Whether, let's say they needed to do, you know, logistic regression, for example, the data engineers would make sure that that data was set up so that you literally just had to run the logic command in whether it is R or whatever it is that you were using and spend more time on the results and perhaps consider a different approach to, you know, whether you wanted to move it through to a decision tree, for example, or a random forest, but to try and take away the expense that was being paid on working on the data side of things. So we had data engineers who could do that.

And I think that sounds like a good way to both make things efficient and to keep data scientists happy. Probably.

Exactly, yeah. And of course there's the mix and match because the data scientist ultimately is the, is the hero role that everybody wants to be. So we had to just make sure they could all they had their strengths. So the data engineers typically came from strong ETL backgrounds, but they were coders to some extent as well, because a lot of the data that needs to be moved around, sometimes it's easier just to write some code to rather than build the traditional ETL funnels that we used to have, so spark jobs, for example. So our data engineers had the capability of doing those. And then finally we found that the data scientists couldn't build production ready algorithms so you could run it in R, get the results. You still needed to build the model to work in real time for when a user came, if it was one of those, or at least to have the calculations available and the results available so that the automatic decision could take place. A quick example would be when a user comes to the site, which charities should we show them and in which order is a calculation that needed to take place overnight. But then we found that it was Changing every hour as soon as there were different interactions that were taking place. So the data scientists couldn't build the solution. For a robust solution, at least to work in, in real time, we needed engineers to do that. So we needed engineers who could understand the language that the data scientists were using, but then who were experts at building production ready systems. So you can see all of these roles. They need to have the same fundamental skills, but then they have their areas of expertise. So the engineers we were training into machine learning so that they could understand what was taking place, but really what they were good at was engineering. The data engineers, really what they were good at was working with large sets of data, but they still needed to have machine learning training and so forth. The data scientists, we reduced their work on the data prep and made them focus more on machine learning. Does that make sense?

Yeah, that makes a lot of sense. I know that can be a hard road to navigate, but I think you've expressed it well in the sense that you want to build up people's skills, but also build up people's skills into what they're they're interested in and what they, they're they do well in.

Right.

And as, as already established on another episode, I, I actually really like data munging and cleaning. So maybe I would actually fit more into the, into the data engineering part. Part.

There's a sickness that Daniel has in this.

Yeah, yeah, I. It is what it is, I won't deny it.

That's a good place you've come to though, where you've looked inward and you can see what you enjoy most about the whole data science process, if you like. Because some people needed some convincing, sadly, because the most famous of those three, four roles is the data scientists, which a lot of the team were aspiring to. So for some we just called them a data scientist, but we knew they were data engineers. Right, gotcha.

So I guess having gone through all this process over eight years, I guess looking back at challenges that you've had, are there any standout lessons learned or things that if you could do a do over that you might do differently? I thought that might be a good way of winding the conversation up.

Yeah, absolutely. I think I really wish I educated the organization as a whole on what data was and what it can do this. And I wish I'd started at that point because I found those explaining that, you know, to different individuals at different points in time. Whereas if we had done a, a bigger exercise on, you know, almost like a transformational exercise and just saying, look, the whole Education organization needs to be educated on this. I think we would have had less problems. Not saying we would have had. We won't have had any, but we would have had less problems. This became even more clear to me when I remember speaking to someone at Facebook and they were telling me that the whole organization goes to a data academy for two weeks. And a small part of that is understanding how to work with SQL, but a larger part of that is understanding how to ask the right question. And this really just hit home to me because if we had done more in educating the organization, we would remove a lot of redundant work as well, where people were asking things because they were just interested or they weren't really asking the right question. And sometimes data teams can be quite literal. So if they get a request from the business, they would do exactly what's been requested. But if you dug down, the real question was something else, you know, and that's the fault on both sides. So I loved hearing that there are other businesses that are creating what they call data universities where they put the whole organization through just to get them up to speed, primarily on, what is this thing called data. Let's demystify some of the terms that have come around and confused a lot of people. And why is it, you know, something that we can make more democratic and you don't have to be technical to understand what data is and how it works because we use it every day. We use information to make decisions all the time. And just demystifying that whole process for the organization. I wish we had done more of that. I think we would have got a lot further. I think we would. We developed some amazing stuff, but I think the road would have been much smoother.

Awesome. Well, I know that I have learned a ton today. I really enjoyed our conversation. Conversation. Thank you so much for joining us, Mike. And we'll put in the show notes a link to Mike's information online. But also he's got this book coming out. You've heard a lot of great principles today, but I think that's going to be the, you know, a really great resource for people that are trying to create this sort of culture in a company. And so we'll put the link in our show notes as soon as that's available. I know it's still in the works, but we'll put. Put it there once it is available. And just. Thank you. Thank you so much, Mike. I really appreciate you walking us through this process.

Thank you very much.

Thank you very much.

Thank you as well, very much. Thank you so much for the invite. It's great to be able to talk about this stuff and me personally, I am really motivated by organizations getting this to work. There's a lot of hype around it, but equally there are a lot of organizations that are just struggling to, to get the value that they can, the game changing value that data promises. So we can all sort of evangelize this a bit more then I think we can really see some, some, some big things happen, you know, not just for commercial gain, but even for us as, as people in an intensely information world that we can, you know, data help us get smarter, live healthier lives and so forth. So I'm very excited about what could happen.

Awesome. Yeah, that's super, super inspiring. Making, making AI practical since 2018 here on here on practical AI.

Yeah.

Thanks, Mike. And really looking forward to the book. Hope to, hope to talk again soon.

Thank you. Thank you very much.

Thanks.

Bye Bye.

Bye.

All right, thank you for tuning into this episode of Practically I. If you enjoyed the show, do us a favor. Go on itunes, give us a rating. Go in your podcast app and favorite it. If you are on Twitter or a social network, share a link with a friend. Whatever you gotta do, share the show with a friend if you enjoyed it. And bandwidth for Changelog is provided by fastly. Learn more@fastly.com and we catch our errors before our users do here at Changelog because of rollbar. Check them out@rollbar.com changelog and we're hosted on Linode cloud servers. Head to linode.com changelog check them out. Support this show this episode is hosted by Daniel Whitenack and Chris Benson. Editing is done by Tim Smith. The music is by Breakmaster Cylinder. And you can find more shows just like this@changelog.com when you go there, pop in your email address. Get our weekly email keeping you up to date with the news and podcasts for developers in your inbox every single week. Thanks for tuning in. We'll see you next week.