#62 From Predictions to Decisions

Hi, I'm Maggie, VP of Curriculum at datacamp, and I know firsthand how overwhelming it can be to keep up with the rapid changes in data and AI. But staying ahead doesn't have to be hard. That's why our curriculum team has curated the best learning content to help you and your team stay ahead, covering everything from data and AI literacy for busy executives to cutting edge generative AI for developers, and everything in between. Whether you're upskilling yourself or your team, we make learning practical, engaging and flexible. It just works. Power your data and AI transformation journey on datacamp.com.

Hello, this is Adele Neme from datacamp and welcome to Data Framed, a podcast covering all things data and its impact on organizations across the world. One of the most exciting things we've seen over the past few years is the maturation and operationalization of data science and machine learning and organizations across the world. More and more so, machine learning is jumping from notebooks and into business processes. However, as machine learning starts driving more and more predictions around complex business processes, finding a scalable way to turn predictions into decisions that maximize value for organizations will be the next frontier to go from operationalization to full blown usefulness. This is why I'm excited to have Dan Becker on today's episode. Dan became a professional Data scientist in 2012 when he finished second in a Kaggle competition with a 3 million dollar grand prize. Since then he's consulted for six companies in the Fortune 100, served as product director in the early days of DataRobot, and worked at Google. Dan has also made open source contributions to Keras and tensorflow and is the founder of Kaggle Learn. Most recently he founded Decision AI which which aims to solve the challenge of bridging decision sciences with machine learning. In this episode, Dan and I talk about his background, his journey, reaching the top of a Kaggle competition, machine learning and business as opposed to Kaggle concept drift, the friction between developing something that is useful versus shiny, the importance of empathy when aligning business objectives with data science, and finally, the growing intersection of decision sciences and machine learning and how Decision AI can help data teams align machine learning with business outcomes. If you want to check out previous episodes of the podcast and show notes, make sure to go to www.datacamp.com community podcast. Dan, I'm so glad to have you on the show. I am really excited to speak to you about your journey into data science and machine learning and your new venture at Decision AI. But before we dive in, can you tell me about how you got data Science and the Various roles you've held in your career so far?

Yeah, you know, in some ways I've gotten into data science twice. So when I first graduated from college, I got a job. It was with a startup. This was, this was a long time ago. This was when a lot of people were selling things on ebay and we had a service where we use neural networks, which were not popular then and not very good then, to help retailers figure out when to post things to ebay and how to sort of optimize their listings, both because of what the state of the art was then and then also how little we know or how far off from the state of the art our little startup was. That was a total and complete failure. And after six or nine months of doing that, I decided, all right, machine learning is never going to catch on. The words sound cool or the promise sounds cool, but it's just not going to work. And I did conventional statistics for a while and got a PhD in economics and moved away from machine learning, though at the time I was an economist for the US government. And then about 10 years ago, someone told me about a Kaggle competition and I figured, yeah, machine learning doesn't work, but I know a lot about other types of other ways to work with data. And so I entered this Kaggle competition and I was not dead last, but I was almost dead last. And I was so shocked to be so far behind the curve of what other people were doing that every night, all the time on the weekends, I would focus on improving, just getting better at building models. And I got up into, eventually into second place out of over 1300 teams in that competition. From that, a consulting company hired me to both help them with some of their projects, but even more than that, train their consultants in the techniques that I've been using in this competition. I did that for several years. Consulting for any of your listeners. I think that consulting is such a great way to get exposed to a lot of different data science use cases and that has served me well for a long time. After that I worked for Datarobot as an early employee in charge of product for DataRobot for three years or so. That was also really enjoyable. DataRobot is an awesome company, awesome product. And then I left datarobot to work at Kaggle and Google. I was there for three or four years and then most recently started Decision AI.

That is so awesome. It's always really nice to learn from someone who's been involved in data science and machine learning for this long. And you have this panoramic view over the field. I want to start off with the second time actually, you got into data science as someone who spent tons of time working on KAGGLE competitions. Can you walk me through the process of growth and learning that led you to have these awesome results?

Yeah, you know, it's very iterative. I think of in some ways, most machine learning algorithms, they start with something that's really bad. So a neural network, it starts with a random set of weights and it makes predictions and it's really bad, but then you've got a mechanism to improve. And so it learns from the data and as it runs before you deploy it, but it improves the weight so that it gets more and more accurate. And you could say the same thing for another algorithm is gradient boosting and it starts out with a base prediction and it's very inaccurate. It's very bad. But there is an algorithm for improvement. And I think the same is true for how do you improve as a data scientist? I started at these KAGGLE competitions and my first submission was very, very bad. One of the great things about Kaggle is that, or at least the competitions is that you can make a submission and then you have a feedback mechanism every day. You can upload a new model or a new set of predictions, say, all right, did this improve? Did it not improve? What do I learn from the feedback of trying it, seeing whether it improved or not? And I just started at something that was. I was really quite bad at it, but every day tried to improve a little bit and every day moved up in one particular competition a little bit and learned a lot along the way. The other thing that I did, which I think most people could really do, is learning from the community. So that is, if you're in Kaggle, be on the message boards. But that could be go to meetups and don't just walk, like, watch, try and talk to people. And I find conversations about data science, I find them so fun. It's like way more fun than working alone. And I've just learned a ton from it. And a lot of it is the stuff that's hard to read or hard to find in a textbook or even hard to find in the great courses like what you guys do. There's something about having a back and forth, give and take where you talk to someone that I think is so valuable.

Yeah, I completely agree that even from my experience as a data educator working with a lot of data scientists, that conversation that I've had when developing courses, let's say, and trying to understand how data science is implemented in the field. What can I learn from them? I found that really contributed to my growth and my data skills as well. You've mentioned that working on and scoring high in Kaggle competitions led you to other job opportunities within data science. And this was a dynamic that really held previously. Do you think that this is a general dynamic that still holds today? What advice do you have for aspiring data scientists looking to break into the field?

Yeah, I think that many people will probably be disappointed by this advice. Competing in Kaggle competitions is not a silver bullet. If you are in the top 1% or it's probably in many cases even the top fraction of 1%, it's a great credential. It will open a ton of doors for you. But tautologically, 99% of people will not be in the top 1%. And so for most people, that thing that I did, which is now even harder to do than ever because Kaggle's gotten more competitive, that I think is not the ideal strategy to get jobs in data science. I think of again, starting with a lot of conversations. Find a community of other learners. Kaggle happens to be a nice one, but there are, there are others. Find a community of data scientists, try and learn from them. And then the most important thing is to build a portfolio of interesting work. So I've hired probably about 10 people directly who reported to me. And then I've probably been involved in like interviewing maybe about 100 other people or the hirings of 100 other people, which is hundreds of interviews over the course of time. And the thing that is really most interesting to me is if someone has a project and it doesn't need to be related to my work, it could be that they love rugby or they love movies or they love whatever and they've done a project which is just interesting and they're willing to talk about that. I find that that pulls me in, that helps someone stand out in a pile of resumes. And then, yeah, there's such a crowd of people getting into the field and I think if you can build a portfolio, make it interesting, put it on GitHub. If the first thing I see is going to be a readme, if you make that readme easy to read and has nice graphics in it, that's going to pull someone in and really stand out. And then if you say my goal is not just to get a first job, but I want a great career, I think that a lot of people spend too much time doing textbook style learning upfront and focus too little on how do I get my first job. Because once you get your first job, yeah, you can learn outside of work, but on top of that going to be spending eight hours a day or however many hours a day working on data science, getting feedback from colleagues who are more experienced. And I just think that, that you should think of how do I get great over the long term? Well, it's getting a job. So I'm getting hands on experience for a lot of my time and getting paid for it early on. And the key to do that, as I said, is to build a great portfolio of things that you find interesting.

Oftentimes I receive questions on how to break into data science as well. And I think like one of the most important things, one of the lowest hanging fruits is just getting into it, doing a project, hosting it on Medium on GitHub, writing a blog post about it, and then as you said, really immersing yourself in the work actually through learning that way. Now I want to talk about as well, given your subsequent experiences after Kaggle. So you've worked on a lot of Kaggle competitions and then you've transitioned to the consulting world afterwards as well. So DataRobot and Google, can you provide an overview of how different working on and applying machine learning is in a business context rather than, or how different it is than a Kaggle competition?

Yeah, it's incredibly different. The number one thing that matters in a Kaggle composition is to make your model incredibly accurate. The number one most important thing that matters in most business settings is to scope the problem so that you're building a model that is addressing the right problem. That it is something where when we get the output of that model, we know how to use that to make business decisions. And so it's, it's scoping and deciding how the model will be used is, is the single most important thing. And then yeah, you can make your model more or less accurate within that, but it just doesn't matter as much. And, and that's part of why I was so excited about DataRobot as a, as a product is instead of fiddling with hyperparameters, which can make your model more accurate or fiddling with how do we do feature engineering? An auto ML tool says, all right, you're going to press a button. However you've set up the data, we're going to make your model as accurate as possible. And that allows you to focus on, okay, where is this model going to fit into our decision making process or how do we prioritize what is important for us to work on versus less important to work on. The thing that I always found, and this is true in CAD competitions, it was true for us at Datarobot, is that data scientists are pulled into things that are cool or useful. And a lot of times that's the cool new algorithm. And I think my job at Datarobot is being how do we focus on things that are useful rather than cool? And you see the same thing where KAGGLE competitions, they allow you to do things that are, I think the competitions are super fun and so they're cool. You see some really cutting edge techniques and yet it's really important to stay focused on what's useful. We've got some big picture business goals. We got to get all the, we got to get all the vaccines to people in the right places at the right time as efficiently as possible. How do we do that? And making your model go from a log loss of 0.3 to 0.29 is not the problem. It is how do we even decide what to model in order to make machine learning useful for that problem? I guess the last thing I'd say that would be remiss in 2021. Not to mention is that one of the big differences in a business context for something like Kaggle is that you really need to worry a lot in real world settings about train test drift. That is the world changing up from under you, talking about all of the.

Different elements, breaking them down as well when you've discussed them. Scoping an AI project and making sure that it's aligned to the business objective and how to operationalize it within a business process is also super, super important. And you've mentioned here concept drift. Can you think about some of the remedies for concept drift and for those who may not be familiar with it, can you also define it?

Yeah. So concept drift is just this phenomenon of the so called concepts that are captured in your data. Historical data, because all data is historical, are not going to carry forward into the future. And so the, the relationships between things that you have historically will change over time. So the, the number one outcome of that is that you can have a model which was very accurate in your training data, maybe even in your validation data or your cross validation, and then you deploy it in the real world and it's much less accurate than you expected. Seeing that to some extent is probably the norm rather than the exception. But there are some places if you are modeling a physical process, there's nothing human about it, then in those cases you see less of it. And for processes that there's a lot of human interaction. How many, how much aluminum will someone need? Well, that depends on economic changes. And so those things tend to have more concept drift or it has a lot of other names. Train test drift is another name for it. Let's see. So the things that you can do to address it, probably the simplest thing that you can do, although this is not perfect, but the simplest thing that you can do is when you validate your model, you say, is this, how accurate will this model be? Use out of time validation. So take training data from. If this were happening today, I'd say I'm going to take training data from 2018 to 2018 and 2019, and then I'm going to use validation data from 2020. And that way, if you were working on a process where this concept drift is going to occur, you're very likely to see it because you see this big drop in model accuracy when you go from my training data, what I used to build the model, to the validation data. Since the validation data is just from a different period, that doesn't exactly tell you how to solve it. And you'll see that this is pretty common with concept drift is there are a lot of things to detect it and fewer options to solve it. But that's probably the first line of defense. And that's something that's super easy to do. The second thing that is harder to do is better, but also not perfect. And you should do both of these is model monitoring. So model monitoring, there are starting to be more tools in the MLOPS space for this. And what you're doing is you say, every time a prediction comes in, I will store some data about that prediction that I made. And then on a regular basis I'll say, how does the data that I've predicted on since I deployed that model, how similar or different is that from the data that was used to build the models? Is that frequently? Well, let me come back to something called covariate shift. Ideally you'd like to say just how accurate is the model? Has the model become more accurate or less accurate after I deployed it than it was when I trained it? Typically that's pretty hard to do because when you make a prediction, you don't have the true value, like the actual value to make a comparison against. So if an example would be if we are making loans and we want to know, is someone going to repay the loan? All right, so we've built a model on historical data who repaid their loans. Now we make predictions, people are coming in to our API, we say this person, they're 5% likely to default. Someone else, they're 20% likely to default. We don't actually know did they default at that time. And so all we have are the features that we predicted on. That's something called covariate shift. If the population of people who you're making loans to now looks unlike the population you had in the past, I.e. covariate shift, meaning the features you're using for prediction, if those are changing, that is the thing that is the way of model monitoring or detecting concept drift that is most common today. Again, it's not perfect. You could have that covariate shift and actually your model is staying pretty accurate, or you could not have covariate shift, but yet your model is becoming less accurate. And so it's not perfect, but it's something that is easy to implement or easier to implement and is done today. Even beyond that, there are a lot of things you can do which are much less common to adjust your prediction. So let me give you a couple of examples. And these are going to seem so subjective that a lot of people push back on these when they first hear them. But I'll tell you why I think they're still important. So if you think that you have a hypothesis that sales in the net for your product in the next year are going to be 20% higher than they were historically in ways that are not driven by changes in the underlying features, you make a prediction and you should just multiply it by 1.2. You should. You should take that subjective belief you have about your domain and incorporate it into how you make predictions. You can use your conventional machine learning model, but again, just adjust the prediction after you get the output of the model. So let's say that seems so subjective, they're a little uncomfortable about it. But the flip side is if you take that prediction and you don't make that adjustment, you're implicitly assuming that something is not going to change or that the world is not changing. And if you truly believe that the world is changing and you have that domain knowledge, I think it's a mistake to not apply it, even if the way you apply it is very subjective and is imperfect. We could even do a step better than that. There are ways to make more surgical adjustments where you say, I think that for a certain population or the effect of a certain variable is going to be 50% larger than it was historically. And so I want to take the impact of that variable historically and I want to make some adjustment after I Get my prediction so that the impact of that variable is 50% larger or 50% less. There are ways to use some machine learning explainability techniques such as SHAP values to figure out what was the impact of a given variable. And now I'm going to do a post hoc or after prediction adjustment to it. These techniques are not very widely spread, but we see the world changing so quickly that I know a lot of people who are taking machine learning models out of production because they say the world's changing too quickly and we don't have a way to address this. And so I think it's going to be really important for us to say, how do we take our knowledge or our beliefs about how the world's changing that aren't captured as data yet and make these types of adjustments?

Yeah, I think what you're speaking about here is a very fascinating change that we're seeing in the world of machine learning where we're really seeing a much more emphasis on not just producing models, operationalizing models, but also integrating human knowledge within machine learning models, as well as explaining different machine learning models and finally monitoring models in production. And it's going to be very interesting to see how machine learning operations, data operations as a field evolves within the next few years. Now, circling back to Your experience at DataRobot, you've mentioned that there is a friction often that data scientists encounter when building between something that is cool and something that is useful. I think a lot of data teams as well often fall through that trap of building something cool. And it's kind of like a resume driven development agenda. Can you speak about that friction and how you experienced it and how you went about balancing between these two elements?

Yeah, you see this dimension everywhere. And I think the key, and it's going to come up in a lot of different places, is just empathy and understanding. Someone is going to use this model for something typically that's one of your colleagues and you need to see the results of your work through their point of view. And so if you say, hey, I used light gbm, they truly don't care and they instead have some decision that needs to get made and you need to figure out how do I plug into this so that I am creating work that is valuable to them and that's purely a matter of just seeing things through their point of view.

Do you think that data scientists can hone this skill over time, that skill of building empathy? And what do you think are some of the best practices that data teams can adopt to really integrate empathy into the process of building data science solutions.

Yeah, in some sense I think it is hard not to, to get this ability, this empathy over time. So if you and I, we work together all the time for a year or if we even went to, you know, went out to a bar post Covid every Thursday for a year over time, I'd say actually, you know, the things that, the times I've talked about basketball, Adele didn't seem that interested and the times I talked about something else actually seemed like you were really interested. And over time I would get a sense for what you care about and what things you don't care about and our conversations would flow to the things that are at that intersection of what we both care about. And so I think that if you talk to the stakeholders or the people who really rely on your data science models, it would be hard to talk to them on a regular basis and not get a sense for, for how they think about these problems and what's useful to them or interesting to them and what's not interesting or useful to them. I suppose the real question is, can you intentionally do it or can you accelerate the pace with which you become empathetic? And I think that, I think it's probably a matter of just asking for feedback and say, hey, here's what I did. How could that have been better? And people will tell you, actually, I didn't understand when you talked about X or the thing that was most exciting to me was why. And if you ask for feedback and accept feedback, you're inevitably going to get a better sense for how others see the world. And you're going to be able to tailor how you present to them to the things that, to how they see the world. And then once you tailor how you present to them, that in turn will drive the type of work that you do, or the type of algorithms you do, or where you focus your attention while you're programming so that you're focusing your effort on the things that are going to have the biggest impact on the people you work with.

100%. I completely agree. Further touching upon this friction, one thing that I've seen you speak about publicly is the different stages of evolution of data science and machine learning. And I've seen you speak about how machine learning historically was all about experimentation and now it's about deployment and that we need to go into usefulness. Can you provide your insight into your thinking here?

Yeah, I think the people who are getting into the field now will be, may be surprised that there was ever a period where companies would Just hire you to experiment and your model didn't need to get deployed, it didn't need to be useful, and that was okay. And yet when I got into the field, seven or eight years, when I was consulting seven or eight years ago, yeah, it was absolutely the case that people would hire us to build models so that they could be exposed to machine learning because they knew so little about it and they just wanted to sort of get used to some of the underlying ideas. That was not the case even probably three or four years ago. And so there was a time when they said, if we, if we have you work and this doesn't get put into any real world process, that is not okay, and we'll, we'll be upset about it. And so three or four years ago, they said, all right, it needs to get deployed. And there was a period of time when, if you got your models deployed, that was great. Getting them deployed actually is a skill both on the engineering side and on the interpersonal dynamic side. And I think it's only in the last year or so when companies have said, yeah, we've been deploying your work, and yet we're still dissatisfied because we find it's not making an impact on the things that our CFO cares about. And so, and if I, if I go back a year ago and you said, think of all the data scientists, you know, who have ever been laid off, I would have said, I, I've never heard of that. I've never seen that, like, data scientists don't get laid off. And I know a lot of people who have been laid off in the last year. So some of that was the economy, but I think even more of that was companies saying, well, we invested for years a lot of money in data scientists. And if we look at our profit and loss statement or any of the metrics that we care about as a business, we don't see the impact. And so now there is a demand more than ever for your work to show up in something that the rest of the business cares about and in the metrics not of accuracy, but in the metrics of profit or loss or customer retention. And so that's a change that I think the data science world is still adapting to. We're not quite there yet of having the mindset of we're going to make sure that everything we do is really useful.

That's very fascinating, especially when you think about the risks of not being able, as a data scientist or as the field of data science, really prove the usefulness of data science within the organization creates this sense of lack of trust from business stakeholders as well as a disinterest and disinvestment in data science. And I think this actually marks a very great segue to discuss your most recent venture, which is Decision AI. So can you provide an overview of Decision AI's mission?

Yeah. So like I said, you've got all these machine learning models are being widely used, the output of a machine learning model. So if you look at whatever TensorFlow or SciKit Learn or XGBoost, their API is always dot predict. And the key insight, which I think a lot of data scientists haven't accepted yet, or a lot of people using machine learning haven't accepted yet, is that predictions don't matter. The only thing that matters, you know, what affects the world is your decisions. And so you get a prediction. And now there's a step beyond that of how do we translate that prediction into an action or a decision? So that's something that is frequently called a decision rule. And we are helping companies make better decisions with the outputs of their machine learning models, or we're helping them come up with better decision rules. So let me give you an example. So you run the data science department for a chain of grocery stores. So one of the most important things that you can do is figure out how much of each of these items that we sell do we need to order from our wholesalers at each point in time, really, to make it more concrete. So in a given week, you're figuring out how many mangoes should we order. You build a predictive model, a machine learning model. It says you're going to sell 1000 mangoes between now and the next time that we can get our mango delivery. All right, so how many should you actually order? So this is the decision rule. So many people would say, all right, well, if we think we're going to sell a thousand, then we'll buy a thousand. But machine learning models aren't perfect. None of them have an R squared of one, or those that do have an R squared of one we know have some other problem. And so if you order 1000, then roughly half the time you're going to run out of mangoes. A bunch of customers will show up and go say, I want to buy some mangoes, and they won't find any. And then other times you'll have extra mangoes. And you and I, you know, we go to grocery stores and we see that it's pretty rare, that they're out of stock of, of produce. And so why is that well, they buy extra stock to make sure that they don't run out. And part of that is they think that running out is, is particularly bad. So what are the considerations that you'd want to, that you'd want to think through? You'd want to think through what's the shelf life of mangoes? Can I buy extras now and then I'll just sell them next week? Next week I'll buy some more mangoes, but I'll keep those in the back room and then those will fill in when, when I sell out. So you'll want to think about what's the shelf life of mangoes. You'll want to think about what's the cost of having mangoes. Rotation vs what is the cost of running out and disappointing customers. You want to think about what's the likelihood of wholesale prices going up next week or going down next week. Because it may be that if I think wholesale prices are going to go up, maybe I buy extra this week to avoid the future price increase. So there's all sorts of external dynamics that are beyond. Here's my prediction for how many mangoes we'll sell. And the thing that we are trying to do is allow a data scientist to encode mathematically all of this broader context so that they can rigorously optimize their decisions for big picture goals. Like we want to, when we make, when we make our mango purchase, we want to figure out what's going to make us money in the long run or keep us happy customers in the long run. Rather than being narrowly defined on, we're going to make a prediction for how many mangoes we're going to sell and then, you know, typically data scientists today don't have any rigorous way to build the, the, the decision rules or the, the broader structure around that. Doing this, I'll be honest, doing this is harder. It's going to lead to much, much better results. I think the benefit that we get from going from or we just sort of guess or have a rule of thumb for our decision rules, which is what most data scientists use today to we're rigorously optimizing that. I think that win is going to be as large as it was when we went from we're analyzing data but don't have machine learning models to when we had machine learning models. So I think the win is going to be huge. It is harder. But you know, I was an early adopter of, of deep learning and I was contributor to deep learning frameworks. Even before TensorFlow existed. There was one called Py learn 2 and the early adopters of deep learning, like, yeah, it was a little harder than the machine learning techniques we'd used before that it got easier because there were better tooling. But the payoff for early adopters was huge. And I think there is now the opportunity is quickly accelerating into a need for data scientists to be really rigorous about how, how they use their machine learning models and what these decision rules are.

Yeah, and I think that like this solution that you're speaking of really sits at the intersection of the problems that we previously discussed of how to go from experimentation to production, but from production to usefulness as well. And I think the idea of combining human knowledge and human assumptions about the business within your machine learning models and combining these extraneous concepts is super useful. So it's really about combining human knowledge, simulation and machine learning to further align results, machine learning results to business decisions. Can you walk us through an example of a project you've worked on?

Sure. So let me walk through a different example. I talked about mangoes, but I want you to see sort of the breadth of how these can be used. So there is a hotel chain, number one most important problem for hotel chain is how they set prices. And hotels can update their prices on a daily basis. So for a given location they have machine learning model. This is a model they have today, which is if we set our price at, let's say we're looking 90 days out for the hotel nights that are 90 days out, if we set our price at US$100, we'll sell five bookings, at some other price we'll sell six. At some other price we'll sell seven. Now which of those is better? But since the spaces that we don't sell tonight can be sold tomorrow and the spaces that aren't sold today or tomorrow can be sold the next day. There is a dynamic optimization problem here. And the way that you would approach this problem in decision AI is one, you would build your machine learning model for how many hotel nights can we sell at any given price? You'll bring that into our software. You can build it in XGBoost or TensorFlow or SciKit Learn or any of these tools. And then you will write out some domain knowledge equations which are typically pretty straightforward that just describe the structure for other structure for how this problem works. So for instance, you would say our revenue on any given night is our price or revenue that we accumulate from sales on any given day is our price that we set on that day times the number of hotel nights that we sell on that day. Okay, so that's a pretty straightforward equation. You'd have another equation or formula that says the number of rooms that we have left over to sell is whatever we had available to sell the previous day minus what we actually sold the previous day. You'll have a couple of other equations for how these things evolve over time, but they're all equations that are formulas that if someone works in this field, they'll know very well. And we can now start on today and say for any given price, how many rooms do we sell at that price? How many, what's our profit or our revenue from that today? We might calculate some other things today and then we are going to use that to calculate what is the starting spot that we're in tomorrow, how many rooms do we have left over? Maybe what is our competitor's price tomorrow, which is going to depend on what we did today because of competitive dynamics. And you may have a machine learning model that predicts competitors price tomorrow as a function of what we did today. So you'll have at that point have the state of the world tomorrow and you'll take your decision rule and say, all right, well what happens tomorrow if we use this, this way of setting prices and we're just going to propagate that forward until the, until either we run out of rooms or it's the, the night that we're selling rooms for. And then we can add together what was revenue from selling on day one, day two, day three, and that's going to give you total revenue. So now we have this simulation environment. People who have done dynamic optimizations or operations research before will see parts of this that feel familiar, that we're now combining that with machine learning. And we've got this simulation environment where we can try out different price setting functions or different price setting rules to see what happens over this whole period, long period of time in terms of not just the things that we typically predict, like how many rooms do we sell on any given night, but what is our cumulative revenue over this whole period of time, which is the sort of thing that people like I keep saying, what does your CFO care about? Things like revenue are things that the rest of the business cares about and we can now rigorously optimize for revenue over a long period of time.

Apart from the hotel industry and some of the use cases you've worked on, can you name some of the other use cases that you've worked on at Decision AI?

Yeah, the two that we've been closest to are financial fraud so there are. Machine learning is very, very widespread in identifying transactions that are likely to be fraudulent. But it's a classification model. So what is the output of that you get? You run a transaction through a conventional machine learning model. It says this transaction is 7% likely to be fraud. Okay, well what do you do with that? Do you reject it? Do you accept it? Frequently? There's some, some third path for investigating it. And which of those you do should depend on a lot of things. Do you think this is a high value customer who you're concerned about upsetting? Is this the beginning fraud on something that's likely to be a long trend of fraud and you want to sort of nip it in the bud? So how do we take that prediction of likelihood of being fraud and make sure that we're using it optimally? So we've worked in financial fraud, we've worked in inventory management. That's going back to that Mango example. All right, we think we're going to sell 1000 units of some product. What do we actually buy from our wholesaler? What do we ship from our warehouse or distribution center to our retailer? So inventory management is another one, I'd say. I'm especially excited about supply chain management applications. And there's a, you know, supply chain management is a quite broad field, but how do we make sure that in our manufacturing process we have all the ingredients we need or all the pieces we need? How are things shipped from one place to the next after they're manufactured? How do we get them to a distribution center or warehouse? All of that is stuff that in addition to inventory management is stuff I'm very excited about.

Yeah, I completely agree with you about the supply chain management because any marginal improvement to a supply chain network that is highly complex, highly optimized will give so many additional value to the business. And given the importance of embedding business knowledge and decision rules into the machine learning models to arrive at usefulness, how important is it, do you think, for data scientists to truly understand the business context around the models that they are building?

Yeah, I mean, some of this goes back to. Businesses have lost interest in pure experimentation. And so now it is absolutely necessary for you to deliver results that are not log loss style results, but are because of the result of what I worked on last month, here's how our business is better off. To do a good job of that. You need to be interested and knowledgeable in the broader business context. And so I think just to stay employed, you need to. Yeah, you need to be interested and knowledgeable about how your business works. And then if we go, all right, that's what it takes to be employed. What does it take to be a future data science leader? The future data science leaders unquestionably will be those who can take the work of a group of data scientists and then speak to executives across the business. And how do you do that? It is really having that intersection of knowledge of the business, ability to speak well, and knowledge of data science. For a data scientist, if they say, hey, I want to be a director of data science or I want to be a chief data scientist, it is absolutely going to be crucial that they understand the rest of business well. But even if they say, hey, I just want to survive in this business, they're going to have to understand it reasonably well.

Even more crucial because data science is now permeating to a lot of different business functions as well. So what do you think are some of the common pitfalls data teams make when they communicate with executives?

Yeah, I think the main thing that the main mistake is that they focus too much on data science primitives rather than the business. So if I buy a piece of hardware, let's say I buy a new computer. If there was not someone who could speak well to me as a purchaser of computers, then you can imagine the person who was a hardware engineer. They might say, the really, the great thing about this computer is that I worked on the bus between the CPU and the RAM and we got the resistance from 6 micro ohms to 5 micro ohms. If they said that to me, I would say, I truly don't care. I don't actually. I don't even know what that means. And I made that up. Like, it could be that that pathway, which I think prob. Has a resistance to be measured in some sort of ohms. Like maybe the thing I said is gobbledygook, but I would certainly say I don't care. And I'd probably say I don't understand. A step better than that would be if they said we decreased the resistance between the, whatever the RAM and the cpu, like that, that little channel. And so it went from transferring 600 megabytes per second to 1.2 gigabytes per second. It's like, okay, that seems faster. Like it's starting to be something that I could imagine caring about. And if I were to do put in a ton of effort, maybe I could translate that into the impact it has on the work that I do. A step better than that is if they release benchmarks and say, for your Machine learning workflows, the sort of thing that would have taken you four hours for computation to finish is now going to take you 2.5 hours. All right, now I understand it. I'm ready to say, hey, this is worth buying or it's not ready, it's not worth buying. But it is really like they need to speak in terms of the things that I care about. And data scientists frequently fall into the pitfalls of, especially when they first start of that hardware engineer who's going to speak to the. To me, or by this analogy, the executive in terms of the micro ohms, and they say, hey, we ran light GBM with a learning rate of 0.05 and tomorrow I'm going to try it with 0.025. Okay, so that is truly no one cares. Even other data scientists don't care about that. Like, yes, you might need to run both experiments to figure out which is more accurate, but that is truly talking it in micro ohms. And for that matter, if you said, hey, I tried light GBM, but tomorrow I'm going to experiment with XGBoost or SciKit learns gradient boosting implementation like, we also junior data scientists may be curious which of those is better, but we really shouldn't care about that. And so the things that you should communicate is, all right, I got this, I got this accuracy. Here's the impact that it would have if this is the accuracy of the model that we deployed for our business. You know, even back to the envelope calculations, here is how many more customers we would disappoint or here's how much customers we'd lose because of this. Even back of the envelope calculations are useful there, but you can say, hey, this is the work I did today. I'm going to try a different algorithm and see if I can improve that tomorrow. I think I can or I think I can't. And a different algorithm is the right level of granularity because the rest of your business just doesn't care whether you're using random forest or deep learning or light gbm. And so I think to always come back to empathy, what are the things that they care about? And now how do I talk about that in their language? And their level of abstraction is, I think, the key. We absolutely aim to facilitate that by allowing you to build out that structure in decision AI around your models so that you can say we're not just here's. Instead of saying, here's the change in our accuracy for or the change in our R squared for predicting demand for Mangoes or hotel rooms or whatever else you'd like to be able to say. If we use this model with this decision rule, here's the impact that's going to have on our revenue or here's the impact that's going to have on the amount of storage space that we need for extra mangoes. And now you're starting to talk in the language of the business. And a lot of data types, especially junior data scientists, are so excited about telling people all they know about learning rates or all they know about gradient boosting that they forget what the right level of abstraction or what other people care about.

It is so exciting to see that the future of data tooling is also trying to solve that problem of how to frame and present data science projects and data science results. Now, Dan, before we wrap up, are you working on any projects outside of Decision AI?

Yeah, you know, I've always, I just enjoy fooling around with data. There was an, there was a while where I was really using decision AI and how we did some marketing decisions as an excuse to do this. But I was looking at the popularity of different data science tools and Python packages. But the thing that I'm most interested in right now is some projects related to climate and I'm trying to figure out the ideal way to connect them to decision AI. But I'm looking at some publicly available day by day weather data in different locations. It's in an S3 bucket. And yeah, looking at differential effects of climate change in different locations. And for people who aren't, who don't follow this field very closely, it's been very differential in different parts of the globe. Some places are getting wetter, others getting drier, some have warmed in the last two decades, other have, have cooled. So it's very heterogeneous. That's going to have lots of, not surprisingly, lots of effects on the world. It's something that I've just been looking at that data and quite curious about and building a dashboard to sort of interactively play with it, but always trying to think about how do I take the work that we're doing with decision AI and also connect it to climate, which I think is probably the big challenge for us of the next couple decades. So that's sort of my hobby project of the moment.

That's awesome, Dan, and it's awesome to see you as well working on some of the biggest problems of our time. And we'll make sure to include details on the show. Notes. Finally, Dan, do you have a final call to action before we wrap up.

Yeah, I suppose the thing that I've found talking to a lot of data scientists, and I'm talking to data scientists all the time. The thing that I would say for people who are just learning is find something that is interesting to you, like interesting in a real world sense, and do a project to learn about that from data. Do a project to work on that, even if it's not perfect. If you're further along in your data science journey, maybe you're working as a data scientist. The thing that I found is that most people who are using machine learning today or have pondered using machine learning have said if we were to build a model, here's how we would use it. I think probably about 80% of the people that I talk to after we have a conversation, they realize actually they can do a lot better than they thought because you slice out so much of the context in order to use conventional or supervised machine learning models. Here's I think the challenge to your listeners is if you've got a problem, you think you are using your machine learning models to make decisions. Well, drop me a line. Danicision AI and yeah, we'd love to hear about it. And I have a hunch that if we chat even for a few minutes or a few emails back and forth, you'll realize there are better ways to turn your predictions into decisions. We're also making decision AI. We've got free accounts, and so we encourage people to come to our website and fiddle around with the product and see if there's a way that they can do a better job of optimizing decisions given the data science knowledge and tools they already have.

That is very exciting, Dan, and we'll make sure to include how people can reach you as well in the show notes. Now with that in mind, thank you Dan so much for joining us today and for sharing your insights.

Thanks so much for having me.

That's it for this episode of Data Framed. Thanks for being with us. I really enjoyed this conversation with Dan for and how he frames the importance of aligning machine learning with value. I'm excited to see where the growing intersection of decision sciences and machine learning goes next and what that means for data teams across the world. If you enjoyed this episode, make sure to leave a review on itunes. Our next episode will be with Sergey Fogelson, Head of Data Science at Viacom. In it, we talk about the evolution of the data tooling stack, his best practices, leading data teams, the importance of democratizing data, and more. I hope it will be useful for you and we hope to catch you. Next time on Data Framed.