#156 Making Better Decisions using Data & AI with Cassie Kozyrkov, Google's First Chief Decision Scientist

Hi, I'm Maggie, VP of Curriculum at datacamp, and I know firsthand how overwhelming it can be to keep up with the rapid changes in data and AI. But staying ahead doesn't have to be hard. That's why our curriculum team has curated the best learning content to help you and your team stay ahead, covering everything from data and AI literacy for busy executives to cutting edge generative AI for developers, and everything in between. Whether you're upskilling yourself or your team, we make learning practical, engaging and flexible. It just works. Power your data and AI transformation journey on datacamp.com.

I just want the world to be so much better at decision making. We're going to embarrass ourselves in 500 years time. We're going to look at all the things we thought were okay and we're going to be amazed that we ever allowed ourselves to do this as a species. Like the way we look at bloodletting and leeches now. And I'm just so excited that we're on the precipice of change, accountability and actually taking decision making as a skill. Foreign.

Welcome to Data Framed. This is Richie. The data scientist role has matured over the last few years to the point where most people applying for a data science job could reasonably understand what they might be asked to do when they got the job. But then in the last year, generative AI has upended everything and the job is changing again. So it's time to talk about how being a data scientist or any other data practitioner is changing. Today's guest is a true star of the data world. Cassie Kozikoff is the CEO of Data Scientific and was previously Google's chief decision scientist. Kassi is such a deep thinker on all aspects of data and AI and so I have many questions I want to ask her beyond talking about job roles. So we'll also get into data quality and designing for data and the rise of decision science as a discipline. Time to pick Cassie's brains. Hi Cassie, thank you for joining us on the show.

Hi, thank you for having me. Great to be here.

Brilliant. So one thing I've heard you say before is that you don't think that data science lives up to its brand name. So can you tell me a little bit more about what you mean by that?

Yes, I think that when people think about data science, the two kinds of images that come to mind is one is the professorish type or at the whiteboard with, let's say, chalk to make it extra old school and bow tie and lab coat for no Good reason. That kind of stereotype, doing very careful mathematics is something so intelligent. That's one kind of stereotype. The other kind of stereotype is the hacker listens to the business problem for two seconds, grabs a laptop and everything is solved. And those are so, so far from reality. And I think that a lot of people fall into data science for this sense that that glamour of doing complicated mathematics is what the job is about, or the glamour of solving enormous problems really quickly in a way that might have happened a few times back in the low hanging fruit days of data that attracts people to the field that then find themselves deeply disappointed and they're sitting around and asking, where has it been six months? And never mind neural network, not even a single regression has happened here. And so the brand problem is partly this sort of disappointment in what the job actually turns out to be and how it isn't well connected with what people think they should study to then get it. And also partly just the realities of stakeholder management and what it means to solve useful problems.

So I certainly agree with the idea that it would be very cool if you could just tap a bit on your keyboard and solve all of the stuff. But it sounds like there are a lot of tasks that data scientists end up doing that are fairly low value. So can you give me an example of where you think data scientists are wasting their time?

Well, that's actually where I was going to go. So I love this collaboration. Thank you. There are two, broadly two kinds of time wasting that happen. One has to do with the data scientist is not set up for success when they join the company. And the other has to do with the data scientist is not trained to be useful to the company. And these are different. So the one that I wanted to bring up is some advice that I have for folks just getting started in data science. Some questions that you should absolutely consider asking in your job interview. I wrote a blog post about this. So if you want to go find them there again and then forward them to whoever is suffering with you listeners, you can feel free to do that. The title for that was pretty cheeky. It said, data Science the sexiest job of the 22nd century. So that's a callback to an article that was written where it's the sexiest job of the 21st century. And the idea is that maybe of the 22nd, when companies are actually ready to make use of you. So here's my advice. I want you to ask any potential employer about the following things. First, whose job is it to make sure that you have data because it is very easy to take a gig where there is no actual data for you to be a data scientist with. And that is a great way to be ineffective because here you are expecting to flex your muscles on data that's just simply not there. If it is your job to get the data, that might be okay. But watch out first. Are you trained and willing to deal with the nitty gritties, the real world nitty gritties of data collection? Because there's going to be a lot of stuff that does not feel glamorous to somebody who didn't expect to get their hands dirty with the real world. Things like if you are doing survey questions, do you have pens and pencils for the people filling out the survey? Did you check that the battery was charged on the laptop? The little things that feel a bit janitorial, for lack of a better word, that folks who thought they were going to take all the highest mathematics classes to make themselves as marketable as possible just feel like they might be above. And then you get pulled into this. Now I think that you learn a lot by doing tasks that you have the impression you're above. When you meet the real world face to face, you learn a hang of a lot. So might actually be a great experience. Now that's it. That's the nitty gritty of data collection. Data design. How do you design that collection? If you have a social science background, you are probably best positioned for that data design piece. But even so, it's not something that we teach in school. And I would love for us to dive a little bit more into this topic later. But you'll be flying blind. That's also very uncomfortable. Now what if you are doing data collection at scale? Maybe you're working with a tech company and the data collection is working with the engineers to start logging things that weren't logged before. Well, that data, those logs have to then end up in an analyzable form for you, the data scientist. And the piece in between is data engineering with a whole lot of IT problems sprinkled on the top usually. Are you qualified for that? Because that is a different job. Data engineering and data science are not the same job. And you might be saying, I want to be the everything of data. I'm going to step up, I want to do all of it. Well, that's kind of like if you are hired ostensibly to translate between, say Mandarin and Swahili and that's your speciality, and then you're told instead you're going to translate between French and Japanese. Okay, it might be fun for you to learn another two languages, but at the end, once you've mastered your French and Japanese, what happened to your Mandarin and Swahili? So be careful. Trying to be the everything of data all on your own is a great way to set yourself into a risky position. So always ask, who's responsible for data? What kind of data is already around? Who checked it for quality? How has it been documented? Did it just come from a vendor? Was it collected by the company? Are there adults around? Now? Speaking of adults, who is responsible for making you successful as a data scientist? Who is the decision maker? Who actually frames the problems that you work on? Who provides the domain expertise? Who gives you enough of the business know how so that you are guaranteed to be solving the right problems? Because if someone just brings you in a room and says, okay, data scientists just do things, there is such a great chance that you are going to run up against the type 3 error of statistics. And so for those who don't remember their errors, type 1 is when you incorrectly reject the null hypothesis. Type 2 error is when you incorrectly fail to reject the null Hypothesis. And type 3 error is when you correctly reject the wrong null hypothesis. In other words, using all the right mathematics to solve the wrong problem entirely. And if you don't have someone who actually understands the business, understands the strategy, helping you work on the right things rather than the wrong things, how are you actually going to be impactful? And the more mathematically sophisticated highbrow your work is going to be, the more support and buy in you need. In a company that's just not ready for that level of model and that level of contribution, it is likely to get looked at briefly and thrown away. Also, who's on the hook for implementing what you make? Because the data science side is often not the hardcore engineering side. And a great lesson of this goes all the way back to the Netflix prize, when. So this is a very famous machine learning prize. It was about improving the Netflix algorithm by at least 10% for recommendations. And so the winning team, they got the 10% improvement and Netflix said, thank you very much, there's absolutely no way in hell that we are putting that into production so you can make complicated, sophisticated data science work. And it might fall on infertile ground, it might just not go anywhere. If it's not put into production, it may as well not have happened. Right? If a model falls in a forest and no one hears about it, then what's the point? So how you're going to work with stakeholders, who is responsible for making sure that you work on the right problems is a big, big thing that you have to worry about. And if the answer there again is you, are you a domain expert of this domain? Do you have good relationships already with all the people around? If you're coming straight out of school, the answer to that is no, no and no. So be very careful and do ask those questions. And then also tools. What are the tools that you're going to use and who is responsible for making sure that those tools work? Corre do you want to be on the hook for figuring out which particular platform is going to be approved by your IT department? What's going to create security issues? That is a long time that you will be spending in meetings and running around and not doing what you think of as data science. So you have to understand that the data science piece really relies on this enormous team and this enormous real world context of stuff. And not every company has that. But a lot of companies just want to hire a data scientist because they expect that you got one, you got a unicorn that's magical and you just put them in the business and magic happens. And that is not the real world.

There's a lot to think about there. And I have to say I love the idea of a type 3 error where you've solved the wrong problem. So you describe quite a few ways in which a data science job can go wrong. I'm wondering, is there like a utopian data science job? Like what does a good data science job look like to you?

First, let's get ourselves past that stereotype of the quick fingers on the keyboard and everything is solved. In the early days of any discipline, you have these quick successes available and that low hanging fruit scenario goes away. Data science has been around for a long, long time and there might be pockets. There might be pockets where like I made a scatter plot here is $10 million. It doesn't work that way most of the time. And so broadly speaking, you've got two kinds of gig. You've got the gig on the team that's already very much a data science team that is solving complicated problems. Now there's pros and cons here. The pros are that you get to work with your own kind. Chances are that the person fielding requests to this big team of data scientists is themselves a director of data science, something like that. A data science leader speaks your language, speaks stakeholders language. What comes to you is the right kind of work. But the low hanging fruit has so thoroughly been squeezed out here that the chances are you are making improvements, like at the eighth decimal place or something. You're not doing these wild write home to your parents impactful kind of projects. You are making incremental improvements because here's a large team of people, they've already done most of what there is to be done. And so here we've got a little plus epsilon that we're all doing. So pros and cons there. Alternatively, you could be the singleton dropped in to some organization that may or may not know what a data scientist is. And this is where it's really dangerous not to have someone looking after your career who gets what your work is and how you're supposed to be doing it, what is your fault if it goes wrong and what is a problem with the company, what should you be held accountable for and what excellence looks like? So here there is a lot of opportunity to contribute in generalist kind of ways, but at the same time, your contributions may be devalued or misunderstood or you might be misdirected. I mean, the worst kind of thing is where you end up being used as a pawn in a territorial dispute between leaders where you're just backing up decisions that they've already made and really adding no value except helping them prop up their claims and kind of bamboozle and bully everybody else with data. So you really have to watch out for that. On the other hand, the pros are that there may actually be opportunities for larger impact because if no one's looked at any data at all, you might do some even basic analytics and discover aha. Here is a way that I might propose that we could create some changes in this org on this team. You could actually end up creating some things that you could feel very great about. The chances that what you need to be doing in a setting like that are really, really sophisticated are lower of course. So you want to do sophisticated work, that much larger team probably makes sense for you. Not all companies have that. And then if you want to do some analytics for inspiration, get people excited about potential directions to explore and it's not all low hanging fruit, but it's a little more wild west then being a singleton could be a good fit. But again comes with that fundamental misunderstanding and potential lack of empathy from your coworkers.

So it sounds like really a lot of it's about your appetite for risk and do you want to have that chance of an amazing impact, but you're living in chaos or do you want to be part of a larger machine?

Well, I'm not Sure. I would characterize that as being about risk. Exactly. The larger machine comes with risks too. It's the risk of standing out career wise. How will you, if you believe that you are brilliant, if you believe you're the best data scientist in the world, you have a very major risk of getting mothballed on that team because projects that would allow you to shine just don't get to you.

Yeah, certainly being able to work with your colleagues and not tell everyone you're the greatest person in the world, that's general good career advice.

Yeah, exactly.

Okay, so I'd like to talk a little bit about how generative AI is changing the role of the data scientist. I guess it's been like a year since generative AI went mainstream. So to begin with, what do you think has been the biggest impact it's had so far?

Cynically, I could say it is a new kind of stack overflow with a bunch of pros and cons. So it does help you write your code faster. If you learn how to test what you're doing faster overall requires some debugging, know how, because if you just take it, plug it in and hope for the best and never check it, it's going to slow you down in the long run. But having some ability to get some help with code can be very useful. Where the actual interesting stuff is, if we step away from our cynicism, is that the way to think about generative AI? The revolution is that it is more of a design revolution than an AI revolution. So from the point of view of the large enterprises, The Googles, the OpenAI's of the world, the foundation models follow the same principles in terms of AI and software development as the big foundation models of all the other kinds from the last 10 years. There's actually not that much difference from say Google's perspective. The difference is that before it was highly embarrassing from a user experience point of view for the user to know how the experience was delivered. Like you shouldn't have to know whether there's JavaScript on the web page. It would be embarrassing if you as the user knew that. Right? The web page just works. And similarly, you get your recommendations from Netflix, you get your search results from Google and at no point does anything yelling at you, We've got AI, we've got AI, it just works. That's how it's supposed to be by that old user experience design philosophy. Now we've got this completely different design philosophy which is very much AI for AI's sake, AI in your face as a Raw material. So from the perspective of the Googles of the world, it's still one large monolithic system. It is designed to be performant in certain ways that are set by the company. And then this is all given to users and users are told, here, use this thing. This is AI, it's a raw material, build with it whatever you want. And there is so much opportunity for trouble and also delight in something like that. And what data scientists should be trained to do, particularly if they are of the statistical bent, is testing things. Question of does this raw material work for our needs? And when we have built upon a raw material that comes from a completely opaque process, like you don't know what kind of data the foundation model was trained on, you can't go debug that. You don't actually know what the objective functions were. So it's totally opaque. You're now going to take the raw material that comes out of that, you're going to use that for whatever your own needs are. If the task is unimportant, who cares, right? If I'm like, hey, ChatGPT, write me a poem about bananas in the style of John Milton, who cares? I'm going to get something out of that. Maybe I like it, maybe I don't. It's up to me, no statistics required. But what if I'm going to start basing some kind of solution that I'm selling? I'm a startup, I'm going to build on top of ChatGPT or one of the other things, and it has to work, has to work for my clients. It has to be useful and effective. How do I think about testing that? How do I think about building safety nets and testing those safety nets? What kind of data should I use for that testing process? How do I know if it works? This business of how do I know if it works is a very much core skillset for data scientists. And I think that we'll get past that very individual usage of those poems about bananas, and then we'll start wanting to actually build things that we can sell on top of these products, and then, oh my goodness, there will be so much fascination with does it work or not? And that's a great career opportunity for data scientists.

That does sound fantastic. But it does seem like it's a very different skill set being able to test the results of some generative AI model compared to, I don't know, running a logistic regression or something.

Yeah. So the analogy I want to use for this is I have this and I should just send you the image so you can put it up. But I have an analogy. That is my kitchen analogy for the AI space. And before generative AI, it would just have four images in it. It would have like a tomato and it would be ingredients and data. Right? Those are the analogies and your data is your ingredients. Then a microwave or an oven. And I'd have appliances and algorithms. They're analogies for one another. Your paper recipe, the model, and an actual recipe that chefs would want to cook from. And then at the end, your pizza. And that would be your dishes or your predictions. And that's from the enterprise point of view. Now we've got this new layer which is that you take that pizza as a raw material and that's your data product. So maybe you could do anything with that pizza. You could spray paint it gold and call it art. Right? And, and then that's a very different quality. That pizza doesn't have to taste good to potentially be good art. You could put on the wall. So that leap is the difference that instead of just serving pizzas at scale, which as companies would have done, now you are giving that, not in a take it or leave it fashion, but build on top of this, do your own thing with it. And there's a whole can of worms about who takes responsibility for that. What does it mean to do it safely? Where should the boundaries be? Where should companies prevent those who build on top of it from doing certain things? I think what's really key is that forget the data product. The gold painted pizza, the gold spray painted pizza. In a large enterprise setting, each piece of this process is somewhat discoverable. You should know who your data vendors are. You should be able to go interrogate those data sets and see what's in there. Are those ingredients garbage? Because any Michelin star chef would tell you that you can't make something great if you're working with bad ingredients. Same thing holds for data. So maybe you made the data yourself and you know all about it. Maybe you bought it from a vendor. At least you can go interrogate, see what's there. And what about the appliances, the algorithms, the math that you're using? That's somewhere in your code base. You should hopefully kind of know what you did. And to some extent, to the extent that you really understand these algorithms, you could see some mistakes coming. Like, let's say you do regression. Well, we know that regression doesn't handle outliers really well. So what's going to happen when you get an outlier? We expect major failure. The recipes themselves and the models, you get those out explicitly. You know what's coded in there. You could look at it now. It's going to be too complicated to read if you're solving these at scale, big data AI type problems, like you're not going to go read the output of the neural network, but theoretically it's there. You could, and you at least know the scale and you know the number of coefficients and whatever, and you see the inputs and outputs directly. You lose that when you're just building on top of the output. You don't know what kind of data went in. You just don't. And you don't know exactly which algorithms and you don't know what corners were cut. You don't even know what kind of ensemble it is. How many models are in there. People are guessing how many models are behind ChatGPT. Right. There's a huge difference in terms of trust and testing and safety when you don't get that full view. And I'm not sure that people have really come face to face with how difficult that will make things for them in the generative AI space.

Do you think we need more transparency about what goes into these large language models then?

That's an interestingly phrased question. I would say that there are many things that we would love to have. I would love to be able to ask anyone anything that's private to them. However, I would be able to. And there is absolutely no reason why an enterprise providing the service would give their proprietary stuff. I mean, for you to truly interrogate the data, you would have to get the data set. But the data set is where the moat is. The data set is the expensive thing that the company would have put so much time, effort, money into creating. And you want to look at it, that means you want a copy of it. Hell no, you won't be given that. And that makes perfect sense.

I think that makes sense because I guess it's becoming easier to create these models, like from a machine learning point of view. So actually a lot of the values in the data that goes into the models. That's an interesting thought. So I'd like to talk a bit more about the impact on different data roles. Do you think that generative AI is going to make some data roles more important or less important over the next few years?

Something that I have found a little bit ironic is that when I ask people to complete the following sentence, I'll ask you, garbage in, garbage out, garbage out. Right. They all seem to agree. Garbage in, garbage out. It will be garbage. You Base your data stuff on garbage data, and you're going to get some garbage model out. That's not going to have good impact. That's not going to work. So we don't seem to have any disagreement about this fundamental statement. And then I do this slightly spicy thing after I give talks. I give a lot of conference talks. People are interested in having me speak. I've got my link for that is makecassietalk.com, we have ways of making you talk. And afterwards I go and I hang out with the audience and I play this little devious game. So I go around the crowd and I ask them, okay, what's your job role? And they're like, statistician. And this one's like leader. And this one says researcher, social scientist, whatever, UX designer. And then I go around again and I say, who in your organization is responsible for data quality? Whose job is it to make sure that the data are well designed, intelligently collected and curated, managed and documented effectively? And most of the time the answer will be the same job role that they have just told me they are. This is chaos. This is unacceptable. How do we permit this? We are building our whole foundation on data. We all understand garbage in, garbage out. And yet nobody can tell me even the name of the job that ensures that what goes in is not garbage. Wow. I have no word. How did we get so many years into this and we still have this problem? And maybe part of that is that we have not made this job sexy. It's an afterthought. Everyone wants to do some fancy mathematics. Nobody wants to think about data collection. If we go back to the story of Fei, Fei Li, who I'm so glad that she's now getting recognized for what she did, which was very brave as a PhD student in Caltech working on AI, everyone told her, go do some fancy math. She said, we need data. Let me go and get volunteers to label photographs and just say, what's in the photograph? Is it rocket science to look at a photograph and be like, there's a cat. I'm gonna write the label cat. No. So she got a lot of flack from her advisor. She kept expecting that she would have to go and work at, I think her family Laundromat or. I'm not sure I remember the details of the story, but because it didn't feel Caltechy enough to be doing that, I think that's such a heroic act that she did. Without her wouldn't have Imagenet, we wouldn't have progress in vision. AI because someone had to go and do the foundational bit. No one wants to do the foundational bit. Everyone expects somebody else to do it. And then they get to do the fancy mathematics on the top of it. This is going to be a huge problem. Now all this generative AI stuff, it is fed by one thing and one thing only, and that is data. And the better your data, the less fancy your math needs to be. Whose job is it? And how do we make new, fresh, talented young people actually study all the things that you would need to know to get the data side right? I mean, that's a pinch of social science, design of surveys and human psychology, a pinch of data engineering, pinch of statistics. How much of it do you need? What kinds? What if it's stratified? How do you actually design the sources that you would have to go for? The randomization schemes, all the real world practical details, plenty on security. If you're going to, you're going to need to know some things in the medical field about HIPAA more than you thought you wanted to know if all you were interested in is mathematics and a whole host of other stuff. So it's actually quite a complicated discipline in its own right. But we insist on telling people that it is not valuable, it's not as good as data science. We're building data science on a foundation of junk. And I had this conversation with a data science influencer and I was asking this person, so how do we make this job sexy? How do we get young people to want to study this, to contribute to quality in quality out, not garbage in, garbage out. And what should we call it? And the off the cuff response there was, oh, isn't that just data janitor? And I thought this is the whole problem beautifully encapsulated. Like imagine you tell young people, come take a degree in being a data janitor. What are they going to do, call mom and dad back like, hi Mom, I am studying to be a data janitor? No, that's never going to work. So it's on all of us to get a little bit more humble and to say how desperately, desperately we need this for all data science. And now, even more desperately than ever before, generative AI Again, that's the moat. Good quality data. Someone's got to get the good quality data. Whose job is that? Who knows about it? How does it work?

Yeah, I can certainly see how saying data janitor is the sexiest job of the 21st century. That's a very tough pitch. Yeah, fairy tough, but it sounds like a lot of this is quite close to the idea of data stewardship and making sure that you've got high quality data like throughout the whole sort of data pipeline. Are there any things that you think data professionals need to know about stewardship of data?

Look, I've kicked around a lot of different job potential job titles for this. Data steward doesn't feel active enough as a position. It's being like the data's security guard. You're just there and you hope the data does its thing. And yeah, I'm not sure it carries the gravitas that you would want, because that's where I want young people going. If we're building on data, if we're building a data driven, data fueled world and society want that data to be good. A data hero is what I would much prefer to data steward, but you know that that has its own problems. Often things I kick around as data designer, but this is so much more than design. There are a lot of things you're going to have to think through and design and deal with the consequences of. And it's not a soft discipline, it's got plenty of mathematics behind it too. So I don't know that I personally cannot think of a satisfyingly hearty term for this that really captures the art piece and the science piece and just that incredible importance of the role that is so neglected right now. And what I would say as a piece of advice for people getting interested in it is that data documentation is so, so hard, especially because that's where the moat is. And because you don't want to share copies of the data, you need some way of allowing your potential client or potential user or data scientist using that data to interrogate what might be in there and would they want to pay money for it, and so on. And for that you need some notion of transparency, which is hard to define, and you need some notion of well documented information about its provenance, its schema, but also where the gotchas are. So how do you do all that without giving the actual data set? That's a really hard problem. And there have been nice attempts to make some headway, some from Pear, people in AI research at Google, some from Microsoft and others. But really what a lot of those papers are saying is, my goodness, we're stumped. There is not just one way to do data documentation. And when you get that situation, everyone's going to start doing it differently. There's no standards on it and what chaos, what chaos will all get. And so I think that a lot of people who just Want to do some shiny mathematics on data that's arrived to them through everyone else's sweat and labor. Don't realize just how wobbly and subjective the whole procedure is to get it to them in the first place and how much effort and attention we should be putting into shoring that.

So, yeah, I mean, I think on that note, maybe like a lot of the challenge with data documentation is it ends up being maybe the data engineers are the ones who know the exact details of how some data was cleaned, but it's actually the business people that understand it. And I don't know whether you have any thoughts on, like, how you make a system that is both suitable for data engineers and business people as well. Like, how do you reconcile those two different points of view?

So interesting that you bring that up. So when I was part of the PEAR project on data transparency, data documentation, one of the things that we realized early on is that to make some headway, it would be worth having workshops or trainings at least in what to think about and what to contemplate documenting before getting started. So that's available, we've published that, we're very proud of it. So people can go check that out. But there are many, many more than two Personas. So from a UX point of view, who are the Personas? It's not just business people or data engineers. There are a lot of different Personas that you as the data documentation expert would need to consider. And yes, it's hard just with two. I completely agree with you, Richie. It's worse when we're talking dozens of Personas that you might want to think about, particularly when you think about data vendors. So a data vendor might be selling to all kinds of clients. Maybe it's a government client, maybe it's a regulator, maybe it's a machine learning team that wants to automate based on that data. Maybe we're just curating it and we don't care. We don't know what we want with it. Right now we're just making this big old hoarder's storage locker full of stuff. And good luck to whoever comes next. Different Personas will have different needs. There's a whole guessing game of who's going to use it, how, that you can't perfectly guess. And something else I want to say about data is that in the old days, when dinosaurs like myself were crawling the earth as statisticians, there was quite a premium on data storage. So you didn't have really big data sets because you didn't have really big Data sets, you would be very careful about what you collected and you would design and curate carefully. And maybe then the mental model would be that this would be like a gallery, exhibition or a museum, like intentionally selected shiny things that make sense. Each column is there for a good reason. And then as data storage became cheaper, then you can imagine the free for all is more like a hoarder's storage locker. Nobody cares. It's just throw everything in there. And this is really great for analysts who are supposed to pull inspiration out of stuff. This gives them more sources of potential inspiration. So they want you keeping only these five columns and toss everything else away. Because you can't perfectly curate, document and design it. They're like, yeah, just toss everything in there, we'll deal with it later. That's literally their careers. They have every incentive to want more sources of inspiration. That said, when we start talking about documentation, maybe the analogy there is the labels in a museum, if you want to chuck more things in without worrying about it and bring that storage cost way down, you're explicitly going to not document. The more things you document, the fewer things you can keep because you need personnel hours to do the documentation. And so this business of just toss everything into that hoarder's attic and worry about it later does not lend itself well to the act of data documentation. So there is this push and pull, there is that statistician museum curation, fully documented, perfect thing. And then there's the data lake, hoarders, attic, chaos. Don't worry about it, let them deal with it later. That's antithetical to that first line of thinking. I'm not even sure that organizations and leaders face this thing head on and actually ask themselves, okay, what are we optimizing for? Can't really have both, not at any reasonable price point. So which one are we trying to do? And chances are that they've hired people who come from particular schools of thought, lines of reasoning, who are not questioning why they have the knee jerk reaction that they do. So maybe you got a bunch of stats people who are like, document the hell out of it. Maybe you got a bunch of analysts to deal with it later. Even that is something that tends to go under the leader's nose and never be addressed head on. It's literally the leader's job to make those sorts of trade offs and judgment calls.

Yeah, I can see how it might be quite hard to get like a lot of sort of chief Data Officer's attention to be like, hey, we need to worry about documentation. Really, really Important, but actually it's a.

Secret because what are we going to use it for? Yeah, yeah. It might not even be important. So someone's going to make that call.

Yeah, I suppose. Related to this, it seems like, again, there are sort of design issues around, like, how do you make sure that this data is accessible and usable by the people who need it? Do you have any advice on what data people need to know about design?

Oh, yes. That is literally what those workshops are for. I would say go and take one of those workshops that the folks from people in AI Research at Google have put out. That is for creating empathy between the people creating, collecting, stewarding the data, the people analyzing the data, and all the other Personas in the space. And how might you actually think about who needs which bit of it for what and what would make sense to document under those settings? So that's actually a really rich topic that I think, saying, here's the training. It exists. Go get it. That might be the best way to tackle it.

Okay, fair enough. And just going back to generative AI, I mean, I suppose that's had a bit of a design revolution in that. Hey, we've got this chat conversational interface now, but I'm not quite sure that's the sort of the end point. Do you have any thoughts on where that's going and whether there'll be other ways of accessing generative AI?

Well, sure. How sci Fi should we be? I mean, let's go. Let's go all the way Sci Fi. Let's go all the way Sci Fi. The idea of a landing page, that is whatever your favorite bank is, bank of America's landing page. That it is one. It's like that. It's designed for all users. You show up there and you get that page's experience with a little bit of customization based on cookies or something. Right. That's the Internet as you know it today. Why would you have to have landing pages? If you could generate experiences on the fly on the Internet, and if you could have an experience generated on one side representing the user interests, and you can have an experience generated on the other side representing a company's interests, brokered together into a seamless experience that would completely do away with the concept of landing pages, which would do away with many of the Internet concepts. So that would be pretty revolutionary. So you asked me for Sci Fi, you could have an on demand Internet. Yeah, you could theoretically have an on demand Internet.

Okay, that's cool. Yeah, yeah. I do like this idea. The Internet is just completely Personalized for you.

Yeah. Okay. And there would be so many issues with that because all the problems with the social media echo chambers are barely a fingernail clipping on the problems you could have if the whole Internet was a personal experience.

Absolutely. Yeah. You never need to have another point of view ever again.

Right.

I think I'd like to talk about the sort of democratization of data science. So it seems like generative AI business intelligence platforms, better tooling in general. It's just taken away a lot of barriers to entry to data science. So how do you think the increased accessibility has changed the role?

So I wrote this article that is titled is Data Science a Bubble? And there I was asking myself, what are the consequences to the data science career of a big influx into the field of role dilution? And just that evolution of a profession that starts with very few hardcore people and becomes democratized. And there I introduced the concepts of the and data scientist versus the or data scientist. So the and data scientist would be somebody who is a statistician and they're an analyst and they're an AI engineer and they're a machine learning specialist. And, and. And somebody who thinks that data science is the everything of data and that you should be expected to be maximally hardcore. You should be paid a lot because you are this amazing unicorn who can do everything. And the end data scientist is incredibly offended by the or data scientist. Just somebody who dares to wear the exact same job title as them. That is a job title that's supposed to come with money and is an or data scientist. As in they just do one of these things. Maybe not even that well, they touched a regression at some point, they made a scatter plot. Now they call themselves data scientists. You can imagine the horror from somebody's point of view. I spent 20 years becoming this good. How dare you? How actually dare you do this? Okay, I feel for both sides. It is horrible to have had the impression that in order to practice in the first place, you have to have a level of quality that's insane. And you put yourself through that very difficult schooling and you show up and you're like, right, this is why I should be paid these dollars, because I'm actually rare and highly qualified. And then you see other people jumping onto the scene to pretend to be you to take your salary and dilute the market. How horrifying. I get that. On the other hand, the and data scientist is a threat to getting things done because there are just so few of them. If everybody has to be that truly top hardcore Level, nothing's going to get done. And it makes much more business sense to specialize. And even if you are an ant data scientist, you will not be doing all of them every day or every week or every month even. Projects come in phases and during some phases it's mostly statistics and during other phases a whole lot of programming and so on. And does that really all need to be housed in one person if it's not used all the time? Maybe it does make sense to specialize. If you have this expanding universe of data science, professional roles, data science adjacent roles, then maybe it's okay if people aren't at the everything of data and maybe it's okay if they only do some part of it. That means that you can have more people in it and you can get more done with data. I think that we are having that dilution. I think that there are a lot of people who have absolutely no shame, who wouldn't even qualify as or just put data scientists somewhere on their resume and hope for the best. And it does not carry that gravitas, that emblem of quality anymore. It just simply doesn't. And so when everybody is a data scientist and it doesn't mean anything as a role, the market's all over the place, high variance in pay, and then you see a lot of people who want to have this hardcore credentials tucking and rolling and trying to call themselves something else. Welcome to decision intelligence folks. Still hardcore around here. But I think that fundamentally, if what data science is, is the discipline of making data useful, that is never going out of style, no matter what we call that job. The more data we have, the more data fueled technologies we have, the more we need people who know how to make data useful. Do they need to know everything about it? I don't think so. At the end of the day, do we need a little more honesty between hiring managers and folks about what the skills are? And that means that hiring managers might need to be more educated on what a good candidate looks like and not just take it from the job title. We absolutely need that too. But making data useful is never going out of style. Job roles, job titles. There may come a point when people are too embarrassed to call themselves data scientists because of the way that economies go.

So maybe naming fashions change, but you think like is the end game going to be lots more or data scientists, do you think? As it becomes even more.

I think so. Because it just doesn't make actual economic sense to hire a person who is vastly overqualified, who's just not going to do all of the things that they're qualified to do on the job. You could hire somebody who is very, very good at being a brain surgeon and also a car mechanic and also a fighter pilot. But are you going to be doing all those things at once? It probably doesn't make sense. It's a big waste. It's a big waste for you to have the qualifications and the thing that you, you're not doing at any given time.

Absolutely. You don't want someone with many, many data skills and then being the person, like you mentioned before, being the person that like checks that the laptop has enough power, right?

Yeah.

Okay, so if the role is changing, are there any skills that you think are becoming more important? Like what should people be learning about?

Well, the role is fragmenting. So the data science universe is expanding and it means that it's now time to ask what flavor of data professional are you? So if you are an analyst, for example, then your excellence is speed. How quickly can you get inspiration out of the data into your own head and into your stakeholders heads without wasting their time, without bringing them red herrings? So that is a game offspeed. And when you are leaning into that, you will be leaning into, how do I make myself faster? At this business of extracting inspiration from data, every skill relevant to that you will want a little more of. If anything really advanced. Analysts are the ones who most need to know how the algorithms work. Much more so than statisticians who just need to know that it's the right thing. Or machine learning folks are just like, I'll get it off the shelf and run it and see if it works. The analyst has this massive multi dimensional space that they have to try to make sense of. So for them a P value is not a philosophical concept as it is for a statistician or decision maker. For them a P value, if they understand the mathematics behind the P value, then they understand how this big space is being projected down into a single number. And when they see the single number and they see the formula, which they know, they can make some guesses about constraints on this big confusing, hard to visualize thing that they were dealing with. So knowing stuff like that is very useful to them as they're trying to get all that inspiration into their own brains quickly. Domain knowledge, leaning into domain knowledge for them also helps them be faster, helps them not snooze past the gems and also avoid the red herrings. So kind of leaning into which part of it you actually do is where the core thing is. Statisticians are Helping decision makers, or maybe they're the decision maker themselves, they're helping themselves make one or a few big important decisions under uncertainty. There are a lot of skills you could lean into about that. And you will also notice that there are skills that seem data adjacent that are just so much less relevant. So think about what you're actually contributing, not in a name of the discipline, but like what do you actually do around here? Do you make automation work? That's a whole nother thing. You might lean more into engineering type skills if you do that. And even in these three categories as AI, stats and analytics, it fragments further this kind of approach on this kind of problem, this kind of setting, and certain skills will be more relevant than others. Again, so maybe letting go of this desire to be the end data scientist and know everything and just ask yourself, what am I actually doing around here? What would make that go faster? Never mind what it's called, how do I help myself do it better.

So we've got back to the idea of personalization again. So from the sci fi future Internet, where everything's personalized, actually your job's going to be personalized as well because the data space is just, well, fragment.

I think that this is nothing special. I think that you had doctors in the early centuries of humanity being the everything doctor and doing it badly on their own. And now you have hospitals full of all kinds of staff who are specialized in much more narrow parts of the process. They know exactly what their job is, they know what it means to be good at their job, and they collaborate together to create outcomes more effectively. It's an expansion of the universe of healthcare. Well, similarly we're having an expansion of the universe of data and data science. And so within that you could now have interdisciplinary collaboration where you would have thought that it was all one discipline, but it isn't. It's people working on different aspects. The soft skills that are backed up by technology and data are fundamentally very different already, even between a statistician and an analyst. And the more the universe expands, the more everybody has to specialize and become a version of a version of a version within what used to be a bigger category.

That's really interesting. And so you mentioned domain knowledge and collaboration. Do you think the soft skills that are most important are changing as well?

I think again the universe is expanding and so there is a different set of soft skills. And okay, here's what I want to say about soft skills. I hate that we call them soft skills. What I much prefer to call them is that they are the skills that are hardest to automate. When technology becomes better, when the UX of these tools becomes better so that you can program faster, for example, when there's less fiddling, when you don't have the strings as factors problem anymore, then you can spend your time on what matters, on why you're here fundamentally in the first place. And there's going to be some part that resists automation. A lot of what we were so excited about in data was just that we could, we could add two numbers together. How wonderful. We didn't worry about what it was going to be for. We were just like, wow, we can do this. And then wow, we can have a spreadsheet and wow, we can make a scatter plot from it. Not what it's for or why, just that we could do it. Well, now there's so much that we can do that the reasons that we're doing it all are hanging on the soft skills. What are we trying to achieve and why are we trying to achieve it? And the better our tools get, the less we huff and puff and do the equivalent of work with punch cards when you could work with a laptop. Is it less hardcore to work with a laptop than punch cards and vacuum tubes and build your own computer from scratch? I would say no. I'd say that the soft skills become revealed faster though, because if you're spending all your time building vacuum tubes and playing with punch cards and all that, you will look very busy achieving a relatively small amount as the tools become easier. Now you will have to be accountable for the why and the thought process and you'll start to see very different thought processes, very human parts behind data science. What statistics is, is effectively epistemology. It is numbers based epistemology, it's philosophy and it's decision making. But if you spend so much time fiddling with deriving a T test or whatever it is, and then going to the back of the textbook and looking things up in that table. You might not even remember why you're doing what you're doing. You just go through the pantomime, the dance of, well, I guess someone told me to do a T test if I have this kind of data, so I may as well do it. Well, what if that happened as it does in R with there's your T test function, go. Now all of a sudden you're forced to do the bit that was actually important, which is like, why this? Why does it make sense? How do I think about this philosophically? That bit's not automatable, is that soft skills, I would say that those are skills that are hardest to automate.

Yeah. I can certainly see how understanding, like, what do you care about and why? And communicating that to your colleagues or whoever else. Yeah, very difficult to automate. And I think these skills are going to be around for a long time.

Yeah. All these tools are just ways of executing the gaps between the soft skills and that's going to shrink and most of it will look like soft skills. I think that's brilliant. The less time we spend fiddling on things we don't need to fiddle, the better, because the fiddling bits, honestly, I have this distinction I like to make between thinking and thunking. Thinking being those flashes of brilliance, the moments when you're actually thinking about what needs to be done, why it needs to be done, where you're on, where you're cognitively engaged, where you're communicating, solving and then thunking is, you already know what to do, now do it. The way in which we glorified the STEM disciplines has led us to wholeheartedly pretend that all this mathy stuff is filled to the brim with thinking. I would not say that I agree that it is inherently any more intellectually engaging than a whole lot of other disciplines. Like, yeah, mathematics is this way of being very careful, very precise and saying relatively little in a very precise way. Mastering symbolic manipulation, but it's not that much more glorious than following recipes in a kitchen. Similar, actually. So what I'm trying to say here is that those very technical pieces that are filler between a lot of the thinking. Yeah, there's some thinking in there too. A lot of it's not really. A lot of it is once you've learned how to do it, safe and simple minded. Yes. It's hard to learn how to do it in the first place once you've learned how to do it safe and simple minded. And then when you take that away, oh my goodness. Now this is the deer in headlights moment, when you have to think about the what and the why and the open ended chaos of what should we do next?

Absolutely. So, yeah, I can see how once you've learned your Python or R, then you can do a lot of thunking there. And it seems like what we're leading up to is maybe the most important bit is like how you go about making decisions. So, yeah, I think decision science is your. Is really your forte. So can you just tell me, like how decision science is sort of related to data science and where the differences lie?

Oh, okay. So this is A place where the branding is awful and the words are confusing. And so a little bit about the history. So decision sciences, not decision science, cellular decision sciences, plural, used to refer to the collection of disciplines that had something to say about decision making. So you would think about your economics, psychology, neuroscience, and your classic candidates. And of course, you would also start to bring in the operations, researchers, statistics, AI. And this is a bucket that was rendered deeply unsexy many decades ago. When I think about what brand colors might make sense there, it's like that grayish green of Excel, like no one's excited about it. And I think that that is because it was interrogated a bit before its time. Now, decision intelligence is the discipline of turning information into better action in any setting, at any scale. And how this is different from the decision sciences is not only is it a modern look, but it's also very impatient with the idea that you have to stop once there's data that, like it's some soft thing. And you did your pros and cons list and then now you got petabytes of data. Well, oh no, you have to hand it off to the AI person. No, you should be able to. If you were truly an expert in decision making, you should be able to turn information into better action. Any scale, any setting, no stopping, not with petabytes of data, not with no formally written down data at all. But how do we think about making the Most out of 10 lunch conversations to make a career decision? You should be able to handle any of it. And so that's a kind of academic umbrella, a more modern one that pulls into it everything that every discipline would have to say about decision making, with zero patience for throwing your hands up in the air and running away screaming when we get to the data science side of things. So then in that setting, data science is part of decision intelligence. The decision sciences, the traditional ones, which were not highly quantitative, are also part of that umbrella. But all the bits in those disciplines that has very little to do with decision making is now another major's required reading and just for interest reading in this one. So decision intelligence is the ruthless pursuit of better decision making. Everything that you would need to know from every point of view that is specifically about better decisions. If decision making is turning information into action, you better be really good at information. The data science piece is there. You'd also have to contend with the judgment piece. Now, in psychology, they tend to call their decision making discipline judgment and decision making. The judgment piece is about selecting the way that you want to decide in the first place, there's not one right way to do that. And so there are smarter ways to frame your decisions. There are dumber ways, but there's not one right way, which makes things very exciting. It also is fun to realize if you have a graduate degree in statistics like I do, some point, hopefully in your degree, you realize, my goodness, my entire degree. The judgment piece was done for me by my professors. I have never been taught to do this. The hypotheses were there and I had to use the methodology that I was taught to most effectively test them, but the why was never there. In fact, you almost get the opposite habit. You are disincentivized from questioning whether the problem is worth working on. You're incentivized towards type 3 error because it's some totally inane thing, some hypothesis about a bunch of rabbits that Sally has in. And instead of saying, why the hell am I dealing with this professor? You're just like, okay, we build distribution, blah, blah, blah. So that first judgment piece of how are we going to go about structuring our decision in the first place? So, so important. Also under that umbrella. Now, decision science today is confusing because it's. I think that I probably didn't help very much because I was one of the big advocates in this space and I chose to go from chief data scientist, title, decision scientist, title. The reason I chose chief decision scientist and not something with decision intelligence is that we joked about it. I'd be like, I'm chief decision intelligence, whatnot, but it's not pithy. So I was just like, all right, I'm going to flip data scientist to decision scientist to make a nod for the importance of decisions as the why. That kicks off all the data stuff, like information on its own. If it falls in a forest, who the hell cares? It's. It's through our decisions, through our actions, that we affect the world around us. So decisions are the important bit. I want to really point that out, even in the title. And then more people got excited by that and now there's decision scientists all around. Now, I would say that if I had to set a general direction for what a decision scientist would have to be for me to accept them as a decision scientist is that they would have to be trained in at least some aspects of decision making, but that their role, the way that they would execute it is not as decision makers, but as decision advisors who have the humility to, if they are helping a decision maker figure out, say, between A and B, to absolutely not care about A and B and this is a weird thing to say, but when I do decision science advising, I have to really, truly embody not caring about the options and caring only that the decision is made as well as possible. So it's not me who's making the decision, it's not me thinking like I like A. So I'm going to nudge this decision maker into A. Or the decision maker's already chosen A. Let me help them feel better about the A that they've chosen. Instead, I'm going to make sure that this decision is made as well as possible. So I'm going to take my trained decision making brain and I'm going to let the decision maker borrow it for a bit. But through their eyes, they're the actual decision maker. What's important to them, their priorities. I'm going to help them make sure they thought of everything, but in their own way. So somebody who does that professionally, I would say, great, I would love to call you a decision scientist and somebody who studies that as well. So that's how it all fits in. Some people also love to say decision science is the qualitative piece and the data science quantitative piece. Within decision intelligence. This is where we realize that things need tighter branding, otherwise we will get confused. That really is, in a nutshell, the genesis of the mess.

That's amazing that it involves so many different disciplines. So you talked about the data science side, you talked about psychology and then so the understanding, like how do you actually persuade people? How do you teach people how to make decisions? There's a lot there. Are there any aspects of decision science that you think data scientists ought to know about?

So I think that everybody would be better served taking a decision intelligence approach. That is having a hard boundary between data science and decision science, I think is not useful. So it would be better if we blurred a little bit there. It would be better if data scientists could take a more decision oriented approach. It would be better if decision scientists didn't throw their hands up the minute things got real with data. That said, what data scientists should absolutely appreciate is that they are a downstream worker of everybody else. And if you haven't trained yourself in the upstream pieces, you are a hazard. You're a wrecking ball in that space. So thinking that you've learned a bunch of mathematics and now you're ready to go and do the upstream stuff that you have a zero qualification for is a hazardous level of hubris. So don't do that. Similarly, because you are downstream of a whole bunch of things, your effectiveness depends on the effectiveness of the decision makers above you. If you are working with a bad one, that will have major implications on your career. So there's another reason why you might want to go and train yourself to actually be good in that space. So that if there isn't already an adult in the room, maybe you could train yourself and learn to be the adult in the room. However, you hadn't studied that stuff either. You are not the adult in the room. Knowing some mathematics does not make you so. So there's a lot to know in this area.

It seems like it's going back to the idea of collaboration and you need to know what your position is relative to others. So just to wrap up, is there anything you're particularly excited about in the world of data or AI at the moment?

What I'm excited about most of all is that there is an interest in accountability for decision makers, for leaders, that wasn't there before. And what this means is that decision making necessarily has to become more interesting. So the way that everybody was passionately running around after data science for its own sake, data for its own sake, no. Why in the mix? Just, let's do regressions. Why? Who cares? I feel like we're on the cusp of major change there, where we might actually start to do it in the sensible way around what's worth doing, why do we want to do it, how do we want to structure it, and holding people accountable for wisdom in those things. And that's going to be great for practicing data scientists, because the problems of dealing with loose cannon decision makers who don't know what they want and don't know what they're asking for, and then that has terrible impacts on data science careers. That's going to be somewhat mitigated by more scrutiny and interest. And for me, I just want the world to be so much better at decision making. We're going to embarrass ourselves in 500 years time. We're going to look at all the things we thought were okay and we're going to be amazed that we ever allowed ourselves to do this as a species. The way we look at bloodletting and leeches now, Goodness, why would people have thought that was okay? Or witch hunts? I think we're going to look at ourselves the same way in 500 years time. And I'm just so excited. So we're on the precipice of change, accountability and actually taking decision making as a skill.

That sounds fantastic. Having data science with a purpose rather than just for the fun of it. And it can be fun, but, yeah, having a purpose sounds like a good idea. Thank you so much for your time. Cassie, it was really great having you on the show.

Thank you, Rishi. It was really a delight to be here, Sam.