Agile and Analytics with Shaun McGirr - AgileData #25

Welcome to the Agile Data Podcast, where we talk about the merging of agile and data ways of working in a simply magical way. Welcome to the Agile Data Podcast. I'm Shane Gibson, and today I'm talking to Sean Maguire. How's it going, Sean?

Good, thanks, Shane. Good to see you again.

It's been a long time. So I think today we want to talk about agile and analytics, and no doubt we'll rift into teams and people and process and all the good stuff. But before we do that, how about you give a bit of a background about yourself for people that don't know you?

Sure. Today, I work at Dataiku leading a team of AI evangelists. And before that, I've had a long career doing all kinds of things with data on many different kinds of technologies. And leading data science teams, being the first data scientist in a place and realizing on the second day of that job they don't need data science yet. That time it was lucky I had worked for you at Optimal BI in Wellington and had learned lots of other useful things about all the rest of the data stack and the data practices. And I got my start in data by working for Statistics New Zealand as a holiday job doing admin things. And one of the tasks that someone gave me one day was a list of two printed out— two printed out lists of numbers, I should say, and they asked me to Can you tell them what numbers were in list A but not in list B? And at the time, I didn't know what a merge or a join or really anything was, and I didn't want to rock the boat upfront. So I sort of looked through and struck things off with a ruler and a pen. I guess this is 2001. And then I realized those printed out numbers were probably in a file. Can you email me the files, put them side by side on the screen? 'Let me copy that data from there to there.' And that was kind of how I realized if you stay curious and interested in solving these things that we call data problems, people just give you more interesting work to do.

Yeah, and as you said, we worked together many years ago in sunny Wellington. I think from memory that was before the term data scientist actually existed, wasn't it?

It was absolutely right on the cusp. So I had come there towards the end of my PhD program and had thought, okay, I'll just finish that off in 6 or 12 months because that's easy. And it took, it took longer. And in the meantime, I fished around for ways to apply the data econometrics statistical modeling skills I'd gained as part of my PhD. And at the time, I think we called it predictive analytics, advanced analytics, things like that. And I Googled some things and Optimal BI came up, and I sent you an email and said, Hey, I know how to do these things. Do you do those things? And then I think we started working together. And I think that was maybe early 2014. And it was just when that term data science was really starting to take off, which like all terms like that, they have a usefulness.

Yeah, and so let's talk about that one, right? So what do you reckon? Data scientist, is it dead? Is the term, you know, is the vision being realized and we're moving on? Are we still in the world where that term is still finding its feet, or did it crash and burn?

I think the term is not going away. It's just here to stay like many others. I remember lots of predictions over recent years about how there's going to be increased specialization. So people will become a marketing data scientist or a geology data scientist or finance data scientist. That still doesn't seem to have happened. And the reason that I still like and respect and maybe use the term is that I still believe, even God forbid, in that Venn diagram of, of the value of a kind of person who can live in multiple worlds at once, who knows about statistics and the usefulness of statistics in understanding how machines have learned patterns from data, and the ability to make those machines do some things at some kind of scale. And most importantly, and the one that I always lean on the most, and I'm most thankful for the training in my career that gave it to me, was the domain expertise. So do you even know anything about what you're talking about before you look at some data or write some code in it? The great lesson for me has been any domain expertise from anyone I've hired or worked with, any domain expertise in anything helps them gain domain expertise in other things. So I think the term, yeah, used, abused, people, employers claim that they can't find enough, but then everyone knows people with great backgrounds and skills who can't get hired. So it's all messed up. It's not helpful. But I still believe in the value of a person who can do those 3 things in that crazy Venn diagram.

Yeah, and I'd probably, for me, I've always, I know we started out with 3, but I kind of formed into 4 steps in data science or 4 sets of skills, right? So definitely with you in terms of the stats background, right? The ability to experiment, treat it as a science project, prove prove the hypothesis and use statistical terms to do, ways of doing that. Definitely around the domain, understanding the business and why it's important and the next best action. Or some action of insight, I think you would LinkedIn post about, just, what are we going to do with this? What value is it going to have for the organization? How's it going to help us make a better decision or increase the value or reduce the risk or whatever? I think there's definitely When we talked about the data scientist unicorn in the early days, you know, those equivalent of a full-stack engineer, that person that can do all 4 things, there was a whole thing around business facilitation, you know, plain English language, being able to talk to C-levels in a way that they understood. I think that was a key skill of one of those unicorn scientists. And then the 4th one for me was engineering, right? They could write code. They could do a push-pull PR request, git merge thing. And they engineered their code so it was reusable. And I think what I've seen, when you talked about data science specialization, that kind of tweaked my interest because I think now we're starting to see specialization again in the engineering domain. So we now have a data engineer, a supposed analytics engineer, an ML engineer, blah, blah, QA engineer, right? We're going into hyper-specialization.

Data ops. Data ops.

Data ops engineer. But I'm with you, we don't see the specialization for the data scientist. 'cause we're specializing on that fourth part of the Venn diagram, the engineering part, but we're not really specializing on domain. Like you, I don't see a supply chain data scientist versus a marketing attribution data scientist. Haven't seen that happen, which is interesting. But what I do see is we moved, from an agile point of view, we got good-ish at at combining the data people and the visualization people and bringing them together, right? We removed often the, this team did the data work and they handed it over to the viz team who did a pretty dashboard, and then the dashboard went to the user and the user went, well, that's not the right number and it's not the right way, and we all fought and blamed everybody else. We started merging those people together, right? Creating teams or squads that work together. But the data scientists tend to always sit outside, right? It was in the old days, you know, we made ETL developers, because I am that old, the Analytics people, right? They used to sit in another team in the analytics.

They floated somewhere, didn't they?

Yeah, yeah, they were somewhere secret. And the ETL team used to hand over data in the warehouse to the analytics team and see what happened. Do you see that now? Do you see that specialization, the data scientists still sit outside a core data team in the organizations that you work with?

Yes, yes. The number of places they sit is bewildering. And in my career, I've, I've been that kind of person and sat in many, many places, most recently aligned to strategy. And it's really interesting why. I think part of it is that The most charitable interpretation I can give is that one of the best things you can give data scientists to do, if they do those 4 things or if the team of them covers those 4 things you said, because there's no one that does those 4 things that you said, if that team has those qualities, then they need to work on very complex questions and solve those questions and problems. Otherwise, how do they justify their investment? And to do that, um, they're going to need to look in all the corners and under all the rocks of the data. And having been that person, having received that data from that ETL team or data engineers or whoever, I've always been suspicious about who made the decisions about what data is not in this dataset in particular. And so I think the most charitable interpretation is if you get your data scientists working on those hardest things, part of their value is in actually questioning all of the other decisions that everyone else has made about data. And so that doesn't make them very good friends of anyone else in a data organization.

Yeah, it's often for me when I'm working with new data analytics teams, I get them to focus on handoffs. So even when they're working, if they're not pipelining where they're handing across teams, they have T-skills within the team, so they can do all the work that needs to be done. They still hand off, right? You still hand off across people. And so, one of the focuses I get them to do is, how do you hand off, right? How do you describe what you need the next person to do in a way that they can understand it? And we know it's not a 100-page requirements document, right? But it's also not, I just need 5 columns, right? And one of them is called age. Some of the stuff I'm seeing now is this term feature engineer, right? Which really is, to me, it seems like a person who writes code that creates the feature flags and a bunch of columns to enable a data scientist to run the algorithms or the machine learning models. But what I struggle with again is now we have a handoff problem, right? So how does a person that needs data for their model explain the features they need, right, in a way that somebody else can write them? And prove that actually those features are the ones they want, and that there's not been bias in the rules that have been used to create those features that they haven't seen that affect their model. And therefore, it becomes easier for them to just create their own code, right? Because they're exploring the data, they're seeing features that may be important, may influence the models. They then create those features, they put them in the model, they see what the impact is. That closed loop happens amongst themselves. It's faster and and in theory safer for them, but it's not scalable, right? Now we have a person who's trying to bore the ocean. So what do you see? Do you see that feature engineering as being devolved outside of the science role?

This reminds me of a conversation I had with a Facebook data scientist maybe 3, 4 years ago at a meetup group when they gave a presentation about whatever crazy cool stuff that they work on. And afterwards I asked them about one of my boring day-to-day enterprise data scientist questions of, How do you get the data that you need? Because you have even more data than I am trying to go after, and I can't get people to give it to me, and you surely have all the data. But how do you know what to look for, and how do you make sure you've got the right stuff? And the guy looked at me like I was an alien, and he said, oh, we have an engineering team, and we just Slack them what we're after, and then, you know, 15 minutes later, they point us to an S3 bucket where that data is sitting. So I think so many of these new jobs and new ideas are generated out of companies with effectively unlimited resources to do this work, which is not the reality of 99% of everyone working in data. So that's one end of the spectrum is gargantuan companies with infinite resources. At another end is the scrappy individual who has to make it all work themselves, otherwise none of it will work. There's another interesting one, which is the company called Stitch Fix, who have a recommendation engine-driven clothing subscription service and a very impressive data team and a chief algorithms officer. And they believe the opposite of either of those two positions, which is that a data scientist should be able to acquire all the data that they need, experiment with models, productionize and maintain all those models, and of course, That means that their 200 data scientists are supported by 100 platform engineers to build all the things that make that possible. So I've not seen feature engineer as a job. It strikes me as an extraordinarily bad idea for traceability, for productivity. And also, I think the thing sometimes, whoever is pushing that, if they're a data scientist and they think that they're getting away with avoiding some hard work. If you've got other people choosing what data to put in the model and machine learning is increasingly automated, you're pretty soon going to be out of a job.

Yeah, do you see, you know, when we're working with teams, often get them to experiment with peer programming, right? And it's a really interesting experience every time because for a seasoned developer, a seasoned engineer, typically they'll come back and go, but that'll slow me down. Me staring at a screen with somebody else, just let me get my headphones on, let me bash the keyboard, and I'll get it done for you. But when they're open to experimenting with it and they do it for a while, typically what we get back is, oh, wow, actually, having somebody sit next to me made me think a little bit more about what I was doing and why, because we're talking and they're asking questions. It stops me going down rabbit holes. Normally I go down a little side lane for 3 or 4 hours and often the person I'm working with goes, well, maybe we don't need to go there or we just do this, right? So they get some value back by having a pair. And we all know that when we use that approach, the quality of the code that comes out is much better, right, for various reasons. Do you see that happening in the data science world, in the analytics world, where a data scientist and an engineer will actually sit together and work on the same code to create those features?

I think it's an extraordinarily good idea, and it's one that organizations are just engineered to be allergic to. Because this idea that you could get more value faster by having, to what the bean counter looks like, two people do one job, It's just not how any organization understands how it creates value. I know personally when I've been able to achieve that, and it is easier actually sitting physically side by side, it is tougher over a shared screen because then you've also got all the other distractions of having 5 monitors and notifications. But the times when I have worked side by side with another data scientist, or an analyst visualization type, or an engineer type, all those benefits have become clear to me. And one thing that I always wanted to do when I was leading teams and never got a chance to do, but I've seen a few customers push it this far in Dataiku, is to have those cross-functional teams working on our problem. Because one of the things that makes creating the time for peer programming difficult is that, well, because all of these data projects are such uncertain bets, and because we have to promise so many things to so many stakeholders, we spread ourselves thin. And so we're all working on too many things at once. And so we can't even find the time to do all this work together. And so what I always wanted to do was to say, was to reach a point where the thing to build was so well-defined that you would take an engineer and a data scientist and 2 analysts, and you would make them a team and say, solve that problem. And you have no other job while you're trying to solve that problem. And I think in that context, if you, you know, that version of agile would be revolutionary if we could ever get data teams to that level of clarity on what they're trying to build and that level of insulation from BAU and multiple priorities and waiting on data?

Yeah, it's interesting. I see the kind of— So one of the things I'm really passionate about is Agile's a bunch of patterns, right? And those patterns may or may not have value to the way you work. So while I like a lot of the Scrum stuff and the flow Flow stuff, to me, they're patterns. And so what I encourage teams to do is find out they've got a problem, right? And then find a pattern from the Agile world that may solve it, experiment with it, see if it fixes your problem, right? And then go on to the next one. So, you know, that's great, but often we need to start somewhere. So I will often encourage a team to start off with more a Scrum-centric way of working. So, you know, I articulate that, well, actually, Alex Shalloway articulates it better than me, and I use his terms as, you know, we are doing small batches. So we have a team that are dedicated to doing one thing, one data product for a period of time. And we're trying to reduce the cycle time for that batch, right, to 2 or 3 weeks. And ideally, we want to move to more flow, more lean-based ways of working with data because it's more natural for the problems we have to solve. But if we think about that, from a data point of view, we've started to see traction of Scrum being used in data teams, copy the Spotify term, a squad of people that are cross-functional and T-skilled that can collect the data, combine it, consume it, present it. But we don't tend to see that in the data science world, right? We tend to, again, to see the data science teams sit outside that team. And one of the things I, one of the reasons I think is coming back to that comment you made about knowing what you need to build. So when we have a team building a data product or an information product, we're getting better at understanding what it looks like, you know, what's the output? Oh, look, it's a dashboard and it's got these KPIs and, you know, we still haven't figured out what value we're going to get from it ideally, but, you know, we kind of know what it might be and we're getting good at reducing the scope, right? We're saying, well, look, it doesn't have to boil the ocean. It's not all the data. It's just a bit about a customer and a product and an order, right? And so we can reduce our scope down and get it within that batch. But from an analytics point of view, we're working on a hypothesis, right? It's a bigger problem. There is no known set of requirements. There's a theory, you know, a theory that I can grab all this data, and if I use some kind of models, it can recommend to the users what they should do next, you know, what movie they should watch next based on that. And that's a hypothesis. Like, we don't know whether the data supports that. We don't know whether we can create a model that does that. We don't know that that model actually has value and that user would find it useful or annoying. So we've got a whole lot of hypothesis testing. And for me, that's why having T-skilled teams in analytics and micro-batching the work to be done is a lot harder. What's your theory on that?

Yeah, completely with you. Every time I've tried to Scrumify data science work, you end up with a whole bunch of tickets or whatever your construct is that explore data, explore data again, fix data, get more data, explain data, It's so easy to generate a series of discrete steps that still lead to absolutely nowhere because the actual steps taken to test that hypothesis would typically actually be much simpler. What I think the deficiency is, is having organizations form hypotheses to begin with. Because the hypotheses— hypothesis is doubt, hypothesis is uncertain, hypothesis could be wrong, hypothesis can't be promised to be true. And so if you have data scientists and you set them out on these ambitious problems to solve, you have to accept that in many or most cases the answer will be there is no solution that's useful and feasible and and implementable. And so a big part of the work of data science, any science actually, is to disconfirm, to disprove hypotheses. And investing in being wrong is difficult for everyone. And I think over time, I guess what I've settled on is that you either need to split data scientists into two groups, one who actually prefers the world where they test hypotheses and are mostly wrong, and a different group who take the ones that have been somewhat validated and turn those into products. Or you have to have the same individuals be able to switch between those two kinds of work. Depending on the team size and the people that you have, one of those would be a good way to do. The way I've visualized that today is you need a small cog very rapidly turning to very quickly invalidate bad ideas. And everything that survives has to go on to a much larger, slower turning cog, still powered by the same little cog, but it turns much slower. And that's when the ideas that survived need to be implemented. And that's when it's mostly politics, getting other people to do stuff, enlisting engineers, making things robust. And so that view of the world that I have is a little bit inconsistent with the view that others have, which is that every data science experiment that you start should be built to production grade from the very beginning. At times I've believed that, but I just don't think that's a way to solve the problem you outlined, that you don't even know what is worth building out of data science in the first place to begin with.

Yeah, it's interesting because in the Agile pattern world, you know, we have this idea of a research spike, which is there's a high level of uncertainty. So there's no point going and actually getting the team to work on something until we have more clarity on what it might look like, what it might take, how long it might be, some hints that this is worth investing in. So, I definitely agree. I think that idea of bringing research spikes into analytics, where we have a hypothesis and it's a high-risk hypothesis. So, we need to do some early work up front to see whether it's got leaks. And we all say fail fast, but nobody believes it, right? Because as soon as you fail, everybody jumps on you. But those research spikes, most of them should fail because you're starting to explore things of high level of uncertainty. And then once it looks like, you know, there's some possibility that this thing could have value, that it could be done, then it goes into a second round, right? Which is, in my view, different people because it's a different mindset, right? Now I'm saying, okay, you know, we know this is possible, but it's still hard. How do we do it? And I'd probably add a third one, which I think is just turning up in the market, which is a reinvigoration of things that happened 10 or 20 years ago, as most things are. But it's this idea of analytical models that have been done time and time again in other places, so the patterns are known. So if I take the concept of a recommendation engine, that's a pretty well-known pattern now, right? So yes, the data is always different. We did an experiment where we, in our startup, we've got a data catalog. So a bunch of tiles that you can see the tables that are there and the data that exists. And I wanted to see how easy it would be to create a recommendation engine under the covers. So based on who viewed the data or who used the data and what you'd look like, it would recommend, you know, so effectively copying the model from Netflix, but for data. And so we did a research spike on that. We call them McSpikeys. We, we said, well, look, we thought this, this model was in the third cog, right? It was a proven model with lots of patterns, and we're on the Google stack. So there was a product called Vertex.ai, or Vertex AI, and it had a recommendation model for shopping carts, right? And it was just as much of an out-of-the-box model as you're ever going to get. So we did a mixed spiky on there, and it killed us, right? Because that model was so tailored The implementation of it was so tailored for user stick something in a shopping cart.

Let me guess, e-commerce website with product description text.

Yeah.

Maybe images with colors in them.

Yep, but think about it, right? For me to apply that model, well, for my team to, 'cause I didn't do it, it was like, well, we've got a tile, right? And that's just a product. And we know somebody clicked on it or viewed it or queried it, right? So that's just some kind of feedback of that it was valuable. And all the model said was, there's a thing and there's some feedback on how often that thing gets used and therefore you're going to look like that and I'm going to recommend it. But it didn't work for us, right? I mean, the key thing for us is, as part of McSpikey, we timeboxed it, right? We said we have this amount of time to prove it has value or it doesn't. And we went, no, right now it doesn't have value for us. So we binned it. So for me, even, you know, those third cogs, right? Those reusable ones, you still got to do the first little cog, right? If I take that thing and I apply it, is it going to work? What's the risk level? What's the value? And then, okay, we're going to invest in it. So, I think that's what we're going to see more and more of those third cogs. And then we're going to see people who specialize and take in those known patterns and analytics and applying them. And they'll become, because we've seen this before 20 years ago, industry models, right? Or domain models or those kinds of things. And people will start to apply them.

Those. We talked about that 5 years ago then, didn't we though, Shane?

Yeah, I think we talk about it on a regular basis, right?

It's Nirvana. One day someone's going to define the meta-churn model and customer churn model, uh, customer lifetime model. I think that, uh, you're right that the patterns on how to do it are clear, the underlying mathematical algorithms are clear, the— we have the computation now to do it all easily. What's different across the cases is what does a 0 mean, what does a 1 mean, what does a 999 mean, what does a— what does bad data mean. And that's, I think, once we can use AI to solve that problem, then we'll be cooking.

Yeah, so, so I think one of the things I'd always recommend people do is when they're talking to their analytics team or their data scientists, be very clear about which of those 3 cogs you're asking them to do.

Can they though? Who's this person talking to data scientists who is supposed to know which cog they are asking the data scientists to turn?

Well, let's go into that because you use the word Chief Algorithm Officer, right? And I was like, oh my God, I have heard this before. It started to come out And so, you know, it almost got me on my rant, which was the CIO, the Chief Information Officer, right? The key there was information. It was, it's not the CITO, right? It's not the Chief Infrastructure Technology Officer. It's about information. But because that role never delivered for whatever reason, we then came up with Chief Data Officer, right? And now we're going into Chief Algorithm Officer. And so, you know, if talking about roles, what the hell does a chief algorithm officer do?

That's a great question. There aren't that many of them. I think Stitch Fix, who I mentioned, was one of the first to have one. And one of the good reasons to have one at a company like Stitch Fix is that its whole business model is built around algorithms that select which clothes to stock. They built a bunch of clever human-in-the-loop stuff so that a human stylist would select clothes for people, but the recommendations would be generated by, by algorithms. So they put algorithms everywhere, and, and that's, um, that's what runs their business. And so if that's what runs their business That sounds like a perfectly reasonable job title to have. But if you're— that's not what runs your business, if what runs your business is machines and manufacturing or money or anything, anything a little older than, uh, after anything a little older than the 2010s, I think we were mentioning before, then I don't know if you need a chief algorithms officer. And so, that does raise that point of, if you and I both agree that there are 2 to 3 cogs, whose job it is to know which kind of work fits in which cog? I don't think the business, which is, you know, not a term that we should all love, but I don't think the, people out in the business who need problems solved should or could know which of the three cogs need spinning for which kinds of problems. I don't think data scientists are particularly good at knowing which one. They like to go down rabbit holes. So, something that I'm seeing emerge is analytics translator, data product manager, people who can actually straddle the world of what actually is going to solve some real problem in the world but who knows enough about the work that they can help the right people get on the right cogs and spin them.

Yeah, there's a term out there, it came out of Juan from data.world, and he talks about knowledge scientists, right, I think it is, which is that translator. And yeah, I love the idea, I hate the term. So I think what we'll end up seeing is analytics product managers or maybe algorithm product managers. Or algorithm product owners. And they will be that person that sits between the team that's doing and the executives that are funding to describe the trade-offs and make that call, that value call, right? So we see that with data products, with a data product owner. They're saying, I can invest the team's time in product A, I can invest the team's time in product B. We've done some initial work and we kind of understand potentially how big it is, how much effort maybe, you know, We understand what it might be used for so we can have some form of value. And I'm going to make the trade-off decision because somebody's got to. And I think when we talk about those three COGs, we're going to start seeing that as, you know, some form of algorithm product owner that goes, oh, this is actually COG 1, right? This is a hypothesis with a high level of risk. We need to timebox it. So they will make the call that, yep, it's worth it, but COG 1 it, team. And this is when you run out of time. Or they'll go COG 2, Okay, we have a little of surety, you know, this is good, let's go, let's invest in it, it's going to be a bit rocky. Or, you know, eventually cog 3, right? It's a proven pattern. Yeah, just let's grab, let's put it in, let's get the value and let's move on to one of the others. So, I mean, we have to come up with better terms than cog 1, cog 2, cog 3, because it sounds like Dr. Seuss to me, but I'm sure somebody out there will make up a term for classifying them.

It's good. And I think the missing link there for me is that product word, which we You know, you and I discussed it a lot 5-ish or more years ago, and I think it's great to have seen a little bit of maturity come to that term in data in general. And I think through that, it will and it should eventually impact this data science and this world of algorithms as well.

Yeah, I mean, we might almost see a term analytics as a product, but the problem is that starts off with the word us, so it's— Interesting with that CAO though, I'm assuming that person's sitting at the top table in that organization, right?

In that organization, yes. Yeah.

So for me, that's the key, right? Whether you call it a CDO or CMO or CAO or CIO, whatever, the person who's helping with data and algorithms is sitting at the top table. So they have a voice, they have money and people, to make things happen. Either they own the team or they can control what's done by the team that somebody else has owned. So they have the ability to execute, uh, and they are empowered to make the value call, right, on behalf of their other C-level, um, peers, right? So obviously they've got to talk to them and all that kind of stuff, but, you know, the CFO makes a decision typically of when, you know, finance needs a new finance system or more finance people or a new portal for procurement or whatever, right? And so the person that's got the C-level for data or analytics needs the power, the mana, to be able to do that as well, I think. And I think that's one of the things we look at when you look at a large organization about how data-driven they are, is who's at the top table with the data and analytics view.

It's a great proxy. It's a great proxy. Who's at the top table who's got that? And then also, given that that's almost never true, how far away from the top table is the person with that credible view of data? And I see a range of people in my work. I see people who should be that person who have almost no team and almost no influence. There's some kind of cheerleader. And then I see people who who do run a whole organization, but they're not high up the organization enough. And often that's just data is not actually that important to that organization. I think it's rare still to see a chief data officer, algorithms officer, information officer who is truly as influential accountable and trusted as those other C-level execs.

And it's quite interesting. So, if you put the 2010 cool kid lens on it, right, for companies that, you know, started in that genre and are data-driven, just thinking about it, I wonder how many of them it's because one of the co-founders is a data person. And they're at the top table by default, right? So they're driving the vision of the company to say data's more important than most other things, and that's why they behave that way. Where for companies before that time, you know, the hierarchical fiefdoms have already been created, and therefore there is nobody at the top table because that's not how the organization was created. That role didn't happen 20 years ago.

Yeah, agree completely. And I talk about it often with large enterprises, government departments, who keep saying that they want to be data-driven. And even CEOs say that, but they leave, you know, the pesky details of how to other people, and those people don't have a mandate, and then, you know, someone starts a data literacy program, and then a bunch of people learn how to read data, but they, they have nowhere to apply it, no time to apply it. And even if they did, what would change about the organization? And so I could— unless you've got a shining example at the top who is constantly, um, changing behavior and the direction of the company on the basis of data in a way that meets all those quality standards and rigorousness standards that we like, it's really hard to see where it comes from at an organizational level.

Yeah, and we see that same with Agile. So, outside of the data space, when we work with teams at the bottom and they start adopting a new Agile way of working, they're Most times they are successful, right, depending on what you describe as success. But when we see an organization try to do an Agile transformation where the goal is become Agile, not change the way we work, and Agile patterns might help us be a little bit better, and the CEO is not coming from that background, so they don't meet with their peers on a daily basis to check in on how they're going, and they don't plan together and all that kind of stuff. They're not agile, so therefore the organization never will be. And so I suppose it kind of comes back to, with data, that Tom Cruise movie Moneyball, I think it was, you know, show me the money. If you're in an organization where somebody at the C-level is yelling at you, show me the data, then you've got a chance of being data-driven. If they're not doing that, then it's hard, right? Because the organizational culture is not about data.

Just on that point of bottom-up versus top-down agile, what's your experience with SAFe?

The way I describe it is I am unsafe. So I try and keep an open mindset as all agile people should. I have a closed mindset when it comes to SAFe. I see it as a prescriptive methodology that doesn't help you describe a way of working. It gives you a structure that looks like your organization with slightly different slippers and names on them. Other tools.

I've had limited experience, but it does come up sometimes in my work where people ask us at Dataiku to explain how our product, which is just a data science and machine learning platform, will fit into a SAFe organizational architecture. And it can be quite challenging.

Yeah, well, as we know, we probably should talk about on another podcast, you know, my prediction is this year and next year you're going to be asked how it into data mesh just as much as SAFe. But going on to a different subject then. So you've talked a lot about what happens when the first data scientist turns up at a company, right? And I think one of the stories you talk about, you know, is data science turns up, there's no data. The answer should be data scientist leaves company until data turns up. In some of your other podcasts that you do, you also talk about this scenario that I love, which is data scientists turn up and they start hassling everybody in the organization to send them that spreadsheet. They start crowdsourcing the data that does exist and they're not putting it in a warehouse, they're not going and doing automated data collection, they're just getting somebody who has it to send it to them and, you know, number 8 wiring it for a while. One of the things I struggle with is that chicken and egg scenario, right? So, if we wait for the data, then we really don't know what the hypothesis is, right? We don't know what that small cog is, right? We're just going to go collect data. But if we have somebody come in and there's no data, then they have to have that engineering capability, right? They have to be able to grab whatever data is there and make it usable before they can go in and prove that small cog. So, you know, for me, I don't understand why the model isn't you hire a data scientist and an engineer and they sit together, a pod of two, squad of two, right? That's the de facto what it should be. And yes, you're going out and harvesting data that's never been harvested before in a semi-ad hoc way, but you try and automate as much as it makes sense on day one, right? You know, you're not doing a sprint zero where you're building an analytics platform out for 12 months, you know, grabbing a bit of data in a couple of days or weeks, building something that is semi-reusable, but not really. If it has value, you'll do more work on it and make it more automated. What's your view? What's happening out there? Is the pod of two something that's going to happen? Are we going to start doing, you know, you can't have a data scientist until there's a squad that does everything? Are we going to stay with data scientists don't turn up until the data warehouse or the modern data platform's in place.

I think, so when we talk within Dataiku to companies of all sizes about do they have an ambition to apply data science at scale and to have lots of different kinds of people do that, different skill levels do that, one of the most common objections to that kind of scaling or democratization of data science is, well, we first have to get our data lake perfect, and then we will think about that. So there's one common response which very much fits the pattern of get all the data, get it all right somehow. I mean, I've never seen perfect data, but it's a very nice dream to have. And then we'll think about interesting things to do with it. But we really need to— so at one end, there's that. At the other end, there's the, to hell with any reusability structure, we're just going to let a whole bunch of individuals loose. Truth, as always, is somewhere in between. What I see in the teams that I work with or advise or ultimately try to convince, there's still a huge amount of functional separation in those data teams. Maybe I'm in the UK now and I don't know, the teams are bigger and so there's more refuge in just talking to your data engineering colleagues. But it's just a tiny handful of at least our customers and our prospects who I've met who are doing something as radical as having an engineer and a data scientist work together in that way. Almost everyone is still running. I have a data engineering lead, a data platform lead, a machine learning engineering lead, a data science lead, some other people who might help those people talk together. But there still seems to be a preference, at least in this market, for different teams doing different work and sometimes using radically different tools to do that. And I understand where that comes from. It can't be The future?

Yeah, so it's interesting. So, I think scaling is hard, right? I don't think I know. One team of 5 people can rock it. As soon as you start to scale, we now have to specialize. Our natural reaction is often to specialize based on the process. We become a factory, you know, where the person puts the diode together and then the next person, group, team puts a diode onto the circuit board and the next team solder it. That factorization. Is kind of a known process. And I don't actually have a problem with that because that's a flow and lean-based process. We, from an agile point of view, we focus on different things though, right? We focus on how that diode's handed off to the team that put it on the circuit board. So in that case, we focus on how the engineering team hand off the data to the analytics team. And then how do we deal with the uncertainty? Because in a factory, we know where the diode is, and we know where it goes on the board. You're not randomly choosing a new place, whereas—

And we're making a million of them a day. Yep, yep. Of the same thing. Yeah.

And that's not what we tend to do with data and analytics at the moment. So the key thing for me is you focus on how you're scaling, how you're specializing a bunch of those teams, and then you focus on the friction between them and reducing it. But what often happens with large organizations, we're retrofitting it. We're trying to take these new ways of working and retrofit it to a bunch of people that already exist. So we start off with a scaling problem. We've got 20 people in our data team and 15 in our visualization team and 5 BAs in the BA team and 16 data scientists somewhere. And now we're trying to retrofit a new way of working with them, and that's hard. So my answer is treat it as COG1, it's a hypothesis. Grab a couple from each of those teams, put them together, tell them that actually their goal is to find a new way of working. And to do that, they have to actually build some analytical products. The analytical products themselves may work or fail. It doesn't matter. It's the way of working that they're experimenting with. That's the hypothesis. And then once that works, say, how do we apply this again and again? How do we scale it? That's what I recommend people do. People do. And yeah, done it with data teams, not so much with data teams that have analytics people in it. For some reason, we treat them as two different things. It's amazing how long, how quickly 50 minutes goes. So before we close it out, anything else top of your mind around agile and analytics?

It's funny, one of the presentations I gave at Optimal, I think had that title, Agile Analytics, and that was maybe 6 or 7 years ago now. I wheeled it out recently when someone on Slack at work asked, does anyone have any recommendations for how to organize analytics teams, data teams, data science teams according to Agile principles? And there's, uh, the thing that our prospects and our customers really want advice on is what they call operating models and organizational change and how do they become more data-driven, more AI-driven. Very few of them ask us about what we've been discussing today about organizing the work of the teams and how they actually do that, which is fascinating because how people do their work is to me the most interesting question. I think the technology has come a long way in 5, 10 years. A lot of things that were are actually technologically quite difficult in terms of handovers and other things, that friction is reducing. I think the teams that do crack those new ways of working are going to have some kind of edge and going to deliver more value. But it is interesting that this discussion we've had is not so much one that I have in the industry and the customers. That we have. I think that might be when you're working for a software vendor, you're talking to a kind of data leader who's making a procurement decision, and they're looking for some piece of a technological architectural puzzle. How you do the work day to day is— I don't know how much thought people are giving that.

Yeah, so I don't agree it's only software vendors that cause this problem or have this symptom. So, yeah, I look at it, I've got two examples that I can back that up with. So one is on, you know, I'm still side hustling, consulting with big companies to bootstrap our startup. And so, you know, I will do either an engagement where I go in and help their teams, or I'll do an engagement where I create a blueprint. Document, right? So, yeah, not a 500-page strategy that nobody ever used, but a smaller document that has the components that you need to get started. And one of those is the operating model. And to understand that, you know, I typically get the team that are currently doing some work in there to explain to me their data supply chain. You know, what are the steps you take from the beginning to the end? And then we have a conversation about how much of that needs to be centralized or decentralized based on the size of the organization, the way you want work. So that operating model is really, really important. You know, doing a side hustle at the moment where we're evaluating and implementing a massive modern data platform. And, you know, day one I said, first thing we need to do is understanding the operating model and the supply chain. And we haven't done that. So now we're putting in tools and technologies without understanding who's doing the work and how they're going to do it. That's high risk. That's called one. The second one is if you look at all the stuff that's published, even from the, you know, the 2010 cool kids, they publish their platforms, they publish their architectures, they publish their code, they really publish the way they work and their supply chain. And Spotify did it and we abused it, right? The market called it the Spotify model. McKinsey went and made it their bread and butter. And so Spotify are no longer sharing with us their way of working, which is gutting because they invested a lot of money in COG1 and hypothesis testing ways of working and hypothesis testing organizational structure of their data and analytics team and their supply chain, and then sharing it with us. And we did bad things to them. So they no longer share. But yeah, I'm with you, right? We don't talk about that. We talk about technology. But it's not just software vendors, it's everybody.

Yeah, it's interesting. Something you said taught me a long time ago was that whatever you're talking about in this data space, you can boil the work to be done down into 3 steps: get the data from somewhere, do some stuff to it, and put that result somewhere that matters. And people forget that, and they go off and buy some things, and they've never even looked at that supply chain, which is just a set of operations of get get data from somewhere, do some operation on it, even if it's an algorithm or some kind of fancy AI, it still fits that pattern. And there's both the supply chain end-to-end of those operations, and then within each of those operations, there's more getting of data, doing stuff, putting it somewhere. And a little bit of analysis of how people currently do that, how they might want to do it, goes a long way.

Yeah, and I'd probably nowadays, I think I'd say there's 5 steps, right? So that's the factory, right? That's doing all the work. I'd say there's a step before and a step after now. Step before is prioritization. You've got a bunch of constraints in terms of the people. How do you decide what work should be done now and what you're going to hold off on doing? And the last one at the end is proving the value was delivered. And the action was taken. So how do you know that that investment decision you made was worth it? And those two things need to come into that workflow now. And on that note, I think we should call it a day. So thanks for your time, Sean. It's been awesome talking to you as always. And I really enjoy actually talking to somebody that's got experience both working with analytics and hands-on and with Teams and somebody that's kind of had an agile mindset from day one, as far as I'm concerned. So it's been great talking to you.

Likewise, and yeah, we'll have that data mesh discussion one day. We will indeed.

And that, data magicians, was another Agile Data Podcast. If you'd like to learn more and apply an agile way of working to your data and analytics, head over to agiledata.io.