DataOps Patterns with Chris Bergh - Episode #59

Welcome to the Agile Data Podcast, where we talk about the merging of agile, product and data patterns in a simply magical way. Welcome to the Agile Data Podcast. I'm Shane Gibson.

And I'm Chris Berg from Data Kitchen. Glad to be here.

Hey, Chris, thanks for coming on the show. Hey, today I want to talk about DataOps. DataOps is a term that's been around for a while. And luckily enough, you were happy to come on the show and talk about the term DataOps because you wrote a white paper, which I would've called a book 'cause I was looking at it now, 133 pages around recipes for DataOps. Great paper around what it is and more importantly, things you can do to implement it. So before we rip into that subject, do you wanna give the audience a bit of a background about yourself?

Yeah, so I'm Chris, I'm a technical person, started my career in NASA and MIT writing code, writing algorithms. Then about 20 years ago, I started to get into the world of data and I managed a team that did kind of data and analytics for the healthcare industry. What we now call data engineers, we called ETL engineers, data scientists. And the team I managed, it was, I thought it would actually be pretty easy to do that because I was a big software guy and it's not a problem to run a data team. What I really found out, it was not easy at all. Things were breaking left and right. We never could satisfy our customers. We couldn't go fast enough. And how do you actually make your customer successful and be customer-focused while sort of running a factory of data? And all those things about innovating, focusing on customer success, and allowing people freedom to, uh, your team to be both productive and innovative at the same time. And that's the kind of challenge that I faced. And, and when we started this company a decade ago, that's the sort of principles at which I thought every data team is suffering from. And I think they, to a large extent, they still are suffering from.

I think what interests me is, as you say, you started your company 10 years ago to work in this space. I think you published the cookbook in 2019. So while some people think DataOps is a new buzzword, it's a term that's been around for a while. And it's a set of patterns that have been around for a long, long time, especially outside the data domain. As you said, there's a bunch of patterns from the software domain that we can adopt as data teams if they have value. So let's just do a little bit of anchoring. When you say the word DataOps, how would you briefly describe it to somebody if you're describing what you mean by that term?

It's a set of technical practices and management paradigms for data and analytic teams to drive customer success and be more productive.

Okay, and then for me, when I talk about DataOps, I tend to anchor back towards the technical practices from DevOps. So that's kind of where I saw it come and how I describe it. The idea is about you build it, you deploy it, you break it, you fix it. The idea is around automating anything that can be automated, that has value to be automated. And the idea of treat everything as cattle. Ideally, we can get rid of it and rebuild it without a human being involved if possible. Whereas when I look at your definition of DataOps and all the writing you've done, it seems much more akin to what I call the Agile Data Way of Working because you're bringing in team design, you're bringing in process patterns, you're bringing in team working, you're bringing in lean patterns. So you're bringing in a bunch of patterns from many different domains and then tailoring them for data teams, data and analytics teams to do it. Would that be fair? Would that be a fair way of how you look at it?

I think that's fair because I got really burned talking, you know, I'm a software guy, so talking about DevOps for data and analytics. And all those words that you just said, I've experienced in building software and managing software teams, right? And the success in that endeavor. And so whether you do automation or deployment or CI/CD, I think those patterns really apply to data and analytics, but they're sort of necessary and not sufficient. And they're not sufficient in two ways. One is that I think there's a set of ideas that come from lean manufacturing because the sort of left-to-right process of integrating data and producing insight is very much an assembly line. And that process has things like statistical process control, data quality testing that are unique. Those lean ideas need to be adapted into data and analytics. And then the really, the other pattern is helping teams get less focused on technology and more focused on their customer. When I talk about it, I try to talk about the end results. It's about productivity. It's about making your customers more successful instead of let's do more automation, which is a means to get there. And so I think they're both true because I think you have to look at it in different ways. Like you, I think the patterns here are not new. I look at it in a very abstract way. Like we have a set of people who are working on a technically complicated thing and how you manage that team or how you run that team, it's very different when we're all working on a shared thing than perhaps politics or perhaps running a school. Those are very different management paradigms. Whether it's a factory that's the shared technically complicated thing, or a software, or a data and analytics pipeline, or set of tools, or warehouse, those management principles all apply. Having us being safe, like saying, "This doesn't make any sense." Trying to reduce complexity and be able to reuse things. Trying to manage from metrics and measurement. All those principles really apply, and you can kind of see the same reflection of those principles in an assembly line or a data and analytics. And so I don't think any of this stuff is particularly new. I think Deming applies. I think the DevOps movement applies. I just think we're, if anything, we're just trying to take these principles and say, look, they have a unique instantiation in data and analytics, but the ideas are old. The sort of build it and they'll come, or organizations where I do one piece and I don't really focus on value delivery to my customer. I think those things are anti-patterns that we should get rid of, but I do think they are pretty prevalent in organizations today. I still think we have a need for, whether you call it DataOps or Agile principles or whatever, whatever the term you use, I think data and analytics teams need it now more than ever.

Yeah, and I agree, right? There's a bunch of reasonable set of patterns that have value in a certain context. And for the next hour, we're going to talk about DataOps, right? Because that's the term you use for coalescing those I agree with you around data factories. The more and more I looked at it, I bounced for many years between teams that craft bespoke Tejan products and a factory that just goes through a series of steps to pump data. And I struggled with that because while I wanted automation, that left-to-right behavior for data, what I saw is a whole lot of complexity, which meant that that process never worked. And so, I ended up in my head decoupling the the process of manufacturing that data versus the teams and the behaviors they use to manufacture it. And often I can go back. So, you know, when you're talking about teams and there was a flavor of self-organizing, self-managing in there. If I think about a lean manufacturing line, there are buttons where when a person's working a machine and they see there's a problem with the line, they push the button and the line stops. That's their self-empowered right to say, something's broken, we need to stop and all coalesce and fix it now before we carry on manufacturing. Often we don't see that in a data team. Often we don't see the ability for one team member to say, hey, the actual process we're using is broken. Let's stop and fix it. I mean, we talk about Scrum sometimes and we talk about agile and we talk about retrospectives, but that self-empowerment of each team member to actually stop the line, fix the process together, and then start the line is often non-existent. Is that what you see?

Yeah, I think there is a marked ostrich phenomena, where people want to hide from the problems that they have because they're afraid of looking bad. And so, whether it's, "I've got poor data," or, "I put some code into production," or, "I've got some code in development that's not working," all these cases are patterns where it's important to surface problems and have a safe way to say, "Hey, here's a problem, let's fix it." and raise it instead of trying to not notice things. And so a lot of what happens in data and analytic teams is quality itself is pushed off on the customer. They focus on the technical issues and value is focused off on the customer, right? It's like, oh, I gave you some stuff, you figure it out if it's useful, you figure out if it's quality. And they're sort of largely mechanical in that way. And I think that's a really bad idea because then you end up building a lot of things that aren't used. And that's just a waste of your time. And I don't know, maybe people are happy with that, but I've never been. I've always wanted to be part of making a difference in the world. And I think data and analytic teams can with data, but they can't when they just sort of vomit up lots of data of poor quality and sort of go home at 4 and don't really care if it's used. And so I think that it's a better way to work to focus on your customers and trying to find that they trust the data, give them what they need, and then try to focus your team on how to maximize making your customer successful. It's just a much happier place to work because you really know that you're having an impression on the world. And I think the challenge is a lot of teams are beaten down because they don't feel successful. So they feel like they're blamed for everything. They have this sort of shit sandwich view of themselves where like they get crappy data, people complain, and they're the middle. And that causes them to sort of turtle up and not want to deliver value. And that's not unreasonable, but it's still not an effective way to work. And a lot of teams who don't do that, they are really effective. I think that sort of turtling behavior in and of itself is not great. It psychologically makes sense. I don't know, do you see that? Where teams are kind of, I'm getting beat up by both sides, I'm just going to focus on my task list and that's it.

Yeah, definitely still. You know, you wrote the cookbook that I'm looking at in 2019. You know, that's 5, 6 years ago and there's a bunch of challenges there that we'll go through in a minute. And I don't think it's changed. In fact, I think some of the challenges you called out then, like return on investment, have become a bigger problem, especially with the financial downturn, the lack of value delivered by data teams, or the perceived lack of value has caused for lots of data teams to be made redundant. I go back to that idea of a factory, and if I think about car manufacturer, they focus on optimizing the factory and the line because somebody's already done the work upfront to see where the value is. Here's a design for a car that we know is valuable. Build the car, push the car out. You don't get to see how people drive it. You don't get to see how it improves their lives. So your job is to focus on that factory, whereas in data, We effectively want to push left and pick up some of that product thinking. We want to work with the customers to understand what value they need from that data. So we've become the product designers. And then ideally we want to watch and observe how they use it at the end to deliver that value, to see how we can iterate it. Because the benefit we have is we are not building a one and done object like a car that we can't iterate in the next whatever, right? Next sprint iteration. We can go back and actually change it like software developers and product teams do. So, it's this balance between automation and factorization and the ability to understand the customer and iterate with them and experiment to solve their business problems. So again, I often go back to analogies around food. And so, Data Kitchen, I've always said I'm a bit grumpy that you coined Data Kitchen well before I got there, because it was like—

Well, yeah, I mean, the terms are there, right? There's a lot of data teams. If I could kind of riff on your idea of If you think about the value stream that goes in data, right, one of them is best expressed as a factory, as you said, the left-to-right movement of data in production. And thinking of it as a factory that makes cars, the manufacturer of insight, all those lessons and stopping the line, those are really good things. But if you look at the lean and Deming-like principles that were applied to software, it's more about the development one, perpendicular to that factory. I want to take a piece of it and change it. Really quickly, and I wanna do that with low risk and fast. And that's a different value stream. And if you look at the DevOps literature, when they focus on value stream management or the DORA metrics in software, they're really focused on that perpendicular. Can I change my change rate, my change break rate? Those are really important things, but data teams are both manufacturing teams and software teams. That's what makes it interesting. And we've always liked the T diagram rather opposed to the infinity diagram that emphasizes those two linked value streams because you have to do both. You don't know what your customers want. You have to give them something that they can see and touch quickly and iterate upon it. Start with something in paper and keep working towards it, right? Information Product Canvas, your innovation, I think that's really great. But on the other hand, once you get something that they use, you've got to make sure that it runs and you want to have a low-cost, low-effort way of running it. You want to have the system tell you if there's a problem, that there's, you know, I got some bad data or the server is down. And all those things are true. And that's what makes it interesting. That's why I find it attractive is that combination of the two value streams, the manufacturing value stream and the construction value stream in DevOps, the software part and the car manufacturing part, because it's both.

right?

You have to do both. So the focus on metrics and how you run teams, that's what makes data and analytics, I think, a challenging field. And also the weight of data is different, right? When I had spent 15 years building a lot of code and running software teams, what I used to think as a software engineer, data is just, eh. But data has weight, it has mass, it has meaning. You have to discover requirements in it, right? You have to interact with the data to learn. Whereas in software, it's like data's an output, but in data and analytics, it's both an output and an input. There's this just interesting parallels in my world of having lived in both domains for so long that I think both apply, manufacturing and software, they both apply to data and analytics. One of the reasons I think I like Data Kitchen as a name, because if you look at a good restaurant, they've got to do two good things. They've got meals that are on the menu every night. That they want to be consistent and high quality, and they want to control costs, and they don't want the master chef to make the same meal 15 times in a row. They want to have someone less skilled take it over and deliver the same quality. However, a good restaurant also innovates and has new items on the menu every, every week or every day. And so you need time to do both. You need to run your restaurant with a very consistent way of producing repeatable dishes you want to foist that work onto people who aren't your key stars, and then you want your key stars to be able to go off and create meals, get feedback from the customer, keep iterating. And it's the same thing with data. It's the same thing. You want to be able to hire a good 22-year-old and have them take over stuff.

And I love that analogy because I can take it a couple of steps further, right? So the example I use is if you own a restaurant and the tomatoes get delivered and they're constantly rotten, As a chef, you're not going to go, "Oh yeah, those tomatoes are a bit rotten. Let me just cut off the rotten bits and I'll carry on." You're going to go back to that supplier and tell them to give you good quality tomatoes, or you're going to go find another supplier. Now, in the data world, we probably can't find another supplier. We can't go and grab that data from anybody else, but we can push back and say, "Stop giving us rotten data." Whereas what data teams tend to do is we tend to try and fix it on our side in the kitchen, and that is inefficient. The other thing is the feedback loop in a restaurant is visible. It is a Kanban board. Because if there's nobody in your restaurant, you have a problem. You are not adding value. Maybe wrong food, bad food, wrong price. There's a whole lot of reasons, but you get that visual feedback that actually the product you're producing is not valuable. Within the data teams, we often don't get that. We're obviously behind another team or another person. I still see the, manufacturing, software, and then the product patterns starting to come in now to fix that problem around discovery, understanding what people actually want, and then improving delivery of value. So I'm 100% with you there. The kitchen analogy just works. And it, like you said, it gives us that balance between teams that just need to do the same repeatable thing day in, day out, and the craft that comes with our domain to solve unique problems for a business. Each time.

Yeah, and I've come at it both from a technologist, but I think I've really come at these principles from a manager's perspective. When Agile started, I'd spent about 10 years as a developer. I was first starting to manage bigger teams, and I look at it really from a leadership perspective, and it really hit home when I was managing data and analytic teams, because the key insight I took was after reading Deming saying, look, when you have problems, it's mainly the process people work in and not the person. And like Deming says, 95% of the time. And so whatever percentage is, it's a high percentage of the time. And as a leader, you own the processes that your teams work in. You get some bad tomatoes from your supplier, you pass them through and the dishes are shitty at the end, it's your fault, right? And it doesn't matter, you can't, it doesn't, oh, I got, my bad supplier gave me, it's like, no, it's your fault as a leader. You own the result. Not the person who cut it up, not the supplier. It's like you own it. And so you have to fix the process. The right thing to do is to talk to the supplier. Maybe the right thing is to change your menu or cut the bad parts off. Like, it's very context-specific. I did some mistakes when I was a young leader. I fired some people because I thought it was their fault. And that I regret, honestly, because it wasn't their fault. I didn't fix my team. I didn't fix their work environment. That's like a really key to me leadership idea, that you own the processes your teams work in. And often when errors are made or results aren't delivered, it's because you haven't set them up to be successful. And that's a really hard thing for leaders to hear. And it's a really— it's like, shit, it's my fault and I gotta work on this. And so it's much easier to blame the bad tomatoes than to realize I haven't built a system that can handle bad tomatoes, and I have to figure out how to do that.

As data people, we've got to realize that our default behavior is to blame the tomatoes and blame the customer for ordering the wrong meal or not telling us what meal they wanted. It's like, "Ah, well, we gave you what you asked for." "Yeah, but you didn't solve my problem." And, "Oh, it's not my fault. I got shitty data from the CRM." That's a default behavior for a lot of teams.

Yeah, and I guess I'm I'm really sick of, this is gonna sound terrible, the whiny little bitches who blame other people. I'm really tired of it. Especially the people who lead teams. I can, you know, a person who's working in a team who complains 'cause their boss is, you know, shoveling stuff to 'em, I have sensitivity to, but someone who's a director or a VP who's blaming the customer and blaming the supplier, I really, they are just whiny bitches 'cause it's their fault and they're not trying to fix it. I hope there's a special rung in hell for middle managers like that.

But then I often, because I coach teams, I get frustrated at teams who say that's just the way it is, they can't change it. And I call bullshit because when you're in an ineffective organization, actually, you have the ability to make more change because the belts and braces that an efficient organization has don't exist. So, you can actually go and do pretty much anything you want. And if you deliver value, you'll probably get away with it, right? And I do realize that people often need the job and the money, and there is a risk profile, but they can still change small things without having to ask permission.

It's creative act of managing. Often finding the real problem is the hardest part, and then finding the first problem to get is the hardest part. That's actually really interesting management behavior. Now, if I think over my career, the things that I'm proud of the most are when things have been the broken the most and I've had an effect on them. That's what I'll look back on, not when things were going well and everyone was firing. It's like, wow, this is really broke, and we we got together and fixed it. And that communal aspect makes you feel better as a leader. And I think you got to start with admitting that there's problems. And, you know, blaming suppliers, blaming your team, blaming your customer— I've done that, right? I've done it. I'm embarrassed to say I've done it. I've been that whiny little bitch, and I just don't like it. And I guess I think leaders should step up and realize that they have a lot of ability to fix things like mindset and that when they do it, it's an improvement. There is some indication it's going much slower than I thought. I actually see the individual contributors, the individual data team members, data engineers, they're actually stepping up and using version control, putting testing in, automating, refactoring, trying to reflect on their work and improve. The effect that's happened has been much more bottoms up than top down, at least in my view. And I think the managers are laggards. Not all are, but managers are laggards in this.

I see some, what I would call high-performing teams. And if I look at part of their success is normally empowering leaders, people that set the vision, get out of the way, and then support the team and remove any impediments or roadblocks that they can see. If we look again at manufacturing, we'd see this term walk the floor. You talk to people in those organizations that are leaders, and they will spend an inordinate amount of time actually on the manufacturing line, walking the floor, observing the process to try and determine where something may need to change. And within the data world, I don't see that. I don't see the leaders sitting next to the team and watching the process. And they should because it's that ability to be disconnected, that ability to be able to sit back and observe the system that people are operating in that actually is really powerful. And being one of the data leaders, you're one of the few that has the time and the ability and the permission to step back outside the work being done to watch the work being done. So, again, one of the patterns, strongly recommend if you're a leader, just go and sit with the team and watch how they work. And draw it, get a piece of paper and do what I call nodes and links. Draw a circle for when work starts, determine who's doing the work, do another circle when they hand off, determine who they hand off to, and then look at how they hand off. So, when one team member passes work in the factories, the next team member, What's the thing that they hand off for the next person to do their job? And how efficient is it? How well described? How much rework does the next person have to do? They're all valuable principles that come out of that lean manufacturing that we can apply to data.

You know, there's kind of two views again of walking the floor. Like I think the sort of production process, we've got some new data, where does it go? And who uses it? And where does it end up? And why are people complaining? There's often no knowledge of that. It's all this sort of black box. Maybe they know where it came in, but then they don't know what teams do with it. They don't know where it ports it, who's extracting it. It's very confusing, right? The journey that data takes in production. And then, as you said, the process to actually put things into production, the development environments, the handoffs that happen, the quality checks. And because teams are pretty vulcanized in that, it's often fragmented. This team takes raw data and ingests it and fixes it up. And they have their own development process and their own production process. Okay, now another team of analytic engineers takes it the next step. And then the reporting team takes it. And they have 3 different processes to put things into production from development to production. And then they have 3 different steps to actually operationalize it and run the data in Data Factory. It's crazy. And no one knows. And so when something goes wrong, you end up with, this sort of intersection of Conway's Law and pass the buck. It's, and so it's like walk the floor, just draw the floor, just know where you are so you can talk. It's amazing how many people get in positions, especially with the sort of techno fetishism that goes on in data with the new buzzwords and the new tools. You have vestige eras of tools scattered around. You have late 1990s, early 20th. You have modern data stack, you have cloud, you have this. Oftentimes you just know where it is because it's which version of the data stack and the team who built it got frustrated and left. And now they're onto the newest stack. And so it's because some of us, I think, are chasing resume building as opposed to value delivery.

I agree. Some teams are formed so that the leader gets their next job, but we won't go there. Actually, I just, again, I was going back to kitchens and cooking and that 3-step process you talk about, the ingestion, the engineering, and then the reporting. It's kind of like having 3 restaurants. So rather than having 3 stations in a kitchen where somebody's doing the initial prep and then it goes over to the next person and the next person that then serves it or, you know, plates it. And they're all talking to each other if you ever watch a kitchen. There's instant feedback, there's quality control, you know, if you pass your part to the next station and it's not up to spec, you're going to get some feedback. Whereas in data, what we tend to do is create 3 kitchens, 3 restaurants, and there's an Uber driver that picks up, you know, half-cooked meal and buses back. In fact, I was watching a TV show the other night and they We had a thing called Relay Cooking. So it was one of those experienced chefs having a competition against each other. And so each chef had 40 minutes and there's 3 chefs in each team. And so the first chef gets given a set of special ingredients and has to plan what goes out in 3 hours. They get 40 minutes, but when the next chef comes on, they get 45 seconds to explain what the rules are, what they've done, and what their vision was. And then they leave the kitchen. And so then it happens 2 more times. That's the data team. But it gets worse. I was just thinking, you know, you talked about ingestion, engineering, and reporting. We keep forgetting about the fourth team, the business user of Excel. Because what happens is we deliver this beautifully crafted meal, 5-star meal to their table. And then what they do is they decompose it down into a bunch of its raw ingredients and they make a completely different meal because that's what the majority of people do with our data with the old export to Excel.

Yeah, that's so right. It's so right that there's like, there's 3 restaurants with Uber drivers and then a wannabe chef at the end who's reconstituting it. And it's such a communication challenge and such a feedback challenge. And that's why these processes that people work in are so important to get right and to think on and improve, because that's, that's the fundamental challenge as one sort of lives in these worlds where you've got 4 teams and the vice president of marketing or sales who wants the insight is scratching their head why it's taken 30 people and 3 months to answer his business question. And why are you doing all this? And it, they're frustrated and what's going to happen. And you've hinted at it is eventually George is going to start firing people. Right. And the good times they hire consultants and you get to keep your job and the bad time. And you can already see how hard it is in some fields to get jobs. And it's the threat of AI also. That's whether it's a real threat or not, that you can produce work faster. And we could have a whole conversation on that. But it does allow more people to get into the field, but it also gives a reason for organizations to reduce headcount. When you have this 4-part, 4-kitchen disharmony of producing food for your customer, and by the time it gets to them, it's cold and stale and looks weird, and they're like, this isn't what I ordered at all. Because that's the case, and you don't know what your customer wants, right? And you got to put a plate of food in front of them before they know if they like it.

That's the thing that, you know, whether AI delivers or not, whether it is a hype cycle or it is something that is valuable and changes the game, the threat to the data teams is it can deliver the same bad meal that we often deliver much faster and with a lot less salary than what we deliver. So that's the problem we have to go against. So let's just look at some of those challenges that you documented in 2019, cuz I'm intrigued about what's changed. So one of the challenges was changing requirements. And the DataOps approach that you described is reduce cycle time. Don't leave a big period between asking what the requirements are and delivering something. Deliver something smaller and faster and get some feedback. Has that changed? I mean, I don't think organizations are ever going to have fixed requirements that never change because organizations are dynamic. What are you seeing? Are you seeing changing requirements no longer being a problem?

It is as it always has been. People want answers, right? And they don't have time. And they're going to either use data or not, or use their intuition, right? And if you look at how leaders process information, they're using social cues, they're using trends, they're using force of personality, and they use data too. And so I think we're always in this struggle that leaders will make decisions, and the question is, are they going to do it informed by data, which increases your likelihood of success, or informed by something else? And I think we still have very powerful leaders who make decisions not based on facts. We need to work towards that because in the end, if you don't make facts, they have this nasty way of coming back at you. Right. And so you want your leaders to be emboldened with the best information possible so they can make the best decision possible. So, and to do that, you've got to get them the right information in a way that makes sense to them. And that's an iterative process. I started doing analytics kind of full-time in 2005. So it's 20 years now. And I had another sort of 15 years of software delivery where I was working with mainly air traffic controllers trying to develop AI software to help them land planes better. That was a very iterative process. And so I've been lucky in my career that I've always worked in an iterative way because the build a large document with a lot of descriptions and then spend months building it, it fails in software so many times and it fails in data so many times. The only way to do it is to put something in front of them, get it 70% right. Learn from it. It's really about maximizing what you don't have to— is the real key here, because you think they want 10 things. Inevitably they want 4, but there's some other 6 things that you thought they needed that they didn't. And it becomes down to waste. It's a waste minimization because the more you think they need, the more that you give them that they don't need, you just wasted your team's time. And so forcing feedback, it helps your team learn. It helps you cut waste. It helps you actually meet requirements. And all those things, I think, are, it's just the way of the world. It's not that your customers are ignorant or they're mean, or, you know, you guys, we don't socially get along. They went out and drank while you pounded the calculus books in college and you're jealous of that. It's none of that stuff. It's just the way the world is, right? We just have different domains and let's work together. I don't think AI, I don't think anything's going to change. That's always the case, right? And if anything, more automation, more tools can help us iterate faster if we work in a system that does it. Just the speed of creating new code is not the problem, right? Because most of the problems are in waste, not in creation. If you can create code 10 times faster, it actually doesn't help the waste problem that much, right? Because if 70% of your things aren't really being used or not needed at the end, that 30% that you've done faster, so what?

I think people get the idea of the small cycle times, sometimes they get it wrong. I was talking to somebody the other day, and I haven't heard this one for a while. I think it has now become an outlier, at least I hope it has. But they were talking about a requirements gathering sprint, you know, 2 weeks to gather requirements. And my point was, that's waste. Because you're going to gather so much in 2 weeks, if you do it well. If you do it badly, you're going to gather nothing. But let's assume you do it well. You're going to gather so much in 2 weeks that you're not going to deliver. Any of it. A lot of it's not going to get delivered, so you're spending this waste. And I think the other thing people forget is the cognition level of our customers, of our users, of our stakeholders. Because if we spend too long building more stuff, then when they first see it, it's too complex, there's too many things it does, and they have to then determine which of those things solved the real problem they had, the real requirements versus the other things that got added on. I'm with you. I go back to reduced cycle time. It's a forcing function to reduce waste. It's forcing us to put something back in front of the stakeholder early, and then we get feedback. So, the key thing about it is the feedback loop is where the value is, not push it out and then move on to the next thing.

There's also the myth of overgeneralizing. If I just have a representation that can handle all their needs, it's a report with lots of pull-down tabs or the right schema format. Then I can answer 10 questions that they have in the future. It's that trap of deferred value. If I do something more complete now, I'm saving myself time later. It is true in some cases, but it's more often not that you're overgeneralizing and you're doing too much and you're wasting.

And I struggle with that one. The data collection process was one that I struggle with. So, there's two kinds of patterns I've seen. The pattern of, if I'm grabbing the data from the source system and so let's say I've got a layered data architecture, so I have some form of what we call history, but you know, data take landing zone staging bronze, I think, in, in Medallion. I'm just gonna land it and it's gonna persist there for a long time. My natural reaction when I'm touching a source system is to bring as many attributes that are easy to bring over that make sense. And the reason is I know the amount of effort it takes me to go back and collect more data. Even though it should be automated and a solved problem, it is typically one of the more effort-intensive tasks. So I'll go, I need these fields to solve this business problem, but these other fields, you know, attributes of customer, I'm gonna bring them across. And so when you look at stuff that I've designed and built, you'll see we run a hybrid form of Data Vault. So we have details about concepts. So you will see my customer detail table being quite broad. Whereas my co-founder Nigel is an engineer and he only brings across the fields he needs. 'cause he sees every other field has been waste. Now, there is a cost of maintenance and a cost of maintaining those extra fields and we're not using them. So cost of storage, it is low, but there is a cost, but it's more the cost of change when they change in the source system. We now care 'cause we have it. If I've brought it across, if we haven't used it, but I still struggle with that one because I look at the cost of acquisition and the time to market of me being able to reuse those fields quickly when I need them versus Nigel's lens, which is around optimization and reduction of waste. So therefore, anything we hold that isn't being used is waste. So, again, it's just balances, and you just gotta decide which pattern, as a team, you're gonna subscribe to, because ideally, you only wanna use one consistently. You don't wanna bounce depending on who it is.

I think it's all about balance, and between taking 10 attributes or the whole table, it's balance and context that matters, and having the discussion that matters, because both can work, and both are right, And so a lot of teams get caught in this sort of fear and heroism. I don't want to do anything because I'm afraid to break things, or I'm going to do everything because I want praise from my customer. And the reality is you need a little bit of heroism sometimes. You need a little bit of fear sometimes. And most of the time you need a proper balance between the two. And so it's the balance between only pulling what's needed and pulling everything. Well, the point is you've got to talk about it. It's not about one or the other. It's just about saying, well, I'm going to pull the whole table because I can get a SELECT and there's a whole bunch of cases in there. And so I just think it's not a holy war. It's about balance and between agility and fragility, finding the way to make things work. And I think that we should, as teams, continually do that, because it's so often the case as engineers, because I want to build the thing that I think they need. I want to build the more general thing. I think that they're going to need this in the future. And then will they need it? Is it waste? That's such a key tension. And in a lot of cases, I think we need more people like your partner who are just saying, let's do less. Let's focus on minimizing waste. And if we had more people like that, because I think the data world is full of people who are like, let's pull it all over and let's have the big Hadoop cluster or big Spark cluster with everything. And then I'm going to vomit up, you know, lots of data to lots of people and magic's going to happen. It's the Field of Dreams, build it, they'll come. And that's 90%. And I think we need more people pushing back saying, let's just do a little. And then the reality is at the end, you're going to have constant tensions between those. And if you have a good co-founder and a good coworker, you're going to be able to laugh about it over beers and say, I got that right. I didn't get it right here. And we'll work towards it. And that's fun, right? I mean, it's the human process. I would think we just have to think, stop believing in our own prowess and ability to overgeneralize the world and really focus more on waste reduction. In general as an industry, because a lot of times what comes with the overgeneralization is overpurchasing, overbuying, overengineering. And there's a lot of waste in software spend. There's a lot of waste in systems. I've met CDOs that are in the process of building the data architecture for a $30 billion company. And they're trying to gather requirements off of it. And I know when that happens, run. It never works. It never has worked. It never will work. Right? But there are still people out there trying to define the data architecture for their entire organization. And so that's obviously overgeneral. Can you start off with one team in one case and iterate your way to get some value? I think that's less waste. And then should you bring one table or a whole table or a part? I think tensions are good in organizations and discussion of it and compromise and working it through. And yes, it would be better if you had a standard, but okay, maybe one person brings over 3 attributes, the other person brings 30, you run for a while, you realize you should focus more on 3 attributes because you end up getting more done. And everybody learns. I love the tensions. I love the discussion because that's the essence of what reflecting on your work's about. And as a team owning your own process, and as a leader, I don't like to make those decisions. I actually like the team. You know, you got a person focused on waste reduction, their intention, fantastic. Let's work and try it one way, try it the other, and we'll work our way through. And that makes a good team because then they own it. They own that decision and not the manager.

I agree. To me, it's about patterns that have value given a context. And if we can describe what they are, then most people will use them because it makes their job easy. But they're not rules. They're not immutable. If our core pattern for the team is bring as few attributes over as possible, like you said, if somebody brings over 3, somebody brings over 5, it's within that pattern. And if there's a reason why we have to bring it all over at the first time because the context says that's the best way of doing it, then that's okay. It's just a different pattern applied to a different context. The other one that you raised is quite interesting. We've lost the art of data modeling to a degree. And there are a bunch of people I know who are expert data modelers. So they can sit in a session with a bunch of stakeholders, they can somehow conceptually model that organization in their head, they can go away and they can very quickly draw a conceptual data model for the organization that is highly accurate, but they are experts. They're like master chefs who can just grab a bunch of ingredients and make the most amazing meal, and I can't do that. So we've got to realize the rest of the team that you get normally can't do that. So we need processes or patterns that help them do it in smaller ways, learn more, get more feedback. And then the last one is going back to the why. Okay, so I'm grabbing all those fields and bringing it across, but that's waste. Why is it waste? Well, if the source system changes something, we now have work to do on a field we're not using. Okay, so one pattern is don't bring it across, but another one is, well, could we automate it? Is there a way that we could actually adapt to that change in the source system for that field without a human being involved? Yeah, can I remove that waste of human effort? And if we can do that, then the cost of me bringing 10 fields over is less because we've solved most of the problems. Now, we won't solve every use case. There will always be exceptions where something happens where the cost of automating it is way higher than the value of automating it. But again, that's the team. It's up to the team to decide where the value is, where the most valuable time for them is to build their factory, their processes, and the data for the customer.

That's what makes it fun to be, because I think a lot of teams are disempowered, unhappy. And if you get as a team to make those decisions and own them. It's fun to learn and watch your team evolve. It sounds like your co-founder and mine are the same. I tend to be bigger picture. I overgeneralize. My co-founder's like, let's just do these 3 things. Let's get something done. And we have the same conversation almost every other week. We had one last week and it's the same thing. And he's going to keep saying the same thing. I'm going to keep saying the same thing. And sometimes one of us is right. I'm becoming a lot more like him. He's becoming a little more like me and you get better. As a team in making those decisions, because there are cases where maybe you can conceptualize the business in 6 dimensions and you're right. And if you build out that perfect schema, you're going to be good. And other cases where you're going to get it so wrong that you're going to cause a lot of waste. And it's always that tension between overgenerality and reducing waste that I think we have to grapple with, but you have to have it. I'd like a lot more teams to start with less and because we, believe in generalization too much as an organization, and we believe in our power of abstraction to relieve us of future work. And I really think that belief in the relief of future work almost always is a failure. And so make sure you do what you do today well and just do that, and then you're going to be able to move on and work from there. And I think it's a hard discipline for people because of egotism, because of the way the market talks about tools. A lot of vendors will say, "You put my tool in, you get all this stuff for free." Anytime somebody's selling you something for free, run.

Right?

So nothing's free. And the same thing with your architectures or your decisions on pulling data over. I like the tensions in organizations. I like empowering teams. I like having customers do it, but it is hard when teams have an existing system. In the real world, this all sounds great, but in the real world, a lot of people have legacy systems that they're still in charge of that they don't really know. Somebody else built it and left, so they're stuck with it. They have bosses who are in the blame game and are— love to blame them. They work in blame organizations. How would you advise people? I have a tough time with that. When you're in a blame organization and you have a bunch of legacy stuff, how do you advise people to make progress with DataOps when they're set up to fail right from the start?

I encourage them to hide what they're doing, which is horrible, right? But that's the pattern that works in those organizations. When people tell me, ah, they're not allowed to spend time fixing the processes that they're creating, I go, how accurate's your estimation process? And they go, we do an estimate and then the manager doubles it and then the next person triples it. I go, okay, so there's no accountability now for what you say versus what you do. So, take 20% of your time and focus on engineering something that is causing you pain as a team. And just don't tell anybody because nobody will notice. Now, that's a horrible way of working that has no visibility, no feedback loops. But if that's how the organization behaves, that's the context you have to work in, then that pattern will work. You just do that work, it's embedded in something else that just happens to take longer, but nobody knows how long it's actually going to take. You're going to get blamed for taking too long anyway, you know that, because that's the corporate culture. So make sure that you make your life better because then the work you do will be better over time. And that was one of the lessons I learned when I would drop into coaching a team. I would typically pick up all the patterns from a previous team and say, these worked, you should put them in. You should implement these. This is what you should do because it worked last time. And then I quickly learned that the context of the organization, the team, meant that often that wasn't true. And even like you say, with co-founders, when Nigel and I worked on projects together, The way we worked together was markedly different than the way we work together now as part of our company because the context is different. And so that's the key, you're crafting it. And then the other thing I really encourage people to do is look for patterns that are well described and apply them because all that thinking somebody else has done, all that testing of those patterns gives you a quick start. You reduce the amount of effort you have to do to fix a problem. And so that's one of the things that the Data Kitchen Cookbook what it did was it gave examples of patterns that solve problems because they're common problems that have been solved before. And so when I look at your 7 Steps to Implement DataOps, you're sucking out what I would determine as DevOps principles or DevOps patterns and applying to data teams. And again, we've got to remember this was written in 2019 when we weren't doing a lot of these things, and I still wonder if we're not. One of the first ones was add logic tests. That was one of the core patterns you called out, implement version control system. Now, the interesting thing about that was version control systems have been around for ages, but it wasn't until we got a tool called dbt that the market seemed to have adopted version control. And I'm really intrigued by that because the version control systems have been around for a while. Data transformation is just code, but nobody was doing it. But then we get dbt and everybody starts doing it. But if I look at the third kitchen, if I look at the BI reporting restaurant, we still don't version control. Very rarely do I see our dashboards and reports being version controlled. If we go back to data ingestion, I still see teams snapshotting and doing no version control of the data. So again, we seem to pick these patterns up and apply them in part of our factory and then not in others. So is that how you see it still? You still see, go and find good patterns, describe them, and use them if they have value?

Let me answer that in two parts. Circling back to the question of how do you handle a team that lives in blame and got a lot of legacy code? And I think you were saying, let's get 20% of the time where they're refactoring and improving. I agree with that. I like a little bit of a different variation. I like the quality circle because I think the most visible things are errors. You're late, data's wrong, customer complaints. I'm a big believer in put every error that you see visible or non-visible to the customer in a spreadsheet, and then reflect on it once every 2 weeks and find something that you can do to fix it in an automated way. I personally have been a believer in that process because, A, you get actionable stuff to do, you're coding things to do and fixing things. And then second is it changes the attitude towards errors in your organization because instead of becoming a personal thing where you blame someone, it becomes an item on a list that I can fix and improve. Improve. And so I'm a big believer in quality circles as the first thing. DBT, in my mind, did more for data ops than Data Kitchen ever did because of coding, because of DBT tests. And I think that because of analytic engineers, and yeah, you're right, it's not applied fully, certainly not in the report domain. The data scientists and their models, sometimes yes, sometimes no. The ingestion teams, no. Governance teams, not at all. They don't even think about governance as code. That's like alien. You know, we still have these words of active metadata that I'm going to active metadata and Magic's going to hear. The most important metadata of any organization is code. Well, that's the definition of active metadata, right? It's code. And so keeping that centralized and versioned is incredibly important. There has been some progress because when I first started the company, 2014, we did our first conference, 2015. 2015 at the Enterprise EDW Conference, we were doing the 7 steps and I asked, how many people have heard of version control? And literally 5% would raise their hands and be like, whoa. And now if I ask that same question, it's 95%. And so that's good, right? People understand version control and they're starting to understand testing due to a whole bunch of people talking about data observability, which has moved to move that ball forward. So, I think those things incrementally are making the world better, but the future's there on some teams and not. And even then, with DBT, they'll use version control, but they won't be managing environments correctly. They won't have any kind of data quality tests to tell them that the system's wrong, or to tell them if the system's wrong in development or production. They don't have accurate test data. There's a whole bunch of intermingled things that you have to do to have really, really high velocity and high quality that I think teams are working on. And often the challenge is just their overwork and their commitment to the customer. There are a lot of heroes out there and hero teams are the hardest. We do some consulting as well, and we work with a hero organization. They're delivering data and analytics about media to people who are making media buy decisions, and they're really working late. A couple of heroes, they're trying and they built a lot of technical debt. They didn't have any tests. They did use some version control, but they had two systems. I think you can make an impact on an organization slowly. They're improving. It didn't happen overnight. They're improving and they're going to keep getting better. And so I think as all of us get a little older and understand organizational change is not a flip a switch, you have to work gradually. And whether hiding 20% of your work on refactoring or doing quality, you should be able to show when you make these changes, you want to show the effect of them. And so, that 20% you want to point to, we refactored a whole bunch of code, we removed 15,000 lines of code, we've taken things for things, and now the business effect of that is we're deploying 2 days faster than we were before. If you can measure it, because I think the other part too of the challenge of these teams who have a hard time is they don't measure anything about their work. There are no DORA metrics for data teams. What do you measure? Do you measure cycle time? Do you measure change fail rate? Do you measure error rate? When you have 3 different kitchens, how do you measure each? As data and analytic teams, we're so unanalytic about how we run our organizations. And so I'm a big believer in pulling ad hoc measurements forward. Just knowing it takes you 3 weeks to deploy. You don't have to measure it, but if you can get rough, or knowing how many errors you have, or how many code commits you've done, broad brush changes. Even scores from your customer, just net promoter scores. Do you like what we're doing? Yes or no? Are we helping you? Yes or no? Send that to all your customers. Find out. You may not like the answer, but find out. And do that once a quarter. I think a lot of data teams would really be shocked at their net promoter score because I don't think they are being effective. I think that would really rock a lot of their worlds to find out that 80% of their customers think they're useless.

If we look at the product, Well, Net Promoter Score tells, gives us one insight around whether people think the product we built is valuable. Usage stats, product teams will record who uses what and how often. The number of times I see data teams not instrument who's using the data and how often, which is an engineering task. This is one that always confuses me is that to engineer the recording of the logs and visibility of who used what from a data point of view. To determine, hey, I built this information product and now you're not using it. What's wrong? What do you want us to fix? Or, hey, we've got all this data and we've brought it in, Shane brought it in, and nobody's using it. Therefore, we can now have a conversation around waste. That should just be default behavior for the teams. I'm with you. And I love that idea of quality circle. More about it, I love the pattern of by a team creating a list of things they know are wrong. That need to be iterated, just in a spreadsheet, and then saying, we're just going to fix one of them. That's where the retrospective from Scrum is meant to take us. It's meant to say, document the things we know aren't working well, and then let's pick one, two, or three, and fix them next time, fix our process. Whereas a lot of time, the team's just focused on operating the process, not improving it. And then the NPS one, in hindsight, The feedback you get as a data team that's not adding value is when there is a downturn, there is no longer a data team not adding value. Potentially an NPS score a year or two ago might have helped some data teams.

That's the wrong grain of feedback, you know. Don't have a year grain or a decade grain.

No, but you know, it's the same as people just unsubscribe to your product because they're not finding it valuable. That's the same as a data team no longer having a job. A bunch of patterns that are out there. I think the core is when you talk about DataOps, you're talking about more than just taking technical DevOps principles and applying it to data teams. You are talking about team design, the team's process, the way they work, the discovery of what needs to be fixed as a problem in the organization with data, all the way through to then building it, to then delivering it, deploying, enhancing it, and then checking that actually it is solving those problems or what do you need to do. So you're effectively picking up patterns from Agile, patterns from product, patterns from Lean, patterns from data, patterns from software, and then you're mashing those up into this DataOps way of working.

I'm not talking about what database you use or whether you use Airflow or Prefect or what reporting tool you use, whether you use data vaults or not, or one big table. Those are good decisions, but none of that stuff actually matters. All the debates that you have that go on in data and analytics aren't really that important when you realize that 70, maybe 90% of the team's time is waste because they're building things or doing things that aren't delivering value. And when you take that perspective that we are just wasting a lot of time, and then you really start to rethink what you should be doing as a leader to remove that waste. And so the biggest thing about DataOps is you're just wasting time and 2/3 or more of your team's time is wasted, your time's wasted. So what are you doing in life? Why do you even want to be in this career? You go to career day at your kid's school and recommend, hey, 2/3 of my time is just— I'm not doing crap. I'm working really hard, but 2/3 of it's just going down the toilet. Are you going to tell your wife when you get home, yeah, 2/3 of my time, it's just crap, I'm not doing anything? It erodes your soul, and it's not a good feeling. And you can kind of deny it and not look at it. But I think if anything, DataOps is just look at the amount of waste your team has and the amount of stuff that's not working. And the way that you evaluate waste is, is your customer actually getting value from it? It's really clear to do that. Not that it's done or that it's shipped or that it's live. It's that it's used and that they need it. Those are really hard metrics to get in your head, but if you back up from that, then everything else gets clear. That's all we really care about is usage. Liking and value.

And I wonder if that's a better definition of DataOps, is the focus on reduction of waste in data teams or data process. And then I come back to your point that there are no DORA metrics for data teams. That's probably one of the problems is people don't know what good looks like. And then I start thinking, what would I do? One of the ones I've already talked about is I would instrument who uses the information products we've delivered because there's a metric of waste. The team spent 3 months, even if they spent 2 weeks building this thing and nobody uses that thing, that was waste. I know a number of people that complain about the number of meetings they have because they see them as waste. Well, if they're waste, stop doing them. Like, you're sitting there going, I do this thing that is a complete waste of my time. Well, then don't do it because you've identified it's waste. Maybe that's a DORA metric, number of hours spent in meetings. And then you've got to be careful about collaboration versus a meeting and how you define that. Cycle time from request to deployment, but the entire cycle time, not just the, you've got the requirements and you're starting to build. From the date the stakeholder said, I had a problem I need to solve with data, to the date you deliver some data that may solve that problem. What is it? Because what I see is a lot of requirements gathering that then goes into Jira Ticket Hill as a backlog. And then gets prioritized in a couple of months' time. The data team are really busy when they get that request onto their plate. But if you're the stakeholder, you are counting from the day you said, hey, I've got a problem and somebody engaged to understand that problem. That's when the time started ticking. That's the beginning of your cycle time, not when the data team started working on it. So there, I think that could be your next goal really is to create a set of example DORA metrics for DataOps waste. Hard.

Yeah, I thought, see, I wrote a blog a while back about the 4 metrics that matter. And I think error rate and cycle time are one of them. I also think this idea of utilization and value, or maybe the cross-product, because you can have something that's poorly utilized but incredibly high value. And it's like the CEO's report. He looks at it once a week, but that's really important. It doesn't have much utilization, but yeah, it's got value. It's a connection. It's the— you can have something that has very little value, but a lot of people look at it because it's a tracking metric for them. Okay. It's the, it's the multiplication of two. And then it really comes down to productivity. How much time is on task? Maybe you could, you subtract out meetings, you subtract out waiting, how much your creative people are on keyboard coding, creating value. We used to have a CDO council and the people would look at it and they'd send out spreadsheets and they'd go, God, my team has 3 hours a week, literally 3 hours a week when they're on keyboard creating value. The rest of it is coordinating, collaborating meetings, And this is fixing stuff that should have been working in the first place, answering customer questions, all this stuff. We are creative people who want to build. And so the time that you're actually building important new things, that's a really important metric. And that's the big driver. And we don't measure what productivity means for our team because I don't think we want to know. All I know is if your customers trust your data, that means you have very low errors. If you can respond to their requests quickly, and get something in their hands that's 70% right, that means your cycle time's fast. If your team is not spending a lot of time in meetings and they're actually trying and they've got time to experiment and refactor, their hands are on the keyboard a lot, that's productivity. And then the last thing is really usage. You've got usage and the combination of usage and value. And maybe we have a study, but it's very rare that I see a team who tracks any level of those metrics. And I think there's a whole industry around DORA metrics and software. There's DORA software companies. I think from my standpoint, I'd love to build the DORA metrics for data ops, because I think it's really important. I've thought about maybe teams would pay to do it. I just, I don't know. When I've looked at it, like we did a consulting engagement with a big pharma company and all I wanted to do was find error rates. I wanted to find error rates and cycle time at the time. No one had it. No one knew it. No one would fess up to it. It's like, we don't know. You don't know? How often? How do you not know this? This is one of the top Pharma companies in the entire world and they just didn't know, or maybe someone knew and they didn't know how to find out, or it was, it's a lot of it's sort of hidden in operations teams somewhere in some other country. So the barriers to finding this information out is hard. I think as a leader, running your team on metrics, identifying what those metrics are, and you do get what you measure. The DORA metrics aren't perfect, but if you can change them and focus your team, that's what you're trying to get your business customers to do. Drive their business on metrics, yet you don't have them. I think that's a bit of hypocrisy from leaders. And I think I would challenge them to start doing it because you can do it really simply. You can do it with, just email me last month, how much time you spent coding and how much time you spent in meetings.

It's about balance. So, we don't want to go down to that horrible one where teams are encouraged to do nothing but code because the collaboration and the planning is important, but it's about balance. And yeah, I think you're right, this idea of data teams ain't data-driven. It's like that amazing chef who's in the kitchen every night and then goes to McDonald's every day to eat themselves. Wow, that's kind of an oxymoron.

What's wrong with you? I mean, you don't even cook at home? Like, don't you get, you know, some quick meals?

We should probably start doing the things we tell everybody else to do ourselves.

Yeah. I go back in my own, I get negative and positive, right? Because I do see People are, I think the adoption of code, the adoption of observability and data quality, I think those are driving functions. I think the adoption of cloud is helpful because a lot of those systems are built with CI/CD and Git built in. Most of those tools that people use to orchestrate, a lot more people are doing things in code rather than in these sort of black box systems where your ETL tool, it's in their code, it's in their format. You don't understand it. It's sometimes binary or sometimes in tables or in XML. It becomes more public and readable. And I think there are companies now, there's 3 or 4 that are involved in sort of code-driven BI or BI as code. That's their, you go to their website, BI as code or headless code-driven BI. Good data is on that. So I think there's good. And then there's companies like Kabul that are basically doing environment management. They're handling all the DevOps in the, I wanna script my environment up with my tools. I wanna tear it down. and you can buy that now. You got your 5 things in your data platform and somebody solved that problem for you. So if you look at the critical DataOps capabilities of environment management with test data, there's test automation. Now people will write your data quality tests for you. My company does that. There are observability tools. We have 2 open source tools to do that. There's version control that's built into a lot of your teams. I think there's a lot of bits and pieces that people are putting together. The biggest thing, I have not seen an overall sort of value stream metrics, DORA metrics tool for data teams. I don't know, part of me thinks there's an entrepreneurship opportunity, but part of me also just says, could you just use Excel? Just track errors, just ask some simple questions, do a net promoter score and email. You can do this really cheaply yourself as a leader, and it's once a month, once a quarter. I think it would really be interesting for you to reflect on just those, those 4 things, how much time people are spending on task, how many errors you have, are people really getting value? And then you can go to the next level, which report is actually being used or not? Maybe we should decommission it. Which table, which dataset isn't providing value? Where are all the errors in this? Maybe the locus is that provider who's giving you bad tomatoes and they're the one who's causing the meals to be bad and you should replace them. Because sometimes they don't even know who the bad suppliers are, who's late. It's not reported on, it's not visible. And so, making— another way to think of DataOps is making the invisible part visible, making a lot of those work processes owned and visible and iterate-uponable and changeable.

They go to a different supplier for tomatoes. I have seen that. I've seen organizations where they had struggled to get the good data out of their source system, and the data team have been highlighting that, but so has the rest of the organization, the operations team, the frontline staff go, that system's shit, it doesn't give me what I need, I can't use it. And so then they replace that source system with something else that's better because they're actually solving a bunch of problems. And then I come back to, with the downturn, if I kind of look at it and I go, yeah, the problem we had was instead of the leaders of an organization going to the data team and saying, you're valuable, how can you optimize your factory? To reduce the cost of what you produce or increase what you produce without increasing the cost. All they did was shut the factory down. And that's telling. There is no value in you as a factory. We're just going to turn it off. We're not even going to have an alternative. Well, alternatives excel. And I think that's where we kind of go. We haven't got there yet, but I'm with you. We're starting to see a lot of change. And that's what we saw in the software engineering world. We saw all the same problems we had around low-quality code, no automation, products that consistently broke, high levels of waste, too much time spent upfront getting requirements and then not delivering anything for 3 years. I remember the software projects that were big and ugly like those, with all those problems and, and our software brethren seemed to have been ahead of the curve compared to us, and bringing patterns to fix that. So I think we've started that journey. The question is, how fast is it going to go?

Yeah, I don't know why. I really had thought it would be faster. And I thought there's something about software engineers that are arrogant and a little bit anal, and they don't like waste, and it bugs them. And it really bugs a lot of people to do things twice and to do things 3 or 4 times because they know really deep down in their soul that it's going to cause them problems. And data teams aren't like that. They're like, oh, I'm just going to copy this code and tweak it a bit, and I'm going to copy this code and tweak it a bit. And then They don't feel the same way, and maybe because they've been beaten down more and they don't feel that they're arrogant. I'm not sure what it is, but I would like data teams to be a little bit more empowered and prissy and saying, look, we have to fix this thing. We have to own our process. I refuse to work this way and I'm going to quit. And I think that would be a good thing for some organizations. If the people who actually know where all the bodies lie and are vital to doing something, they threaten themselves to quit. That would be a good wake-up call for some organizations. Not that they quit because they're overworked, but they quit because they're like, we're not improving our process.

Yeah. And they go work for somebody that wants them. Yeah.

They go work for somebody who does.

Yeah. I've thought about this a lot. And the thing I come back to, and it's a small thing, but I think it is the thing, is when you're a software engineer, you're in charge of the entire product, especially with the DevOps community. Now you're in charge of the infrastructure, but more importantly, you're char— You're in charge of how it's used, what it captures, and what it delivers. The end-to-end cycle of beginning to end is you or your team. In data, we get given the data and somehow we use that as an excuse that we're no longer accountable for the final product. And I come back to that's the core anti-pattern. That seems to be the problem. And then a lot of the successful data teams I've seen, It's because they pushed back and had the original data problems fixed, which made their life easier. But that doesn't happen very often. And so maybe things like data contracts will be the change for this problem that we saw DBT change for version control.

Shane, I really appreciate that because I do think data quality is a version of the tragedy of the commons in data. The source data system is often crap. And so we built this engine that scans data and does a whole bunch of data quality checks on it automatically. And we were originally thinking of it from data engineers, like, I want to put it in my process and auto-build tests because it saves the data engineer time. And then we did some market research. We talked to about 35 data quality people this summer. And what we realized is there are people in the organization who really want to improve the data. However, the people who own the source systems the data is often good enough for them and they, they don't have a reason to. However, organizations want the data that that team owns to be improved. So the challenge is how do you get data quality people to have influence when they have no power? And so I think what we've done is we spent millions of dollars on our open source project. It has a nice UI, it's complete, it scans your data, it builds a whole different variety of data quality scores. Not just data categories, but what we realized is that like any influence position, when you have no power, you got to be very selective about what you try to influence. And don't say all your 10 tables are crap. Like, don't be wasteful like that. Pick the 3 columns that are critical data elements for your bank. Pick the 10 columns that this important model for your data scientists are using. Pick the 5 columns that really drive all the reports for your quarterly goals and just work on those, right? And have a tool where you can create a scorecard from them that can monitor them and build a whole bunch of data quality rules automatically, and then give a package to the people who fix them so they can understand what the problem is and what you want. And so if you can do that, we're hoping that that'll be the linchpin. And I really like your idea because if you can actually fix data quality, then people will start— they don't have anyone to blame anymore. Anymore. And they can't say it's that person's fault. And then it actually really is your fault. And then you can feel that way and then you can start working on your process. So I like that idea. It's my hope as well that those teams themselves, they can become— a lot of data quality projects are not agile. They're very waterfall requirements-driven. If I just need to look at my data, I'm going to write the— and I don't think you need to do that. I think you can scan your data. Our tool, within an hour, it gives you over 50 different data quality checks automatically run, automatically size. And you can start there, turn some things off, add some things that are domain specific and work from that and give that list to people to fix, to push back. And if you are smart about it and don't do a Daima Data Qualities category of 10,000 tables and say, you've got to go fix your fax number, it isn't right. And like, who cares about fax number? Nobody uses the fax number. Pick the attributes that just provide business leverage. Do that, show that they improve, pick some more, show that they improve. And then lo and behold, within a year, you'll have much better data quality functionally for your organization. Then you're right, maybe data teams will start to say, look, I can't blame source data quality anymore. It's really my fault. And maybe that'll be the linchpin to move us forward. That's what we're hoping is data quality is the linchpin to start agility.

So, if I think back, I'm old enough to remember Lean, Six Sigma. Data quality management, those kinds of patterns that help change organizations because they focused on quality as being one of the core things they focused on. And maybe that's what the change needs to be, is that actually quality of data is a focus, not just a bunch of governance stewards sitting there complaining about how bad it is and doing stock takes. Maybe it actually has to become one of the key drivers of the organization.

Because it's good enough for the team that owns the data, but it's not good enough for the teams that use the data. And, and if you make the rules of what they need to do, the people who own it, if you can make it clear to them what they need to do, they will fix it, right? And if their boss's boss says you got to do this, they'll fix it. Because it's making what's invisible, the use of my data by other people, the tragedy of commons, it's making it visible through a very specific dashboard, a very set of specific actionable recommendations. And that visibility, I think, will enable people to stop, like you said, being data quality people who are kind of, they're fun to talk to, they're interesting people, but they're kind of bitter because no one listens to them. And we're trying to give them a nice specific bat where they can kind of knock some people on the heads about the attributes that matter. And we did talk to organizations who were very successful and they were successful on specifics, like one bank who very much focused on their critical data elements that were for were reporting to the government, they didn't fix all their data, they just fixed some of their data. And because it had to be done because they could get busted as a bank if they didn't do it. And not all banks do that, but that's a really good reason. And lo and behold, those elements then can be used for lots of other things because they're of high quality now. And then they said they're starting to do more elements that aren't related to compliance reports because they're related to revenue-generating activities.

I'm just gonna close this one out now. So you do a lot of writing, create a lot of good content, and you have open sourced some of the cool products that you've been working on for a long time. So if people wanna find you, read what you're writing, find these tools, where do they go?

They go to datakitchen.io and you can get a link to our open source. You should be able to play it out. We've got demos, you can try it right away within an hour. We've got a great blog. We have two books on DataOps Cookbook and Recipes for DataOps Success. We have two training programs of that. We've had about 4,000 people do our online DataOps training. We've had about 1,000 people do our Data Quality and Data Observability certification that we developed in the last year. I'm trying to write once every week or two about these problems. I'm 61 now. I'm not going to be working forever, but I'm still hopeful. There are days I feel like taking my toys and going home because people are doing some dumb stuff. It's like, I'm not going to play with you anymore. Today isn't that day. I do feel like keeping my toys on the table. And I appreciate talking to you, Shane. You make me feel good about having my toys on the table and not going home and pouting.

I think there's a bunch of us that have been frustrated for many decades and we've tried many things to solve some of the problems and some of them have been solved and a lot of them haven't. So it's not easy, but keep fighting the good fight.

It's better to be in the arena trying than on the sidelines with your toys to yourself.

I agree. Excellent. Hey, look, thank you for the time. That has been excellent, and I hope everybody has a simply magical day. And that, Data Magicians, was another Agile Data Podcast episode. If you want to learn new patterns that you can apply to craft your own agile data way of working, head over to agiledata.io.