Ways of Working with Scott Ambler - AgileData #47

Welcome to the Agile Data Podcast, where we talk about the merging of agile and data ways of working in a simple, in a simply magical way. Welcome to the Agile Data Podcast. I'm Shane Gibson.

And I'm Scott Ambler.

Hey Scott, thank you for coming on the show. I am mega excited about this one. I've been a big fan of your work for many years. And what are we going to talk about today is this idea of ways of working. How can teams build their own ways of working? Why should they do it? How do we enable them? What steps should they start with? But before we rip into that, can you just give the audience a bit of background on your career in this world of agile and data and ways of working?

Yeah, definitely. So, I've been in the IT space since the mid to late 1980s, working in various positions in various industries in various countries. And I'm actually based in Toronto, Canada, but I was an international traveler for many years, and I guess I still am. I started out as a programmer, but very quickly moved into the data space, became a baby data architect, I guess you would say, a phenomenally junior data architect at a very large financial institution, and pretty much did their first data warehouse long before data warehousing was even a concept and learned a lot. I actually was the holder of most of the corporate data at one point, at least the metadata about the corporate data at one point. And the reason why the data architecture group had brought me into the fold was basically because I had more information about what was actually going on than they did. And people knew it. Then I moved into the object-oriented space and human-computer interaction, like user experience now, and modeling. Did some work in the mid-1990s in the CMM space, a very heavy process. At the time. I actually wrote the first CMM Level 5 compliant process, if you instantiated it fully, for object-oriented development. And basically, once I finished that work off, I was pretty much disgusted by CMM and moved on from that and went to the other end of the spectrum and started focusing on agile stuff, UML and agile modeling, agile data. And took on a lot of the— in the early 2000s, took on a lot of the not-so-sexy topics in the Agile space, like modeling documentation in particular. And it was interesting, now we take it for granted, but back in the day, it was radical, like doing things on Post-it notes and on whiteboards. And although to be fair, people were doing it, but nobody was talking about it. And if you pick up modeling books from the late '90s, early 2000s, there isn't a single sketch in them. There isn't, except for the books that I wrote. And you gotta take those out of play on that one. Yeah, I keynoted at an academic modeling conference once, and basically ripped into them for that, right? So I had their conference proceedings in my hand and it was really good, some really solid papers. But basically I was asking the audience, do you guys sketch and use Post-it notes? Raise your hand. And they're all raising their hands. And raise your hand if you've got a paper in this conference proceedings. And it was really thick. Good portion of the audience had published. And I said, so how many of you have included sketches in your papers? Everybody's hand went down. And I knew it would because I flipped through the proceedings and I said, okay, so this is a problem. You're not writing about what you actually do in practice. You're writing about this theory, which is why pretty much everybody ignores you. And so that was, you know, wasn't the most popular guy. But anyways, it opened up the modeling space in the agile world. And so all the things we're doing now were really was radical 20-some-odd years ago. Also did a lot of work in the data space, did some basic fundamental stuff there long before other people came, including yourself and all the good work that you've been doing there. And arguably it was too early. It was a good 10 to 15 years too early because the data folks couldn't accept the fact that Agile was real and here to stay, and they didn't want to hear it. I mean, Agile folks didn't care. Their idea of data knowledge was, "Oh, it's a database. We'll just encapsulate access to it, and that's all I need to do. So no worries." And that would be it, right? And then they'd go off and make horrendous mistakes. So I did a lot of work there, but then when I joined IBM in 2006, I was their chief methodologist for IT within IBM Rational. And there, myself and a few others, including Mark Lines, developed Disciplined Agile Delivery, which then evolved into Disciplined Agile, and then eventually into PMI's Disciplined Agile Toolkit. And that's where we really made it clear that teams should choose their own ways of working and choose their own WOW. And we described how to do it and different techniques, but it's all about choice and understanding your context and giving up on this concept of best practices. Like, best— the term best practices is, it's a wonderful marketing term, but it's an absolute lie. And it's what people want to hear, but it's observably not true. The best you can do is let's do the best we can for the context, for the situation that we face. What makes sense for me in my situation would be a spectacularly bad idea for you in your situation. So we have to help people instead of telling them what to do and say, this is the official practices of methodology X, and if you do these, then everything will work out for you. Instead, you need to help people understand, here's the things you need to think about, here's the considerations and the issues that you're going to have to deal with, and here's options to deal with those issues. So choose the right thing for you, choose the right ingredient off the shelf to make dinner for yourself tonight, right? And your dinner tonight will be different than what you made last night and the night before, and that's okay. But you need the right ingredients and use them in the right way for whatever mood your family is in tonight for dinner.

I'm a baby in the Agile Data space. I've only been doing it for probably almost hitting 10 years now. But I remember when I first started this journey, I definitely came out of a best practice methodology-driven mindset. That's what I got taught when I was learning in my early career. And then I got introduced to this idea of Agile and I was like, you're a bunch of weirdo hippies.

Pretty much.

No way, that's just chaos. How the hell can you do that? And I remember there were 3 3 seminal things for me that, that helped my journey. One was a series of books by Ralph Hughes, which was giving me the insight of agile terminology and agile things in data warehousing language, right? I could see the blending of those two. There was yours around what I now realize was a pattern library, a bunch of good practices that have some context and I could pick and mix things that fitted for my customers. And the third one was Laurence Koor's stuff around interactive modeling with customers using stickies. And I delivered a session for a bunch of data engineers just lately, and I was showing them how they can use the 7Ws in Beam to elicit data requirements early and lightly. And one of the questions that came back from one of the engineers was, why don't you just use ERDs? And for me, that was, yeah, you try and do entity relationship diagram modeling with a stakeholder and see what kind of feedback you get, because you lose them. I think in the data domain, we still haven't adopted that idea of light engagement with our stakeholders with a language they understand to get what we need to do the next step. We view that I think that in the data world, we've got better, but I don't think we're anywhere near the level of sophistication in the space that we should be.

Exactly. Recently, I've developed and started delivering a workshop in agile data warehousing. I used to do that a few years ago, but then had to stop when I went to PMI because they just weren't interested in the data stuff at all, or in enterprise architecture stuff, or a bunch of other things they bought and then just put on the shelf, basically. And rightfully so, it wasn't their business. But one of the key messages in that workshop is that on a data project, if you're doing a data warehouse or BI solution, you want to be usage-driven, not data-driven. The data is actually a secondary issue, and that blows the minds of the data people. And this data focus is one of the primary causes of their high failure rate, is because it's just too narrow and it doesn't really reflect what people want to do, right? If you look at Laurence Kor's work, he talks about data stories, and in my stuff I talk about question stories, and they're very similar concepts, but they're both usage-driven requirements, and that's what's critical. If you don't know, you know, what questions, how people are gonna use the data, what questions they're trying to answer, I don't care what your data structures are. I don't care what you're storing. I don't care what data technologies you're using. It, it's all nonsense. And this is why, if you look at a very classic problem in the data warehousing space is the first release is a loser. And it's because you build it, you release it, and then the stakeholders come along and say, this is not what I want. It's 90% there, but there's 10% that's desperately missing, and I can't do anything with this because this 10% is missing, all that sort of stuff. It's not until your second or third release that you get it right, or at least you start getting closer to getting it right. And it doesn't have to be that way. And the final problem is that they wasted, sometimes wasted years doing data modeling, and nobody cares. And they don't even keep it up to date usually. In practice, even they don't care sometimes. If nobody's using your models, why did you bother creating them to begin with?

What we're finding now, especially with self-service, the whole practice of an analytics engineer, this idea that you can build code that transforms data without modeling. And self-service has massive value. We allow more people to do the work that used to be done by specialists. But when we go to that model, that self-service paradigm, we often lose the rigor or the skill. And we've definitely seen that in the data modeling space. And again, cloud technologies have given us some massive benefits in the data space. So we can now deploy platforms and build them and cobble them together much faster than we used to be able to do in the old days of installing Oracle databases and ODI on servers. But effectively what we've done is we've taken a screwdriver that delivered a hole in a screw nobody wanted, and then we used a power drill to deliver it faster, but it's still a screw in the wrong place. We just delivered it faster. So we've got to get into that model of early feedback. How can we build something small, get some feedback from our stakeholders? Because at that stage, we're going to learn what we did wrong, what we misinterpreted, because we don't have a shared language with them as much as we hoped we did. So just thinking about that and just thinking about this idea of way of working, because that's the whole focus of the podcast today. How would you describe way of working? How would you describe WOW if you had to explain it to somebody from the beginning?

So your way of working is your process. This is how we go about doing our work as a team, as an individual, as an organization, as the case may be. And the fundamental thing is you want to do what's right for you and do the best that you can in the situation that you face and always try to get better. What that tells you is you need to understand what context you're in. You also need to understand what you're trying to achieve. In Discman Agile, we talked about having process goals. Basically, what are we trying to do and, and what do we need to be you thinking about? So there's a little bit of process knowledge there. Once you understand what you're trying to achieve, like I'm trying to understand the requirements for this data warehouse that we're building, for example, then you need to understand what are my options. How could I go about understanding these requirements? Maybe I could do ARDs, maybe I could do user stories or data stories or question stories or use cases or this or this or this. Maybe you could do screen sketches or some random data sketch that I've never seen before, right? But that somebody else has figured it out. And the idea is that you've got choices, but which one of those techniques is right for you in your situation based on what you're trying to achieve, what your skills are, what your culture is, what your preferences are? And then how do you choose which technique is right for you, particularly when you probably only know 1 or 2 of the 15 that I just listed? And number 7 was probably your better option. But if you don't know about it, and then if you don't know what the trade-offs are, how the heck can you make a coherent decision to experiment with it and see if it's right for you? Because just because technique number 7 sounds like a good idea doesn't mean you're going to be able to pull it off. I hope you can, but stuff happens. This is the idea, right? You've got to have enough knowledge, or somebody on your team has to have enough knowledge, or you have to have some sort of a toolkit or some sort of magical AI tool perhaps, that can give you advice as to, here are your options, you know, here are your choices between them, choose wisely, right? So it's the Indiana Jones thing.

I think for me that the idea of a toolkit, that kind of what resonates. And just lately I've been on a whole analogy of food in the data space. I don't know, I always seem to go back to food as the analogy and I liken it to a buffet. We have a bunch of choices that we can put on our plate. We may have team members that are allergic to egg. So some of those food choices, they're not available to us as a team, given our context. And so what we're doing is we're picking from this pattern library, we're picking from this buffet of choices, and we're fitting the ones that work for us, given our context, and solve our problems. And then we're saying, okay, that looks like it's going to be a good, nice meal. Let's try it, right? Let's experiment with it. And that's what a lot of teams don't do, I see, is they pick things, they draw them, write them down, they therefore become immutable. This is our way of working, and they don't go to the next step, which is actually we're just guessing. We think that pattern's going to solve that problem, but we need to experiment with it. We need to test it. We need to do a retrospective on it that says, did it? Yes, no. If the answer's no, don't carry on doing it, find something else, right? If the answer's yes, then lock it in for now, but at some stage you're probably going to change it again. Is that what you found is that your way of working is incremental, isn't it? It's a toolkit that you're constantly adding to and removing and iterating as all agileists should.

Exactly. And actually in DA, we use the buffet metaphor for quite a while. And we also used cooking your own dinner as well, because you could compare that to going to McDonald's and getting the Big Mac meal deal. And yeah, you can eat that every night, but it gets boring pretty quickly. And not everybody in the family wants it. The interesting thing about the buffet analogy is that if you actually observe a family at a buffet, everybody gets fed, but they'll all go and choose their own food. And they're all much happier than whatever it is you were going to cook them instead. And sometimes you'll pick something. What the heck? What is this? I wonder what this is. And you'll pick it and it's great, or you'll pick it and ooh, that's not for me. And you'll leave it on your plate and it gets taken away. A little bit of waste. You had an experiment and some experiments work well and some don't. So it's interesting. Now it's a little bit more work for the people. It's not just, I go to the table, sit down and eat what's in front of me. It's, I've got to wander all the way over to the buffet, make my own choices. Wander back to my table, eat, and maybe rinse and repeat a few times and do that. So, it's a little bit more work, but I get a better meal. I get a meal that's going to meet my needs. And everybody eats differently. And that's absolutely important to observe because every team is unique. Every person is unique. Every team is unique. If you want to be effective, you really need to allow your teams to have unique ways of working, just like you allow your children to have pick-your-own-food buffet. Now, with my daughter, sometimes I'll say, you can't eat, you can't just go to the dessert part of the buffet. You gotta have some vegetables, you gotta have some real food first, and then you can have some dessert. So you gotta give them some guidance, a little bit of governance is required, but you gotta be smart about that. Be so, you let the kids pick their own food and some experiments don't work out so well, and that's okay.

If we look at the data space, like I said, I think we've lost the art or the skill of modeling to a degree at the moment. It's starting to come back. We always go through waves of decentralization, self-service, and then moving back to more managed, governed, skills-based, and then we go back out again. One thing I've observed is data teams are really good at looking at processes from a data platform point of view. The idea of collecting data, profiling, cleansing, conforming, consuming it. They understand that process very well. That's their specialty within that domain. But if we look at business process or team process, that whole idea of process engineering that we had from many years ago, most data teams don't seem to understand that and they seem to struggle to take this idea of they can map their data process and they can see nodes and links and they can see the first step and the second step and they can see the entry into that step and they can see the exit and they can see what the dependency for the next step is. But when you ask them to do that in their way of working, who's going to do what? And then what are they going to produce? And then what do they do next? They seem to struggle. And have you found that? Have you found that skill of understanding team process and business process and lean process is lost at the moment? In the data space or just generally?

I think it's a general issue. I think the data crowd tends to struggle with it more just because all these communities are self-selecting, right? So the people who move into the data community tend to be really good at structural thinking and understanding data and all that sort of stuff. And they might be a bit weak on UI and they might be a bit weak on business processes and stuff like that. So then as soon as you ask them to step out of that safe space for them, then yeah, it becomes a bit of a mess. But frankly, it is a hard skill. It's a level of abstraction that a lot of people aren't used to thinking at. It's really hard because there's so many "it depends" situations, right? You really can't describe it well because it's the same old thing, right? You ask people to describe what they do. "Tell me about your day." And they'll miss 50 things and they'll describe things that they didn't do and that aren't needed. And they'll describe things that only they care about. And if it was to disappear tomorrow, nobody would notice. Things like that. It's a hard problem and it's gonna change anyways. Imagine you had to define your process for tomorrow. Here's what I'm gonna do tomorrow. That plan will last maybe 10 or 15 minutes and something's gonna happen. Somebody asks you to do something, you, you get a call, something breaks. Earlier today I was coding and for some reason Python wasn't seeing my GPU anymore, which is problematic cause I'm doing some AI stuff. So I really want access to my GPU. So now I'm gonna be fooling around for a couple hours probably. Trying to reinstall CUDA and all that good sort of stuff. It is what it is. My day is shot. Because that's anywhere between 5 minutes and 5 hours. I don't know yet. But so stuff happens. Your situation changes and you're going to change up the way that you work. And that's okay.

If we look at the Agile market, at least, we've gone to the death by frameworks, right? Death by methodologies, in my view. And naturally, as humans, it feels comfortable. It feels comfortable that rather than building our own toolkit, having to do the cognitive load to what are my problems, what can I find, how do I test it, how do I see if it's going to solve that problem in the way I work. It's much easier to get a picture somebody else has drawn with a bunch of steps and then follow it. Have you seen that? Have you seen that the market's moved away from this kind of craft, something that's fit for purpose and customized for you and your team to adopt a standard methodology or framework yet again?

Yeah, so I think there is a lot of comfort in being told what to do. And even with intellectual workers, they still want to generally want to be told what to do. And organizations also want to tell their people what to do. The leadership wants to be leading and be managing and doing their jobs. But the fundamental challenge is that these methods aren't fit for purpose for whatever your purpose is, right? You get some things like Scrum, which are like, there's almost nothing to it and it's so wishy-washy. That you can force fit it into anything, but then it doesn't really give you much advice as a result, other than to run a few meetings when it gets down to it. And then at the other end of the spectrum, you get the SAFe's of the world, and they're overly defined and they're prescriptive with one way of doing things. They talk a good game and they give you a few options now, so it's gotten better, but it might work for some people, but for the most part, it's probably not going to fit well for you. So then you're still going to have to tailor it. And it's so complex that you don't know how to do it. And you don't know what to take away. You don't know what to add. So you're still sort of out of luck, but you've got a pretty poster and you've spent a lot of money on training. So things generally don't go well. But it really gets down to you need to know what you're doing. And that's a hard message, but you need to be skilled. You need to have the skills to do your job. And I would say this to executives. I would say this to management. I would say this to the people in the trenches. You need to have the skills to do your job. You need to understand what your job is and how it fits in. And you need to understand whatever the purpose is of your, of the team that you're working in now. So if you're on a data warehousing team, you'd better have an idea about data warehousing. If you're on an AI team, you better have an idea about AI and be pretty good at it. And this I think is missing. These methods don't go into the details. They try to be situation agnostic. So yeah, you can do Scrum, but it tells you nothing about data warehousing. Tells you nothing about building apps, it tells you nothing about marketing and like all these other domains where you're applying it into. So you're left to deal with that anyways. In that case, people being led down the garden path, it's the art of the possible, you can do anything you want, which is absolutely true, but then you're stuck figuring it out on your own anyways. And then there's not a lot of advice there other than, well, hire some expensive consultants who generally also don't know what your domain is and what you're trying to achieve. So this is one of the reasons why the shine has come off of Agile coaches lately is because there was just this overwhelming number of people who weren't qualified and they took a 2-day course and maybe were on a project or 2, now they're pitching themselves as a coach. How offensive is that? And then, and now they're wondering, why can't I find a job? Because the entire community is not really qualified. And people have finally figured that one out. So it's problematic. So you're still stuck anyway. So yeah, you get pitched this story that we can tell you what to do, but then it's not really, you don't really get it. And to go back to the food analogy, it's like telling people, yeah, you can cook your own meal and here's the book, The Joy of Cooking. Good luck. No, you need to have some, when you talk about fricasseeing something, well, what the heck is that? I don't know. You better have some basic cooking skills to get the job done, right? Where are you going to start?

For me, I differentiate leadership from coaching. My opinion, it's a strong opinion, is coaches should have played on the field. They should under— understand the domain. And then they spend all their time not telling somebody how to do it, but coaching them how to do it, right? There's a difference between telling and coaching. But if you don't understand what they're doing, then you don't have the context. So that's what a coach does. And then a leader gets out of the way. So for me, a leader doesn't have to have domain knowledge, but they have to empower the team who does to do the work and lead them and remove the barriers and let them get on with the the thing that they're experts at, help them get rid of the roadblocks that are in their way. So, if we look at that, let's say we are lucky enough that we have a team and that team's been suddenly empowered to build their own way of working. That team's in the data domain and they have some skills in that domain around data and all that good stuff, but they have limited skills around their business process or the way they work. And you are coming in to help them. Where do they start? I know there's no one way to start for every team, but generally, where would you tell a team to start working first in terms of building their own way of working?

I generally start wherever they are, right? So if they're already in progress, then I start looking around. First, I want to observe and get a handle on what's actually going on rather than what they're telling me is going on. But I also want to start trying to understand what are their main challenges right now and where's the pain. And let's deal with the immediate pain and hopefully try to get some quick wins as well. It really does depend on what situation they're in and what they're trying to achieve. And is it realistic? So I've been on a lot of teams where they've got this completely unrealistic date to hit or this completely unrealistic scope for their budget or whatever's going on. And they've basically given up or they're frustrated and they should be giving up. They haven't quite figured out to give up yet. I try to help them understand where their main problems are and then start knocking them off one at a time. As we're doing that, it becomes a coaching thing of of helping them understand that they've got options and asking leading questions and pointing them in the right direction sometimes too. And sometimes as a coach, you do have to tell them what to do. If you're a hockey coach and the team's milling about on the rink and the other team's about to score and you say, "Get your butts down here and take that guy out." They're nowhere near that level of figuring out that they need to be protecting the goalie or something. Sometimes you do have to give them more advice than you probably would like to. But often it's really all about guiding them into helping them make their own decisions and then standing back. And sometimes you pretty much know it's going to be a bad— this is not going to work out well. But you know what, let's see what happens. Going back to the buffet analogy, my daughter, a couple of years ago, she went to the buffet and grabbed a big plate full of shrimp because she likes shrimp. But how much shrimp can you really eat? And so we found that out. It was about half a plate, which is pretty impressive, but still not the full plate. So I ended up eating a little more shrimp than I probably wanted to that night. It's a bad idea, but let's see how it plays out.

You know, this is great. Oh yeah, and definitely, number of times I've said to teams, in the early days, I said, that's never gonna work. I've never seen that work. And then experience taught me to say, I haven't seen that work before. I've seen teams try it. Here's the problems they hit, but give it a go, cuz you know, you get surprised. And for me, it is about that context. So I frame it in my head now about a team starting up from scratch will focus on certain patterns. We focus on the team building, the ways of working, the team charter, those team forming skills. If you go in and you're helping a team that's already down the track, then you're starting to look at where the problems are and what needs to be changed next to help their way of working. And then if you're at the stage where they're scaling, they're trying to break out to other teams, and that's a different problem, a different set of context. And for me, actually, I've got to the stage where I don't actually care which problem they want to solve because it's not solving the problem that we're there to help them with. It's helping them understand that it's their way of working, it's their process, and they need to be able to define and iterate it. So, it's teaching them the skills or giving them the knowledge of how to iterate their way of working regardless of which problem they're solving next. Exactly.

Teaching them how to learn. Yeah.

And so, that success, right, is when that team can iterate their own way of working, they don't need us anymore. That's when we've done our job because they're just going to keep building. Better practices. And so that's one of the things in Choose Your Well that I loved was this idea that you always focus on the goal first. So what's the goal we have to achieve? And then what's the problem that's stopping us getting to that goal? And then which pattern could we adopt that may solve that problem to achieve that goal? And I really like that way of thinking about it, which is that end state and then experimentation to get us there with our practice, with our way of working. So that was something that I'd forgotten actually until I read it again. It was like, okay, forgot that little puppy. So this idea of a hybrid bunch of good ideas, because again, if we talk about way of working, I've followed what you've said for years in terms of there's a bunch of patterns out there. There's patterns in Agile, there's patterns in data, there's patterns in product, there's patterns in Lean, there's patterns all over the place that we can adopt. And they have value in given certain contexts. But what we end up with then is we end up with teams working different differently, because that's our goal, is they're going to work given their context. But from an organizational point of view, it seems scary. Hold on, don't we want to standardize all the teams? We can go to that horrible Spotify model problem, which is Spotify had a strategy and the strategy was people build their own way of working. Somebody documented it once and everybody goes, oh, that's how Spotify works. And we talked to a few people from there and they're going, bollocks, actually. That was just one visualization of one state at one particular point in time. But as organizations, we always want to go, actually, let's just adopt that and put that onto every team. Have you seen that? This idea of standardizations across teams seems to be more important than enablement and getting the job done.

Yeah, exactly. And this is what CMM and CMMI were all about back in the day, having repeatable processes. The politest thing I can say is that's just ignorant nonsense. It's observably not true. Every person is unique. Every team is unique. Everybody's in a unique situation. And all that stuff is changing constantly. So having one way of working across all teams. Now you might magically be in a situation if you were a consulting organization and all your salespeople were selling was the exact same basic type of problem or same type of project to all your customers. Yes, maybe you could have a reasonably repeatable process across all those teams, but you'd still have teams of different people and the clients would still be different and it would all fall apart instantaneously. So say all the planets were to align. And you could actually somehow find yourself in a situation where every single team following the same process makes sense. Within a few hours, the situation will have changed for at least one of those teams. And then suddenly they've got to start doing something else or continue to apply the same inappropriate process for them, which, and if they're smart people, what they start doing is they start doing the right thing anyways, and then they start creating the artifacts that you insist on to make it look as if they're following the old process. You fake out the process. Process. And then that's just all useless overhead. That's probably not what you're looking for. And the interesting thing is often the people who are pretending to govern are unable to detect that's happening because they're almost always looking at the artifacts as opposed to the actual actions of the people that are going on, that the people that are actually doing the work. This is just, it's no good. It's what people want. They want a simple solution, but the world is too complex. And I think what's happened is I work with organizations and everybody's overwhelmed. The complexity's gotten too high. They have all this legacy, all this technical debt and all this data technical debt. And the customers are asking for a wide range of stuff. The technologies are changing constantly. You can't get caught up on the solutions at all anymore. Everybody's just overwhelmed, including leadership. The leaders don't know what's going on either. They're trying to to get a handle on what's happening and it's just too much for them. And yet they're trying to make decisions and they just can't. It's far too complex and it's not what people want to hear. And somebody comes along with this snake oil, follow this methodology for doing this type of project or for doing it across your entire organization. And it sounds good. Like the story is always good, particularly when you don't really know what you're listening to and you're overwhelmed and you desperately want a solution and somebody comes along with this semi-complex thing that sort of makes sense to you. And then they're slick. They got a slick salesperson or a slick story, and it sounds good, so you adopt it. And a couple years later, you find out it didn't really work out for you. It's a disaster. So you go on to the next snake oil salesman. And this is what these organizations are doing constantly. They just lurch to the, the next hopeful solution, building up all this technical debt and cultural debt as they go. And not really getting the job done. The only way you can be successful these days is you've got to step back. You have to accept complexity. You've got to embrace the situation that you're in and realize that you need to get a handle on it. You need people that understand what their jobs are and what to do and how to do it and be willing to learn. Because when everything's constantly changing, you've got to be constantly learning and changing yourself. And this is hard.

And it's what Shagility was all about. Yeah, skills and knowledge. That's what, why we get paid so much. And often I see facades, like, like you talked about. One of the things about methodologies or standardization is it makes people comfortable because they see something that looks familiar. As humans, we look for patterns and we go, seen it before, feel comfortable about it. But we see this facade behavior. So an example I had was worked with a team, the team started rocking it in an organization. They were amazing. Organization, as, as often happens, they see this little amoeba, this little virus start to happen that's working differently, and they try and kill it and they The team had some good umbrella or shit cover. So they managed to survive. They had a leader that protected them from the political norm and the team survived. So then the organization changes and goes, oh, okay, this thing looks like it's working. We should adopt it. But what the organization did, because it was a PMO, Project Management Office, led organization, they brought in a BA to write the methodology on this agile way of working, and that person at no time and talk to the team. They didn't talk to the team, they didn't absorb it. And then this document came out, said, here's our agile way of working. And the team went, what do we need to do? We need to change the way we work to that because that's not what we're doing. And we tried that bit, didn't work for us. We haven't tried that bit, maybe that's interesting. But this organization standardization seems to be the way we're taught, right? Because we're taught to chicken farm where at school, we're taught to follow a set of steps and do it that way. But if we talk about that waste, that idea of somebody just spent 6 months of their life writing a document for their agile way of working that's never going to be adopted. Another example of waste is the team are operating in a certain way and then a project manager has to distill it into a Gantt chart for the PMO because that's the standard way we report across the organization. If we focus on documentation within the team, their way of working, what level of documentation does a team need to do? To document their way of working? Do they document everything they do? Do they document nothing? What have you seen in terms of the right balance of describing the way they work while they're building it?

Once again, it really does depend on your situation. So, for example, if you're in a regulatory environment where legally you have to have a defined way of working, then you're going to probably do a little more writing than if you weren't in that situation. But it really depends on what your team needs and what your what new people would need. So as people come into the team, how do you onboard them? Some organizations will, here's your Process Binder, read this, and then try to stay awake while you're doing it. And then at the end of the week, we'll come back and talk to you, and maybe next week you can start doing some work. At the other end of the spectrum, you just start and you start pairing with people and you learn through osmosis how things work. I've seen processes defined as simply as a single page on the wall, a piece of paper on the wall for collocated team, here's our agreement, here's how we intend to treat each other, and here's our rights and responsibilities, basically. I've seen 50-page documents, several hundred-page documents that nobody ever reads. Even the new people, they get 10 pages in and it's all over by that point. So the problem with documentation, people probably aren't going to read it. Don't write the documentation if you don't intend to keep it updated. And so if your way of working is constantly changing, then You'd be constantly changing whatever documentation you've got about that. Same old rules. Lighter is better than heavier when it comes to documentation. Keep it as concise as possible. Just barely good enough type thinking. I would not invest a lot of time in writing process documentation unless I'm in a regulatory situation that requires it. And always read the regulations. Don't let the bureaucrats read the regulations and tell you what they say. You need to read the regulations yourself because what the regs say and what you're told the regs say are often very different.

Yeah, it's that whispers, right? As multiple people read and talk, we change the language and we change the intent. Yeah, I'm with you. I think it's the right level and that's hard. One of the patterns I've seen that was particularly successful was this idea that the documentation often was targeted at new team members. The problem we're trying to solve is new person coming into the team, we have this unique way of working, how do we describe it? It as distinctly as possible to get them started. And so the documentation focused around that. And the new team members were always tasked with this question that if you came on board again, what's the one thing you wish had been documented or described for you that would have made your life easier? The, oh, I wish I had known that. Write that one. And then what's the 3 things you didn't bother to read? And then over time, when you get enough ticks in that box, you decommission that piece of content because people are telling you it has no value. And then, like you said, the level of documentation needs to align to your cadence of change. If you're constantly redefining the way you work, you've got that high cadence of change, then less documentation because it's going to get out of date. And out-of-date documentation is far worse than none because people trust it. They read it and go, oh, that's the gospel. And if it's out of date, they don't know versus if it doesn't exist, they're going to go ask a team member, how do we do this? And they're going to get the latest version. So I'm with you on that. And so that idea of continuous improvement, we get taught to do continuous improvement in our code, continuous improvement in the way we deploy and the way we build and our data. And that whole continuous improvement is something that's baked into us now when we work. But very rarely do our teams get taught to do continuous improvement in the way they work. Now, if we look at Scrum, that's where the retro came from, the retrospective, right? That idea that we're going to force a ceremony where we're going to force the team to say what went well, what didn't, and what are we going to change. Because we know by bringing in circuit breakers, by bringing forcing functions, people will learn that behavior. It's quite brutal, and it is a form of waste if we think about the amount of time we waste with that team meeting to do that retro versus if they just retrospectively change the way they worked every day. But we know those forcing functions are important. But outside Scrum, we don't tend to see that continuous improvement of the way we work, of our processes being endemic, do we? Or have I got it wrong? Because Lean's all about process review and continuous improvement, isn't it?

I think it really depends. So I would argue that I don't think we really see a lot of improvement in Scrum. It would have changed. Like Scrum hasn't changed all that much since the mid-1990s. It's evolved a bit. But considering how many people are doing it and how long it's been around, it's really stagnant. And I think it shows it. And people have been saying this for years and then nothing happens, be just because of the culture of the Scrum community. It's more about holding a retro, right? So I think the challenge with forcing people to have a retro every week, every two weeks, whatever your sprint length is, that it becomes so regular that it becomes nothing. And are you really improving? Are you really having a retro? Are you really identifying what we should fix and then fix? It's that get off your butt and deal with the issue or issues. That's where it all tends to fall apart. Because otherwise, these Scrum teams would evolve away from Scrum within a few months. I guarantee that. Because every time I've been coaching a team and we actually did retros and I held their feet to the fire, by putting measures in place. So I'm a big fan of measured improvement. If you identify, hey, we need to fix X, whatever X happens to be. Okay, fine. But what are we going to do to fix X? Let's do this. Okay, fine. What, how are we going to measure? How are we going to know that we fixed X? Oh, I don't know. We need to figure that one out, right? What improvement do we need to see? And how are we going to measure that over the next month or two, however long we think it's going to take to fix X? And then do we see improvement? And then, and it's not just declaring success. Say it takes you 2 months to address whatever this X thing is. I want to keep some measures in place to make sure we're not slipping. Because what'll happen is the shiny new problem syndrome. You think you fixed one thing and you really barely fixed it. And then you go on to the next thing and you forget about X and then your behavior slips and you start doing not X because now you're working on Y. And I think this is the challenge with a lot of teams. They really do stay stagnant. It's that lack of discipline around improvement that really gets them into trouble. Like I said, feel free to quote me on this, if Scrum teams were really doing process improvement, they wouldn't be doing Scrum anymore.

I find Scrum's a good place to start. It gives us some really light patterns that we can explain, we can coach, teams can adopt, they can learn, and then they should get the hell out of Scrum. They should go, why are we doing 2-week iterations? Why don't we just make it continuous? Why are we doing retrospectives as a meeting at the I mean, why can't we do it every day? Why are we doing daily standups? Why can't we just continuously talk? But without those forcing functions, without those experiments to show a pattern of learning and then measure whether we've got the success, we don't do any of those things. So I find Scrum incredibly useful because it's a set of valuable patterns, but it's not the answer, it's just part of the journey. And that measuring work versus measuring the way we work. That's incredibly difficult. So, if we're measuring work, it's hard enough. So, we can say, yes, we can see people are busy. They're doing work. Everybody's busy. Tasks are being done. But are those tasks moving any of our dials? In the data world, did that data provide any action, any outcome, any value to the organization? Yeah. Hell, was that data accurate? We don't even measure that half the time because we don't know what accuracy actually looks like. In the product world, in software, we can say, yes, we put a widget on the page and people can type into it, but did that change the behavior of our customers? Did that customer behavior change actually deliver us more or less value to the organization? Those measurement things are really hard. And when we apply them to the way we work, to our processes, it's hard as well. If I don't do this, is that a positive outcome or a negative outcome for the team and the organization? So measurement seems to be, as humans, one of the hardest problems for us to solve. Have you found that in that we talk about experimentation or patterns? Okay, if I apply this, given this context, it should make things better. But measuring what better looks is incredibly difficult. Have you found that?

Yeah, definitely. So the challenge there is, are you really measuring what's important? So for example, in the data world, it gets back to the data is a secondary issue. It's really the usage of the data, the questions that I'm helping you to answer. In the case of a data warehouse, if we're helping people answer questions and make better decisions, that's what I want to be measuring, right? Did we actually help you answer the questions that we promised that we would help you with? And more importantly, have you moved on to better questions? Because over time, things are going to change and your priorities will change. That's what I really want to be measuring. So, in the case of way of working, if it's a technical practice, like some DevOps thing, Then yeah, I can instrument the tooling and I can measure how much are we at, how often are we doing, are we actually integrating and how often are we running the tests and stuff like that. So there's some easy things to do there, but it's usually not the case. So it gets back to the problem with X. So what was X? What are we trying to improve with X? X would be, I want to improve data quality. So then I should be measuring data quality, not the fact that I've adopted some new technique or some new tool. That improves accuracy? Have we actually improved what's important to us? So measure what you're trying to improve or what you're trying to adopt rather than the techniques that are helping you hopefully to improve. And so it becomes fuzzy at that point, right? Because quality improved, you know, I want to improve quality and I changed this, I changed this way of working to help improve quality and quality went up. But we could have also changed 2 or 3 other things in flight as well, which might have also had an impact on quality. We don't quite know, because some other technique that we've been experimenting with was really the source that had a side effect of improving data quality, and we just didn't clue in. So anyway, it's hard to measure. There's always going to be some qualitative decisions going on. We wanted to improve this, the metrics are showing we improved this, it was probably because we changed this way of working. Okay, we'll, we'll continue working that way until it doesn't make any sense.

And it's that macro versus micro. From a data point of view, the way I frame it now is I want to understand the business question the stakeholder wants to answer. Yeah, so it normally starts with how many, how much, how long. Then the next thing I want to understand is if I give you the data or the information to answer that question, what action are you going to take? I want to understand all the customers are going to churn. All right, once I give you that, what are you going to do? What action? Oh, okay, I'm going to go and do a save off Okay, well, if you take that action, what outcome are you expecting?

Yeah.

Okay, I do a save offer, I'm expecting less people to churn. Okay, and if that outcome was successful, what's the value to the organization? Oh, okay, I expect less people churning, lifetime value is going to go up by 5%. Now, that's great. I've got a nice cause and effect model there from the old balanced scorecard days, but Doesn't mean that we can't have gone and done a dumb campaign that caused churn. It can't mean that we took a stance politically in the market that people didn't like and they churned. So there's a bunch of other macro, you know, things that may affect that value, but at least we've thought about it. Now, what I haven't done is taken that pattern and applied it to way of working. The best I get is what thing are we doing? What change are we making? And what action do we expect to impact? We've got a problem. We've delivered some stuff to stakeholders on a regular basis and they turn back and say, that's not what we wanted. Yeah. So typical data problem. Okay. Let's change the way we engage and let's do something slightly earlier, slightly lighter, using a different language and see if that impacts it. So our action is to engage with them earlier and get better requirements effectively. And then I can see that, I can see we have engaged. I can see that they felt that was a good conversation. I haven't flowed that through now to say, what's the actual outcome of that action? And then what's the value and how do I prove it? So, I'm still struggling with that measurement part of changes to the way we work. Still work to be done, I think.

Oh, for sure. Yeah. And just get used to being frustrated. There's a lot of qualitative stuff going on there. And there's so many interconnections that it's really hard to tease out. We changed this one thing, how much of an effect really happened? And there's things that are going on that you're not measuring at all. So, you'll adopt a new way of working because you want to improve quality, But a side effect of that new way of working is you're now ticking off one-third of your customers. Customer satisfaction is going down while quality is going up. But you're not measuring customer satisfaction, or if you are, you're not relating it back to this new way of working because you never thought about it. And this is one of the things I've continued doing, that is when I describe a technique, I describe both the advantages and disadvantages because there's always trade-offs. And I think this is where where a lot of people writing about methodologies and ways of working really miss the boat. And this is something I learned from Kent Beck, like, 25 years ago. When he described XP, he was very clear. This is when this will work. This is why it works. This is when I suspect it's not going to work, and here's why. And I extended that thinking to the detailed practice level of, here's the advantage and disadvantage of this practice. And this is very true in patterns. If you look at the way patterns, or particularly process patterns, are described, there's the context, the original context in which this pattern will work. Work. And then when you apply it, here's the new context that you're in. If you successfully apply this pattern, there's good things and bad things about this new context. This is just the way the world works. When I'm describing choosing your way of working, I describe it in terms of risk. What risks are you willing to live with? What risks do you want to pay down? What risks are you willing to increase and learn to live with? Because you will. There's no perfect solution here. And this blows people's minds. And this is why I am always concerned about this term best practice, because people spin it as, well, this is the best way of working and this is truly awesome. Yeah, in the context in which you came up with it. But what were the downsides of that? Were you even able to detect the downsides, or are you so inexperienced in your own technique that you haven't seen it fail yet? Because it will fail, guaranteed. It will fail at some point for some reason. And if you're not aware of that, You might not want to be pitching it. Be honest about it and say, you know what? I haven't tried it in this situation. I suspect it's not going to work, but I don't know for sure because I haven't tried it. If anybody tries it, let me know. Fair enough. Best you can do. Don't pitch anything as this is perfect and it's going to work for everybody in every situation because it's not. And this is what people imply. They won't blatantly come out and say it that way, but they imply it. All these people want to sell you certification and certain things. They're implying it. For sure.

It's about describing anti-patterns as well. So if you know you've applied this pattern in a certain context and it didn't work, describe that anti-pattern, describe the behavior of why it didn't work. It doesn't mean it won't work next time in that exact same context. Because as you said, there's other contexts around. There's the organizational mindset, there's the way the organization is structured, there's the budget constraints, there's a lot of other things that impact it. But describe that anti-pattern. Patterns. An example for me would be, our stakeholders are frustrated because they can't get any work into the queue. The actual pattern, introduce the concierge. Introduce somebody that's going to go and talk to them early, gather something, put it in the queue, engage with them, don't just say you have to wait. That's a great pattern. The anti-pattern, now what you've done is set a promise that you're not going to deliver because if you haven't changed any other way of working, their work's not going to get done any faster. But you've set up an anti-pattern where you've had a conversation, you've set an expectation, and now you're going to disappoint them even more. So that's your trade-off. Your trade-off is, do you want them to feel like they've had some input and then manage the expectation? Or do you want them to feel like actually there's just no way they can get their work done? Or do you want to find another pattern, which actually fixes the problem, which is the flow of work and the speed you're doing it? Describing those anti-patterns is important. But what I find is describing a pattern in the written word is incredibly difficult. Because I can do it waving, I'm waving my hands now. I can wave my hands when I describe a pattern. I can look at people's faces. I can listen to the questions and I can change the context of the way I describe it. And yet with a pattern, there is that standard pattern for a pattern, which is, here's the thing, here's the context of the thing, here's how you apply the thing, here's some risks or anti-patterns where it's dangerous for that thing. You're trying to distill all this knowledge so somebody else can pick it up and apply it. But there's so much context around it that it's hard to do in the written word. Do you find that? Do you find distilling patterns down to things that you've written it, I can read and apply it, that is an incredibly hard process?

Yeah, it is. And this is why, if you look at the work of the patterns community, they can take years to write up a pattern language. And it's common to take years to write up a pattern language and get it right. The first few iterations of what you think is a great pattern language, tend to be total garbage, and that's okay. But you learn, and they shepherd you through, and it tends to be a multi-year process, but you end up with some pretty good results. And the challenge is it's also so slow, but then you end up with these multi-page patterns. If you read the Gang of Four book from the mid-1990s, there's, what, 25 patterns or 30 patterns in that book, and it was a 400-page book or 500-page book. You end up with these 10, 15-page patterns. Now, they're source code, so blows out the size. These patterns get pretty big pretty quickly if you're doing it right. When DA, we had 1,700, 1,800 practices put into context in the toolkit. So you can't have a 5-page description for 1,800 things, right? That just gets too huge too quickly. So we would describe each practice in a paragraph or two. The trade-offs were bullet pointed, and then we would link to, and oh, if you're, if you're interested in learning more about use cases, is after reading a couple paragraphs about use cases, go read this book or go to this website or go watch this video. And so we would point to the details and leave it up to you if you were interested in that. But because we were focused on just putting things into context and helping you decide, and that's it, not teaching the actual practices. And we still had ways of putting the practices into context to help find— like earlier we were talking about how do you find this stuff? Stuff. You've got to be able to traverse these things called process goal diagrams to get in the right context. So in this context, here are some good options, and here are the— here's what each option means, and here's the trade-offs, and then go read a book if you have to learn more. And that's hard, right? But it's the way it is. Reflects the complexity that we actually face. It gets back to this challenge that people are overwhelmed with the complexity. Even when you simplify it to the point we did, they still get overwhelmed with the simplified version of the complexity. And yet you ask them and it's, oh, it's so big and it's so hard.

Yeah.

Okay. What would you take away? Oh, I don't know. Oh, but now I think about it, you're missing these 5 things. Yeah, we know we're missing those 5 things. We haven't gotten to them yet. But you just told me to make— it needs to be more complicated, not less complicated. I think is the challenge that people just overwhelmed and they don't even want to put the effort in to understand what they're trying to achieve. So I'm in school right now. I hinted at this and I went back and I'm working on a master's. Masters in AI. So a wide range of students, and I'm having a lot of fun, and we're all learning together, and it's all good. A few of my fellow students— there's a huge range, right? And a few of them are a couple years in, so they're younger. And it's interesting, they're in a master's program, and a few of them are anti-education, and it's, I've pulled myself up by my bootstraps, and I learned how to program on my own, which is great. I'm sure they're great programmers, but I can safely say there's this very long list of things that they don't understand, and they probably don't even know they need to understand, or their architecture skills are probably horrendous, and many other things and testing skills are probably not there. And that's what they're learning. But this is the problem, right? Here they are in this very complex domain and learning the hard way that maybe there is a little bit more to it than what they think. So a bit of a wake-up call. And I'm watching another gentleman right now, a friend of mine, and he's writing a book and he's researching this stuff and he's constantly surprised. Oh, this isn't the way I understood it. And he's learning so much, right? He's learning a lot and which is great. And I'm thinking, you know what, like all the stuff that you're learning now finally after being in the industry for 30 or 40 years, that was second year university for me. So it's great that you're finally catching up. Maybe if you've been a little more open-minded to get education, you wouldn't be learning the basics now, at least stumbling into the basics now. I don't know how I got off on this tangent. I think people need to invest in their education and finding a good university program or college program and whatever it is that you want to do is hard, believe me. Me, but it is worth your while to invest in understanding the fundamentals of what your job is, because the fundamentals don't shift all that much. You look at the data community, the fundamentals really haven't changed that much. There's some new toys and, yeah, NoSQL databases, but you know what, object databases, blah, blah, blah, right? And there'll be some new toy to play with in 10 years, but the fundamentals are still the fundamentals. And yeah, there's more volume and it's changing faster, but you know what, in the late '80s, I was dealing with what I thought was a lot of volume and it was a lot of volume at the time. There's a couple more zeros tacked onto the end now, but so what? The technology can handle it. My laptop, I suspect I have more processing power on my laptop than I had on the mainframe in the late '80s. My workstation is more powerful, my real workstation, than my laptop.

And it's about trade-offs decisions. I was talking to somebody the other day and they were talking about their codes for their warehouse, their transformation code. And the question I had was, how many blobs of code do you have? And they said something like 10. And I was like, okay, that doesn't gel, right? The pattern of small number of blobs of code for the size of your organization, the amount of data sources you've got and the time you've been running. So my natural reaction was, okay, there's lots of lines in that code. Yeah, they're massive because they're going from source to consume and the code is the model. And then I'm like, okay, there's some trade-off decisions you've made. You're not modeling, you're not decomposing. There's a whole lot of things that are anti- patterns in that. But there's also value in doing it as one big blob of code. Was that a conscious decision of you or your team? That was my question, because I don't care that you've done a pattern that I don't agree with. I just care that you've done it for conscious decisions, and you can change those decisions when it happens. And the other thing that's complex about describing patterns for me is patterns are patterns of patterns. An example I use is when I describe change data capture, the ability to see that something changed and I want to recognize it's changed. Our English language says CDC is change data capture. There's a pattern. But we also use that exact same term for log mining to say we're going to go and hit a database log to see that change. We use a slightly different language around delta detection if we're going to compare two tables' states and say what changed, but they're all change data capture.

Patterns.

So now I've got this one word that I'm using multiple times for multiple different patterns. And so for me, when you're describing a pattern, it's make sure you put something in the front. Change log, change data capture. Delta, change data capture, and then describe why they're different. Because we often have patterns that are made up of other patterns. Like your friend, I'm a failed author at the moment. And what's been interesting for me is there's been a bunch of patterns that I've used with teams for many years and found teams very successful. And so I coach teams on those patterns on a regular basis.

Process.

When I started writing, I could write what the pattern was, the anti-patterns, the successful application of it, but I had no idea where it came from. What was the origin of that? And the one the other day, I was just updating my article on T-skills or T-shaped, and I was reading your one where you go, don't call it that, call it specialist generalist. And then you had the T-shaped diagram that I use as a template with all the teams I coach. And I was like, oh, Ah, did that come from Scott or did that come from somebody else? And again, it's that origin story of where do these patterns start, where do they get blended? And I'm intrigued by that. Where's the origin of some of these things? I just heard on the radio today that the first zombie movie was, I can't remember, it wasn't Dawn of the Dead, but it was the other one in 1968, right? That was the first time somebody actually made a movie about zombies.

Night of the Living Dead. Yeah. George Romero. Yeah.

But there's an origin story, right? We know that was the first video that was published cuz people probably made some at home. He wasn't, I'm a zombie guy.

It wasn't. It was the first zombie movie of that genre. Yeah, so Romero kicked off what we now— pretty much all the zombie movies are basically the same story or the same basic pattern. But before Romero's stuff, there was zombies, zombie movies where there was like one or two zombies and there was like a witch doctor would bring somebody back to life. And you could argue Maybe the Frankenstein was a zombie thing too, but not really. But, but yeah, no, there was some zombie movie stuff before that. But Romero kicked off the billion-dollar, I don't know what it is, but it's a huge zombie industry now.

So yeah, there we go. Agile data, ways of working. We've got some loves. Zombies movies are one of mine and Zombieland still is my favorite comedic go-to, sit down and relax movie, watch it time and time again. So patterns, patterns are patterns. You go back to the origin story. I'm really interested in that. So there's a lot of complexity in there. And the last complexity I just wanted to cover before we close out is scaling your WOW. Let's say we start off with a team, small team, they build their own WOW. We've now said, okay, we're not going to PMO it, right? We're not going to write it up as a methodology and mandate that's how teams work. But we want to scale the learning from that team and those patterns, and we want wanna share the patterns of being successful with the context and the anti-patterns so other teams can pick it up. How do you typically describe to an organization when they wanna scale their way of working to other teams that are doing similar work? We're not gonna take the way of working from a data team and try and apply it regularly with a software development team, these shared patterns. Let's say we've got a data team and we're just scaling out to 2 or 3 more squads, pods, teams, whatever you wanna call them. What do you recommend to customers? How do they do that?

There's a couple of patterns or common strategies to do that. So for example, I'm having communities of practice or communities of, uh, like coaching teams and tribes— or not tribes, guilds— and I can't remember the other term from Spotify. It's basically another team or person outside of your team whose job or responsibility that they've taken on is to share ideas across other teams. All the people that like test or are interested in testing will belong to the Testing Guild, and then they'll share great testing practices and testing strategies across teams and help each other to learn. They'll be— or having coaches, and coaches going across multiple teams and helping teams also to interact better with each other as well. That's really where the coaches really take off and really have a huge impact is helping teams just work together more effectively, disparate teams. There's definitely ways to do that, but also understanding the context counts. So just because something works for your team might not work for my team, or won't work exactly for my team, and my team's going to have to fiddle around with it. So for example, my software team might have figured out that continuous integration is a good thing. We did everything we need to do to set it up and got it going. And then we share that with the data team and it would fail miserably because continuous database integration is significantly more complex than just continuous code integration. Now, it's not that you can't do it, it's just you've got to do it a different way because you're dealing with persistent data, which the application development team wouldn't be dealing with. You need to remember that when the context changes, you need to expect that some things are going to work, some things are not going to work. Or they're gonna work in a different way, you need to get into learning mode, basically, and experiment a bit and make it work for you. And that takes courage and it takes skill. I don't remember if I ever told you the database refactoring story, but years ago, when we first started working on database refactoring as a technique, we didn't know how to do it, but we knew it needed to be done. And so, we assumed that we could figure it out with no idea of how to do it. And that's how we all stuck to it and we experimented and finally figured it out. Figure it out. It was a bunch of smart people that were working on it and all had pretty good clues, but, you know, we still needed to work it through. We could have easily given up and never figured it out. And this is the challenge, right? You got to stick to it and tough it up, particularly when you see other teams doing it. It's— database refactoring is a bit difficult because nobody was really doing it at the time. If you see another team and they figured out how to do continuous integration, you should probably assume you can figure it out and that you're not special and that you're not in this weird situation where it's not going to work out. It's just that you've got to, your team's got to figure it out and make whatever changes that make sense for your environment.

And go and ask that team because most people will share apart from the methodology people that are trying to sell you certifications so they have to file all their content. We should, if we see a pattern, we should share it because we're using everybody else's patterns anyway. And that reminds me, I was flicking through the Big Book of Wow. Little Book of Wow. And I got to page 28, which is the context diagram. And for people that haven't read it, basically what it says is like a line and it has a left and a right, and there's a bunch of lines, and that's your context. You have something on the left and something on the right, and then depending where you fit, you get to choose patterns are applicable and more likely to work or they're not. And it reminds me, I was coaching a team and we had to figure out their way of working, so we just came up with of 5 lines of things that were important to them. Do they want decentralized teams or centralized teams? How does the organization work? And on that slider, we put a dot where they felt they were, and then the patterns aligned with that. And I found that is a really great way of working with teams to start. And if I'm scaling, if I know we're going to scale, which we always do actually, so we're always going to scale. When we're successful, we wanted more people to do it the way the successful team has. Going back to that slider and say, the team made this decision to use this pattern given this context. That is a great way of visualizing the reason they made that decision. And are you in the same context? Or even go back to them because it's been 6 months or a year since they did it and say, has the context changed? Has the pattern changed? So that visualization of why you're choosing that pattern for that part of your way of working, I found a really valuable diagram. And again, can't remember if I saw it from you and then just used it or I saw it somewhere else.

It's the scaling context factor. That's something actually in Disponential Delivery back in the day at IBM, we were working on that in parallel with some work that Philippe Kruchten, one of the original people behind RUP, was working on. He had something called the Octopus Model and it had 8 factors and we only had 6. And so he was not the Octopus Model. And we basically released at the same time and we were talking to each other and sharing ideas. So yeah, those are the two sources there. In DA, we continue to evolve it and tweak it a bit, whereas I think Philippe has moved on and started doing something else. I'm applying the same sort of thinking to data quality techniques. So what are the complexity factors or the, the factors you should consider when identifying data quality techniques for your situation? I'm still working on this, and the problem is there's so many data quality techniques that you got to rate them, but it's just work. So anyway, so I've got a, I've got a growing database of this material, and what's really interesting is putting various life cycles into context. Lately I've been giving a, a talk on machine learning and building up to a machine learning life cycle. Here's how it works in the real world as opposed to what you maybe have been told. And then I start saying, wait, so where are the data quality issues here? And so I start looking at the, the data flows basically. And for this data flow, these two factors of the five are critical. The other three factors, yeah, they're nice, but I, I don't really care about them. I really care about those two factors. But this flow over here, I care about these two factors or these three factors. And for obvious reasons, right? It's always pretty straightforward. And so knowing that I care about these two factors, how do these 25-some-odd techniques rate? And I've basically got a Python code to generate a quadrant chart. So I said, okay, these three techniques are in the top quadrant. So I probably want to adopt one or more of these three techniques. And these other techniques over here in the bottom quadrant are phenomenally bad ideas. For this problem. But then, oh, but then I look at this line over here and while these two factors that, oh, look at this. So I, there's four techniques that might solve this issue. And two of those techniques were in the bottom quarter on that last chart. So it tells you, you're going to be, one quality technique's not going to get the job done for you. You're going to have several because for this type of initiative, these are the issues and therefore you got to choose the right techniques. For the job. Whereas if it was a data warehousing project, sorry, you got different quality issues there, or at least different quality priorities. And as a result, you'll have different data quality techniques if you want to be effective at it. It takes a while to build up to this, to describing it, but it's a really interesting way of looking at it because you can rate the quality techniques on these 5 things. And I really need a 6th factor but unfortunately, the 6th factor depends on the culture of your organization, which I can't judge. But I can tell you how to judge it yourself and then throw it into the mix if you want to. And the idea is that we can rate all these quality techniques, and then if you understand your lifecycle that you're following, then it's a fairly straightforward analysis to, you know, figure out here's my priorities based on what— on the way I'm working now. Therefore, here's how I'm going to choose these data quality techniques And it's a different set of factors than what we talked about in DA because the factors in DA were of a higher level and you couldn't get down into the weeds. But now, because I've narrowed the focus down to data quality, I can get down into the weeds with specific things to data quality that may or may not apply. Actually, a lot of them would apply for other issues, but these are the 5 factors for data quality.

That idea of allowing somebody to enter in their context, context and then have the machine recommend patterns that may fit their context. That's valuable because as we talked about, we end up with this pattern library that's so broad, so massive, but there's more we're missing. So it's not that we can ever take anything away because every one of those patterns has value. It's figuring out how we can quickly say to a person, what's your context? And then try these because we think these are the most likely. So just looking at time, just to close that out, you've been on a bit of journey in your career from development through to coaching and writing up patterns and building out the DADS stuff, and now back into education again. What is your focus these days, and how do people get hold of you if they want to talk to you about it?

So the easiest way to get hold of me is just drop by scottambler.com, and it's obvious once you hit the site. So I'm focused right now, I went back to school for a master's degree in AI. So that's going to keep me busy, and it's a part-time degree, but that's going to keep me busy until the end of the war. And I'm doing some coaching, consulting part-time as well, mostly around data quality issues, particularly in the machine learning space. Because everybody's getting, it's garbage in, garbage out, right? People are getting beat up on their data technical debt. I'm having a, having a really good time doing that. I do webinars for user groups and chapters and all that sort of stuff, cuz I enjoy doing that.

I've always said sharing is caring, and given the amount you've shared over the career that you've had, I think that shows the amount you care in terms of sharing this knowledge and making it open and freely available to everybody. Again, big thank you for how much your content has helped me in my career, but also just the the fact that you share it so openly with everybody who wants to use it. On that, look, thanks for coming on the show, and I hope everybody has a simply magical day.

Fantastic. Thanks for having me.

That, Data Magicians, was another Agile Data Podcast. If you'd like to learn more on applying an agile way of working to your data and analytics, head over to agiledata.io.