AgileData #15 - Leading agile data at scale

Welcome to the Agile Bi podcast where we chat with guests, or sometimes just to ourselves, about being agile with teams who are delivering data analytics and visualizations.

Welcome to the Agile Bi podcast. I'm Shane Gibson.

I'm Blair Temperaux.

And I'm Chris Lagrange.

Well, welcome, Chris. Thanks for coming on the show. You and I have known each other in the Wellington space for quite a while around that BI data and analytics space, but not so much in the land of agile. So what we tend to do when we have somebody new on the show is ask them to give us a bit of background about where they've come from, where they've been, how they entered this world of data analytics and particularly agile.

Great. Thanks, Shane. And thanks Blair, for having me on the show. I'm really excited and happy to be here. Yeah. So as you say, I've been working in the data and analytics space in Wellington and around the world for the last couple of decades and have worked both at the deep technical level as a developer and as an architect, and also worked in leadership roles and management roles for a number of different organizations and also worked in professional services and consulting in that space. I've worked in predominantly the private sector. I've only just recently stepped into in the public sector and working in a large government agency. But certainly what I've found over the years is that Agile has become very much a key enabler for success in the data and analytics domain. And I think that a lot of the challenges we've struggled with over the years with successful data delivery and data warehouse delivery, has come back to a lot of the key disciplines that Agile brings to the table.

Normally when I'm working with teams, I'm working with teams. So I find that I get bought into a team that wants to change the way they're working or there's a group people that want to do that. We're at the coal phase. We're working with the teams in terms of their mindset, the tools and techniques they use. But often we'll be working with an organization that has started their Agile journey, uses the word Agile at organizational level, but really isn't adopting that business agility. So an example I use often is the funding models. The teams are still funded on project based funding rather than a pipeline of people that do what's the highest priority next. So for me, you're probably one of those more interesting people that are sitting above the team, still in the data analyst space, but above the team in your organization. So how are you seeing that difference between Agile within a team and the way they execute versus adoption of Agile as a business and their business agility.

Yeah, that's a really good question because I would say where I am now, I'm very fortunate that the leadership of the organization has embrace the idea of Agile adoption and an Agile approach not just in the delivery of our technology projects, but across all initiatives across the organization. And really looking at organizing the whole organization around the concept of value streams, the concept of delivery in ongoing increments and ongoing delivery rather than big milestone based drops and things like that. So that's quite key which has brought a lot of those funding questions to the table. And really I first did agile in 2007. I was working in Alaric Telco and we implemented Scrum at that point in time and put together a team of seven and kicked off our ceremonies, kicked off our standups got underway and about a month in things were actually going pretty good. We even brought some of the key business stakeholders into the, into the Sprint team and actually had them on site co located with us and it was all going great. And then I got a call from our finance accountant who said the burn rate for our project was, was five times higher than any other project. And of course it was because the full team was involved end to end. And in that organization we saw that immediate cascade of where we had moved from a classic waterfall model where I had a BA assigned, then I had the architect assigned, then the developer, then the tester, then the operations team, suddenly all those folks were involved at once and the burn rate was so high that the organization became terrified that they couldn't sustain that. And, and so they pulled back on it and we actually ended up pulling the plug on that test three months in. And we actually delivered quite a lot of value over that period, which was the disappointing side of it, which was actually the team delivered a phenomenal amount of value and a really functional solution and the business stakeholders were ecstatic over what they received. And yet we couldn't get support again to take that approach. Because what ended up essentially happening is we spent the same amount of money in those three months that we probably would have spent in 12. We probably got to the same outcome in three months that we would have got to in 12. We just got there four times faster. But all the organization could see as we're spending four times as much money.

Really finance need to be involved in the adoption as well.

They do, and they have to really get their heads around it. And this is where a lot of the challenge that we're unpicking in the organization I'm in now is that training and education of what Agile is, what the Agile principles are, what the sort of fundamentals are in Agile for people that historically organizations haven't even thought about putting on Agile. So we've got our executives, our CFO, our deputy chief execs, a lot of our GMs from across the business now doing Agile training and actually going through the formal process. One of our associate DFOs is normally the CFO stood up at our last PR planning day and said, well I'm now an agilist and I understand it. And a few other people in the room said you're not an Agileist yet.

So out of interest for that is, is the training that those senior people going through the standard training of Agile Mindset, the manifesto, the principles and those kind of things, or is it tailored for. Okay, so what does this Agile thing mean to you and your type of role? So as a cfo we're going to potentially move to a lot more operational costs versus capital cost. That's right, because ideally we've got internal employee teams doing the builds, less contractors, less vendors kind of thing. Right. So is that the kind of training you did, you tailored it?

So it's been a bit of a mix. So the training that I just mentioned, most of that was actually the leading safe formal courses. And from that we've then established a lean Agile center of Excellence inside the organization. And that group is working as an advisory to some of our key governance groups like our portfolio executive committee that oversees our full program of work. And so they are providing that advice on a day to day basis to some of those key leaders in addition to the formal training that we've done. I didn't have control over how we approach that training with them. I probably would have put them on the Agile Fundamentals type training before I put them on the methodology type training. So yeah, so you know, you kind of, you take what you can get, so to speak, and it's great that we're investing in that. Yeah, I think that it takes time for people to get their head around the Agile fundamentals and the Agile principles and I think that'll drop at different speeds for different people as they come to that sort of moment of. What's the old IT term? Grokking Agile.

Sorry, but yeah, as they get that full understanding of it, as they get a full appreciation for it, that takes time in terms of getting our finance folks to get their heads around that. Yeah, it's a learning process. Still, we're not there yet. We are starting to understand that we need to stop funding things based on what I call a one and done business case to deliver a thing. And when that thing's done, we capitalize the asset and we're done. We are starting to look at funding value streams where we assemble a squad, that squad continues to develop against a roadmap, and then as that squad implements, we capitalize assets along the way.

That's actually an interesting one because we're talking about technical debt. And one of the things that I think slightly different in the data analytics space is the value of our data degrades over time when we don't keep working on it. So if we create an analytical model, we score that model, we fit their model, we put that model into production, and we know that over time, the behavior of our customers, stakeholders, citizens, or the data we're using, things will change and the accuracy of that model will degrade. In the old days of analytics, before we had the data science and AI buzzword, we always talked about monitoring those models and figuring out when they need to be rescored, retested, retrained with data. If we think about the old data warehousing days, we always had a team that maintained the warehouse and kept it up, but they also then were the ones dealing with the issues that came through when things change. And so when we're in this land of agile and we're still using block funding models, we don't really bake in that time and cost to keep refreshing and doing it. And if we take the technical debt idea and actually apply that to data, there's a really interesting idea about making data actually an asset. So as we've developed the data and moved it into somewhere that's useful, we take the operational costs of the team that delivered that, we create a physical asset out of it and we depreciate it. And that depreciation cost goes back in as points or something where the team's allowed to use that value to refresh and recycle to keep it healthy.

That's right.

Interesting idea.

Yeah, there's a great quote that I love that it's not mine. That came from a colleague I spoke to recently who said that data is a liability until you make it an asset. And that is very, very true in the data warehousing world. And most organizations will find this true in their core systems as well. And to your. I mean, that ability to actually expose and understand the value and then realize the value is Where a lot of organizations struggle. I think the job is done when the data is collected and made available and it's not. And that's where I think Agile gets you a ways down that path. I think that comes back to business culture and IT culture and investment making. The organization I'm in now has always used its capital depreciation to fund the reinvestment into its assets for life cycle maintenance, but not really from the concept of continuous enhancement of those assets. What we actually ended up doing organizationally is centralizing all those capital depreciation funds back to a central body and then allocating those out to our program.

We've moved from individual capital funds and individual capital allocation to funding capital as a stream across the work program. So instead of looking at this business case, allocates this much capital and you get this much money and then you're done. We are starting to look at instead going, well, we know we have, I'm plucking imaginary numbers out of the air. I'm not using our real ones. We have $20 million a year of capital that we're going to spend. We're going to break that into five program or four program increments. Each one of those is going to be $5 million. And then we're going to decide what we're going to do inside that program increment that's going to add up to $5 million.

So bringing a prioritisation framework at that level to say here's your scope, which is effectively this many people for this amount of time.

Yes.

What is the top priority things right now for this increment that needs to be delivered?

Yeah, that's right.

That's the trade offs.

Exactly. And it also means for us, we've actually stripped away a huge amount of the front end work that goes into conventional project funding and capitalization planning. So we have a principle where if the work is under $2 million in total cost, we only require about a 5 page lightweight business case to get that funding released and get that work underway. And then we just track the spend PI by PI, look at the forecast, look at the tracking, and then we manage variations as we go. And generally speaking, that doesn't mean we've got a whole bunch of $1.9 million programs. It actually means we have a whole bunch of $100,000 projects and we're generally managing capital inflow and resource inflow. And actually what that then teases out is the real organizational constraint, which tends to be our people's ability to complete the work and that then drives a much more tangible and realistic prioritisation discussion at the organization level. Because typically it's not prioritization based on what has the most capital benefit, it's what's achievable based on the resources that we have.

So you might find that something is a high priority, but the skill set's tied up with something that's a higher priority.

That's right.

So therefore priority number five is the one that's achievable with the skillset or the team that's available.

Exactly.

Cool. And that almost sounds like you've kind of taken the Gartner type. 1, mode 1 mode 2, theory of fast lane, slow lane, but changed it so it's actually usable to low governance overhead because of the risk and the dollar value and that kind of stuff and a lot more visibility.

That's right. And culturally it's also meant we can give our teams the confidence to try something, see how it goes, and then come back if they need to do more without having to know all the answers at the very beginning of the project. And that I think is fundamental to Agile. If you don't have that ability to put your trust and confidence in the team and then know that you can control the throttle.

That is where I think a lot of organizations fall down when they try to move into Agile. They think, I still want to know what it's all going to cost up front and I want to know what I'm going to get for that. And I'm going to want to know what my rate of return is and what my return on investment is at the beginning and only then will I release the funding. Instead you go, well, no, I'm just going to give you 10% of what you asked for and then you're going to tell me what you deliver off that and then I'll decide if I continue or if I decide to stop.

Do you find that those smaller value projects, so the 100k ones where you're doing a lot more MVP or Russian spikes, so you know there's a bigger piece of work you want to do?

Yes.

Some people quite rightly are asking some questions that you can't answer because you haven't done the work or the research or thought about it. So you go away and do a small bit of work and the funding model to then come back where the outcome actually is a set of answers to the questions that were unknown.

That's right.

And a slight more surety that the plans got able to be executed.

Yeah, we are absolutely seeing More of that. That's where we're actually, I think now at the next stage of the challenge we're going through, which is how do we get more MVP thinking into the organization. So that to me, as much as your agile teams and your program group need to understand that perspective, the real change you need to see to succeed in that is in the rest of the business, the rest of the organization, they need to think about their problems differently. They need to think about their problems as what's the next thing I can do in this space to get me further to the solution rather than I need the complete solution. And I need to. Because it's sort of like the old agile metaphor, build a scooter then, or build a skateboard, then build a scooter, then build a bicycle, then build a motorcycle, then build a car. Well, if you go into a car dealership, they're not going to try and sell you a scooter. And so most people are still trained for the I have a need, I'm going to go and ask for a solution. And I've already conceived in my mind of what that solution is. So if you tell me that I should get a scooter first and then you give me a car later, they're going to go, no, no, no, no, no. I know I'm eventually going to need a car, so I'll just buy the car. And it's breaking down that mindset. So in our world, that means if I say I need to give you a full customer feedback, closed loop solution that tells me what all of my customers experience is through each of my channels and you go, well, can we just do that for one channel? I go, but I'm going to need it for all channels eventually. So can't you just build something that does every single channel and that's that challenge you get into.

Have you noticed with more regular demos.

Mindsets change?

It does.

People are seeing something regularly.

Yeah. Once you prove that you can increment and that you prove that they can get something that they can use.

Yeah. So the question of what's next is.

That's right. Yeah. But that's a leap of faith for a lot of people to sort of step back and say, I'll let you go, you'll come back to me in three months and you'll show me something meaningful that I can actually use. And that's also where the other challenge is that, you know, you still have to get your own technical teams thinking that way as well. And in data warehousing, in Particular there is still this mindset that we have to have all the data first before we can do anything with it. So there's this concept of well yeah, I can iterate but I still have to do a massive data integration build before I can actually deliver anything off the top of it. And the challenge with that is it still adds oftentimes months or years to the cycle before you can get to a prototype. So you've got to change that. Thinking back to what an almost use the term you need that data steel thread of what is the key thing I want to answer how do I integrate in a way that gives that answer without worrying about all the other dependent data sets and assets that come through?

I call it thin slice. Right?

Yeah.

How do you thin slice it so that something gets in front of the customer?

That's right.

And it's workable in production?

Well, it should always be production ready. Right. So for me, definition of done, we've tested it, we've documented it, these things as professionals, we should always do it our job. And that's our definition of done.

That's right.

But yeah, how do we get something to somebody else that has at least some value even if it's really small? Because we knew that it's 90% back end built.

Right.

We're doing the initial used to be called Sprint Zero. Right. And I'm a great fan of Sprint Zeros anymore. But I was, but not anymore. And often we sometimes guess and guess wrong. So an example I've got is one of the teams I was working with. We all guess that CDC change data capture off a database, the core system was the highest priority.

That's right.

So that's what if we had gone into a Sprint Zero, we would have built that capability. The first information product we got given as the highest priority for the organization became a deliverable that was an emergency because a project had just gone live and delivered no data and reporting because everybody descopes it. What a surprise. And that source was an API.

That's right.

Because it was a software as a service app and there was no way of getting access to the database.

So you can't consume the entire applications and data model. You have to choose that fee. They almost think of it more as a stream of one data set. Yeah, exactly. And that's a mindset shift for a lot of people. That's where the move to cloud and the move to software as a services is spinning a lot of data teams because they want to be able to fully consume a data model and they want to create a CDC structure against it and raise that. And there's a philosophy I've developed. We're not a philosophy, use the term surprise earlier. Blair. I have what I call the classic three surprises of data engineering. And I think agile goes a long way towards helping those, but those three surprises. So surprise number one is it's always going to be harder to get the data than you think. So when you talk about CDC systems, when you talk about moving to cloud, this is actually getting harder.

Getting the data that you need in the format that you need, it is going to take longer than you think. It's going to take more time. And so if you're trying to get all the data you need up front, it's going to slow you way the heck down. So you really need to think about what is the data. I really need to do the first mvp. I'll just focus on that and then I'll go back for the other data assets. A lot of practitioners have a tendency to focus on, well, I know it's going to be hard to get the data, so I'm going to try to get all of it so that I don't have to go back again. And you end up with these data warehouses that are almost a complete replica of all of the IT systems in an organization.

So we call that Data Lake now, don't we?

We do, yeah. Yes.

Data Lake is. You know what, that was really hard. So what we'll do is stop transforming it so that it's easier for people to copy Data Lake. That's another thing I can sell as a consultant. No, sorry, I sounded jaded there for a moment. So that's surprise number one. And that's the big challenge. Surprise number two, the quality of the data will always be worse than you expect. So once again, that means the earlier you expose the data to the end user, the sooner you're going to see some of those problems surface. And I think of data quality in a number of different ways, but obviously there's this quality and structure. So if I've got a field which is first name, last name, and I'm loading Chris LaGrange. Well, oftentimes I see systems accidentally attach the LA as a middle name rather than as part of my last name. So that structure gets wrong. Data warehouse teams have gotten pretty practiced at catching those sorts of anomalies. But my name is L A G R A N G E, not L E G R A N G E. But I'm the only person who knows that. So until an Ended consumer who actually can look at the data and say, yeah, that's right or wrong, knows that's wrong, you won't see that. So the sooner you expose that data and the more regularly you expose real data to real users, you will only then start unpicking all of those data quality problems. Because once you get over the access problem, the next thing that slows you down is your quality problems, your data quality issues that you expose in all of these systems. And that gets you into your third surprise, which is that every time you put data in front of business or in front of an end user, the requirements will change. And we treat that as a surprise when it happens. But it's actually a benefit of good delivery, because in our world, people need data because they don't understand what's happening. So when you give data to them and they suddenly can see it, they learn from it and they build a deeper understanding of it and their understanding matures and evolves and they start asking more questions. Those are positive. Oftentimes we're trained in conventional IT systems engineering thinking to think, no, we have to have all the requirements at the beginning and then we're delivering to those. What Agile gives us is the ability to say, we know we don't know everything at the beginning. We know we want to iterate, we know we want to flex. And one of the big powerful things Agile lets you do is say to the business, we're going to put the data in front of you, you're going to change all your requirements, we're going to embrace that change because that actually improves the solution longer term.

Yeah, I think the way I think about it, and we're doing a bit of research around using Chatbots as an interface instead of a BI tool, because that's a new hot area anyway. But it looks like it won't. That's one that actually might have value to end users.

Yes.

And so the way I think about it is, I think about it as my next question. So if we think about the requirements change, in my view a lot, because somebody will say, the question I have is, how many customers?

Yes.

Right now, that's just their first question. It's the first question they have in their head that they can't answer and they can't make any decisions without understanding that question and the next question. So you give them that and then they're like, excellent, actually, I need to know where they're located. It's not a change of requirements, it's just until they understand the answer to the first question. They haven't really thought about what the next question is.

That's right.

So that iterative approach that Agile gives us, where it's. Here's the answer to that question. What's your next question? And if we team up with some form of automation or data ops as it's called now, we're adding the next bit should be slightly faster than adding the first bit, if possible.

Yes. Right.

We get some speed from that point of view, then we can behave in a way that meets their needs quicker.

That's right. And that comes back to we've got to get better at the things that inhibit the ability to iterate rapidly. And your definition of done that you talked about earlier, I think is quite key to that. Done means documented, tested, productionized, rigorous, auditable, traceable, full lineage. All of those things need to come with the solution out of the box. We have tools available these days that automate a great deal of that and can speed that up. But that is also where teams often get bogged down, is in all of those. They'll think, oh, I'll give the business a spreadsheet with the data, or I'll give the business a table I've populated manually with the data. And then they'll have to wait another six months before I automate and document and harden that entire solution. And once again, that's where we fall over. So that ability to break that down, but also to deliver at speed without creating technical debt as you go essential. And that's not just true for data and analytics teams, that's true for all IT systems teams these days. And I actually think that's where most of the drive towards cloud is coming from. It's less about the cloud costing model, which has some benefits, and it's less about cloud as.

A somewhat turnkey solution. I think it's because actually in practice people are finding that that hardening and that ongoing application management and that evergreen maintenance is really challenging for most organizations to do. So they're looking to cloud to provide all of those things so they can get back to focusing on the value.

Do you find though? Because every data analytics team I'm talking about now is moving to the cloud with the quote marks around it. And for some reason, and I see this in testing, the data landscape or the data domain always seems to be behind.

Our peers on application development, hardware management or compute, all those kind of things in terms of tools and capabilities with rigor. I'll give you an example. In the old days we had VMware and VM servers and those kind of things. And I've seen a move to cloud with kubernetes and those kind of things and a lot of that rigor is gone with it. If I look at some of the, what I would call legacy tools, so I apologize to those vendors, but the data stages, the Informaticas, the Sasdi studios, the Oracle odis, those legacy ETL tools, in my head they had a lot of rigor built into them because they've been around for so long and they just had added features that made everybody safe, made life easier. Automated some things. Lineage classic 1 yes. Most of the teams I'm working with now have moved to the cloud and they've become far more open source centric. And so the benefit of that is no upfront license fees, a lot more flexibility. If you architect it right in terms of automation and data ops, you can swap pieces in and out. When you find something slightly better, you get access to the source code so you can add features that you need without going into a 12 year release cycle. But the downside is those tools have the capabilities that made us safe.

That's right.

Like an example Blair and I were talking about before you arrived, moving from a relational database to hold Data to a NoSQL name Peer key data store.

Yes.

We lose a whole lot of things that in the data world we got for free. Well, it was in the license, referential integrity. We relied on it often because it made us safe and it was just there. Do you find that in the data analytics space? The move to the cloud has got heaps of value, but we actually lose some things that made us safe for a while that we have to reinvent or re implement.

Yeah, we do. Yeah. And I'd say that's an absolute key challenge that we're seeing as well as you move to the cloud, I think it really comes back to the fundamentals. Know why you're doing it, don't go to the cloud because CIO magazine said you should be in the cloud or because there's a blanket policy for your organization that we're going to be cloud first on a everything those things are there. I think they're good guidance because there's a lot of benefit from the cloud. But know what those benefits are, know what you're trying to achieve with them and don't just do it blindly thinking it's just going to be better. Because you're right. I think the cloud is very much where we were. The cloud technologies that we've got in the space are where we were in data and analytics probably around the early 2000s where there was an explosion of vendors and capability in the marketplace. And I think that explosion now is an order of magnitude bigger than it was back then. I forget I've lost track of how many data data vendors Data Flock is tracking. I think I want to say it's in the 5,6000 range now. There's a huge number of them. What we're going to see, I think in the next few years is a collapse in a consolidation again, much as we saw in the late 2000s with a lot of the big vendors swallowing up a lot of those smaller integrators. Synopsys goes to Oracle, Cognos, Cosmos, and.

We'Ve already seen that with Salesforce buying Tableau, that's Right Click buying Tunity and some others. We're seeing the consolidation of of the thought leaders getting bought out by the big incumbents.

And that's kind of the nature of the cycles that we see. And it's very true in the open source space because in the open source communities that's where a lot of the innovation is coming from. It's where a lot of the new ideas come from. But most of the open source products out there are a very smart and very capable group of people working on solving one problem. And in data warehousing we have about a thousand problems to solve to build a solution. So we're now back to the putting all the pieces together in order to get there. So, oh, I've got a streaming solution that gives me my feed, but that doesn't give me my auditability in my lineage. So now I need a metadata solution that can catalog and track that alongside it. And I've got to put all those pieces together again. What we saw a decade ago was most of the big vendors were starting to bundle big application suites together to actually solve that problem. So you'd buy e business suite and you'd get all of that in the box, so to speak. And some of it was good and some of it wasn't, but you knew that you kind of had a full set and now we've got to go back to building that set ourselves again. And you're right. I think a lot of teams are struggling with knowing all of the pieces that they need to pull together and how they do that. And it is adding that complexity back into the equation. And that's a big learning curve for a lot of these technologies as well. And we're back into the old challenges of interoperability and compatibility and standards.

So all of that comes back through again. I think that's actually unfortunately what keeps guys like you and I at work.

I always say I feel it's a bit like the stock market market. Well, the financial market. There's a seven year cycle in it. The bit that makes me really grumpy is I know there's a seven year cycle and I've seen it for three cycles.

Yeah.

I still haven't figured out how to make billions of dollars out of that cycle. So I get quite grumpy with myself because I know it happens. And I can tell you kind of the pattern that we see and I still don't know how to leverage it.

I think the secret is to already have billions to spend.

Take some risks.

Yeah, yeah. So if we go back to one of the things you see really early was this idea that you put your senior people through some of the base training and leverage that. And I think what I heard was it was kind of like a taster, right? It was a. Yeah, it's a bit of an effort. And some of that stuff won't be relevant to you, might be a bit too detailed, but it's an immersion that gives you insight that will then have you asking more questions that are relevant to you and your role and what it means to you. And then you created almost a coaching office is the word. I know it's not word you use, but that's effectively a bunch of experienced coaches who could then engage with those people and have a conversation at the right level, at the right time about the right thing.

That's right.

To help them answer the questions that they have each time.

That's right. And alongside that we are adapting our governance structure and our governance principles and practices as well to identify those challenges. And we haven't solved for all of them. One of the big ones we're working through at the moment is we, like many organizations, have some teams that are purely project funded and those are actually the easiest ones to move to an agile funding model because largely speaking they're all being charged to the projects being charged for the capital. That's fine. We have a group of other teams who work in oftentimes more operational roles that occasionally do project work or part of the team is generally doing some project work and part of the team is doing ops work. And we're struggling a little bit in that space to get those teams into the full sort of agile model once again because a lot of their work is much more reactive. And there's also those lines of business that have their own sort of dedicated teams delivering services to them as well. But those teams also provide some services to the rest of the organization. So that's where a lot of BI teams sit. Oftentimes you'll find a BI team has some key business groups, oftentimes marketing or finance, who are sort of those most important stakeholders that always get what they want and then they deliver stuff to everyone else as well. And we have those sort of challenges that we're unpicking and trying to understand how we continue to enable some of those key value streams to occur. But at the same time make sure we're still prioritizing across the organization and make sure we're still looking at the whole organization comprehensively and not letting someone's, to coin a phrase, someone's little private force achieve their goals, which may or may not be the most important thing for the, the organization to achieve is quite key.

And because the organization you're working with is so large, you must have that scaling problem where at a senior level the vision or intent of the organization, the strategy or what good looks like over the next five years is probably pretty well known.

At a certain level. I hope as you start cascading that message down, as you start scaling to teams, then that it's like whispers, right? The message gets changed.

It does, yeah. And I think that's, I mean, look, I think that's true in our organization. I think that's true in every large organization I've ever worked in. The more layers you have separating your leadership from your execution, the more it becomes, the more structured you have to be in how you message what your strategy, your vision and your goals are and how that works. In a perfect world, everyone would have a strategy map which would go from the high level key objective for the organization down to each line of business, down to each unit to a measure that's in somebody's performance plan which tells. And if you crank out 15 widgets a month, then we'll achieve our KPI for the year. And I've never seen that. In reality it doesn't work that way. In reality, in reality it's kind of setting a direction and then everyone else finding out how to get there. And I think that's when we look at Agile, it gives us that ability to course correct much more rapidly, I think, but it also gives you an opportunity to go the wrong way a heck of a lot faster.

But I think we also get some patterns right. So we.

We were talking to a while ago about okrs. I think it was Andy Cooper and.

Then Blair and I went to the.

JAFIC conference, Agile kind of group meetup thing, two day conference that Nomad8 ran last year. And there was a whole conversation around OKRs there, which was quite interesting. Another one's value stream mapping. Again, that idea of seeing end to end across an organization, what's happening and where the value is and where you can tweak it. So there are some patterns that we can leverage that help with that, that shared understanding of what good looks like as an organization and where we're going.

That's right.

It is hard.

Yeah, it is hard. And it is. And we are doing that at a comprehensive organizational level now. We are moving to value streams that go all the way from frontline to backroom it. And we're bringing those teams together to work side by side on solutions in a joined up fashion. Really early days for us in that space. But it's really exciting as well to see that actual that change happening. And as I say with Agile there's the methodologies, the frameworks and the practices and those are good, but I actually look at those and sort of say whatever your methodology is, get good at your methodology.

Just be good at what you do and solve for that. Because even if your methodology isn't perfect, if you're good at it and you know how to make it work, then it'll probably get you where you need to go. There are no bad methodologies, there are just bad fit methodologies for different organizations, bad implementations.

So again with that JAFIK conference, there was a really interesting conversation and what it was was so they worked on, I can't remember what it was called, but the idea that everybody could throw some topics up on the wall, there were slots, you have a certain number of topics and then you created your own little space and then you could go and be part of that topic.

And if you open space.

Open space, yeah. If you got bored, you could go to something else halfway through. You're allowed to leave halfway through. That's okay, right? And if you. All that kind of stuff and one of the ones was an interesting conversation about the state of the Agile market in New Zealand with Scrum masters coaches and vendors and Big five. And it got a bit heated. But one of the interesting things that came out of that was this perception based around the case study of McKinsey and Spark. And so there was a statement in that around 3,000 people that made redundant Spark because they went agile. And that's a really bad thing for the Agile brand. The angel mindset. There were a couple of people from Spark actually there, and they said, really? Actually, it's not true. There's about 300. Right. So pretty handy conversation, which was interesting to watch. But later in the year, I'd kind of gone to some Agile meetups and I met some people from Spark and I was talking to them and what they said was as part of their move to being an agile organization, and they took more of a Spotify approach with tribes and chapters and those kind of things, they quickly realized that they had somewhere like six to seven levels of middle management of team leads. And in the new way of working, some of those people would move into a chapter lead or a try. You know, they had roles and they had skills, but even after they'd done that, they figured out that there were more people than there were roles or things to do. And so as a result, those people.

And so I look at that and.

I go, okay, that feels to me that the organization has made a commitment to try and change the way they work. So you talked about PI planning, right? So I'm assuming that your organization started the Safe Journey.

Oh, yeah, we're in PI 13.

Okay. As I've probably said on the podcast, we haven't recorded one for a while, but I've become far more vocal about. I have lots of concerns around SAFE as a methodology because I see it being flagged as a methodology, not a way of working that has value. And you can tweak it more as here's an out of the box solution. And if you follow all these things, but there's lots of things in there that I think have real value from what I could see. So things like release planning, if you have more than five squads and you're working on the same thing and near the same day, the idea of a release train and being on that has value. The PI planning, the idea of getting people to grab and seeing from a shared vision point of view how you're going to go over a period of time, still struggle with 500 people being in a room is valued, but I can see some value in that process. But the other one that I came back to was, especially in Wellington at the moment, SAFE kind of indicates that the senior people in an organization have made a commitment to change. Because it's an immersive methodology, the organization should change the way they behave. Is that what you're finding? I am feeling safe.

I think to me, safe is an opportunity to cause that change to occur. And sort of many years ago, I was working as an architect and one of the fundamentals in enterprise architecture was almost the if an organization is centralized, decentralize it. If an organization is decentralized, centralize it. Not because either of those frameworks is better than the other, but because the change brings the value and the process of change flushes out all the inefficiencies and it flushes out the redundancies and it flushes out the other things that the organization has probably accumulated over time. It's a horrible thing to say in a lot of ways, but looking at the change that Spark went through, I think that agile was the change, but it wasn't the reason that they went through that structure review and they had redundancies. I think that that was a secondary effect of the change happening in the organization. Not we're doing safe, so we need less people. I think it was actually the change triggered a review of the composition of the organization and that happens. I don't care what methodology you go to, if you actually embrace it and you decide that you're going to change the fundamental structure of your organization to be successful, then yeah, there's going to be disruptive change to the organization.

With that organization they're in a fast moving environment with new products, new technologies and it's been a large organization for years, Spark Telecom and it's had constant change in terms of structure and that. So again, it was interesting that it was in Wellington. There's some perception it was the Agile badge that caused this. Yeah, but you could actually go through.

Five restructuring, you know, the telecommunications regulations that were made in the, in the, you know, late noughties and the, you know, the, the structural separation caused far more change than that. And you know, so yeah, and then, and, and then that organization and many organizations, you know, changes, unfortunately in any big organization is an inevitability to an extent. All organizations over time, you know, you tend to see organizations contract into a new model of working and then over time they expand out again. They have to scale and yeah, they scale back up and then they go through a contraction again. Unfortunately, that seems to be the nature of corporations and I don't think that that's an agile problem. I think that's just the way that these things work.

Let's look at Data World, right? We had staging areas in the old warehouses where we put all the data before we transformed it, made it available. We treated that as a locked room. So all the analysts would go out and buy OLED cubes. So then we kind of made things a bit more flexible. And that was great. But then we started locking it down again. So everybody went out and built data lakes.

Exactly.

So we go through those cycles.

Even it is that natural cycle, it's that it seven year cycle again. And it's just how it presents itself. So to me, if you develop any skill as an agilist or as a BI person, the biggest skill you can develop is the ability to adapt and change and to anticipate not just the little changes that happen day to day in your job, but the big changes that are going to come along and how you deal with that. That's Darwin's debris revolution in a nutshell, is your ability to adapt to change, is your ability to survive in that environment. That's pretty much the core of what we're touching on there I think. And I think for an organization you can't not do something because it might affect and cause change. You have to think about what you're trying to achieve and you have to pragmatically look at what you need to do differently and how you need to work. If I look at the IT sector in New Zealand, if I look at the BI sector in New Zealand, I don't think anyone who's in BI today needs to worry too much about their job. I think there's a continuous growth in this space. We're in one of the most high demand professions in the world and that's not a bad place to be. But I think that we can't rest on our laurels there. I think we have to continuously look at how we get better at what we're doing. Because over time what I have seen is that the old way of doing things has given way to new methodologies and new practices and new tools and technologies which make it much faster, much leaner, much more ability to. We have much more ability to do things at speed. If we're still doing things now the way we did them 20 years ago, we will become obsolete. And that's where I'm seeing a lot of the challenge in a lot of BI teams as well. Agile is kind of causing them to change. But if I look at most of the things that Agile, the BI teams are struggling with, with Agile now it is an Agile that's causing problem for them. It's actually that a lot of their practices are still very old school.

Come back to automated testing.

Right.

You know, we still have to build our own automated testing capability for data and there's not a lot of reasonable patterns out there.

No, there aren't. And a lot of Data warehouse teams really have a hard time even getting their head around what testing looks like in a data warehouse environment, how to automate it. And you know, and that is one of the challenges, I think in New Zealand we struggle with that. I think that's a global problem though. I think good practitioners in the different domains with those key disciplines that understand what needs to be done and how to execute it successfully and also then how to convince the folks that are managing them, leading them and funding them that these things need investment in order to be successful is also quite key. It's funny, you talk about testing and one of the things that I've seen many, many times in my career is that management first pushes back on project costs because the projects look like they're going to be too expensive. So oftentimes what teams will sacrifice is the intangibles, the non functionals, the ability to do reconciliations, the ability to do replay and reloads, all the technical debt that gets created from that, all those things that a good testing and quality framework would give you.

Or if we look at ETL tools where they're flow based, so the informaticas of the world, or the new Apache dags, airflow dags and those things we start off with, each of the nodes in the flow is a distinct thing. It has none in and out, it does one task, it's all repeatable, replayable, and then we get under pressure. So you start seeing code nodes, you start seeing an in and then a big block of code and then an out. And so we lose the value of any, we lose all the value of our automation.

And you often lose your replayability there because developers are optimists and they tend to build code built to succeed, not code that's built to cope with failure. So yeah, you get into all those situations and that's it. And so unfortunately what then happens is the system goes into production, we have regular batch problems, ETL problems, other issues come up and leadership turns and says, well, you guys aren't building this very well, are you? And that is the constant defensive pressure that I see in data teams. And I think that's where as practitioners.

We'Ve got to find ways to, in an agile world, but in a cloud world as well, continue to maintain that, that focus on a good quality, robust solution that runs well, but that is actually delivering value regularly and continuously. And that's the risk I see in agile for data as well, is that we get into this mindset of just give them the data as quickly as you can, as Often as you can and strip away all of the non essentials. Well, those non essentials are actually essential and we have to continue to do those too.

Or if we're not going to do them, then it's a trade off decision made by the product owner stakeholder. And it's a clear one.

Yeah, it has to be. Get it in writing. But yeah.

Well, acceptance criteria, I mean, I've got an example of one organization that effectively their analytical models are disposable. They have a team of 10 data scientists and the acceptance criteria is a model was built quickly, the model was accurate, the model was run and then you move on to the next model. So they've prioritized speed of model speed to market over reuse and automation. And that's all good, right? Until they carry on with the conversation and, or, but which is. And it'd be great if you could run that model for three months repeatedly. And it'll be great if the other dynasty scientists could pick it up and reuse it. And so then it's like, okay, so now it's not going to be as fast as a disposable model because we're building all that rigor.

Right? That's right. And then you have a change management exercise as well. And so you've suddenly got the need to understand who all of your users are who are impacted by exchange that comes through. It's funny, you talked about release train. Well, one of the biggest things I think data warehouse teams get out of that whole mindset and that whole shift is in release trains. Finally there's a place they can generally go to see what change is coming from all their core systems that's going to disrupt them. So many data teams I've worked with have struggled with the constant disruption of their source systems, making changes that they don't know about. They find out about the 11th hour. That's where I think we talked a bit about PR planning. Those PR planning days. When I take my teams to those days, they're kind of on two missions. One is to receive the work that our team needs to do and to work with the other teams that are asking us to put story points on our board. But the other is to walk the room and see what else is going on and see what's coming and say, hey, you guys need to come talk to us about that. That's going to cause us a disruption if you don't actually mitigate the impact to us along the way. And that's been quite a powerful tool in our toolbox to ensure that we're actually getting better at seeing what's coming, seeing the lay of the land, seeing the IT change landscape and anticipating it far better than we have in the past.

So think of it as collaboration at scale.

It's collaboration at scale, yeah. But it's collaboration that is actually, it's transparent and it's open. It's an opportunity to literally walk around and have a look and ask questions and find out what's going on. Which is a lot harder when everyone's at their desks working at different speeds and different cadences and stuff. So. So as you say, I think safe can be incredibly prescriptive and is kind of treated as almost a turnkey agile methodology. And I think for that, I think that is a limitation. I think organizations should take from it what value they get from it and leave on the table the things that they don't think they'll get value from. Like anything else.

It is not a one size fits all. Even the scale agile folks would say it's not one size fits all. Yeah, they give you their four tiers of scale, but certainly even within that, how you implement those practices is very organizational and cultural based decision making that you've got to make as you go through it.

And if you wanted to bring in, you know, my viewers, if you want to bring some of the Spotify patterns. Yeah, yeah, then use them. If they fit, if they work, you know, if you want to, you know, if you want to bring in something or innovate on something or iterate on something because you think you can change that and make it slightly better, that's great as long as it doesn't become, like you said, out of the box, fixed. You can't inspect and you can't adapt the way you work because it's just the way.

Well, that's just that. And even, you know, we're to able, we're using safe in the organization. But I've also adapted guilds and squads into my unit and we're using those because from the Spotify model that's actually a bit of a gap that we saw. And I think that guild model is fantastic for making sure we maintain cohesion across the different disciplines that we have in the organization.

Yeah, it's useful, it's worth the effort. And then I've used a lot of the patterns out of discipline, agile, the dad, and then now they've moved to the pmi, you know, it's like, okay, are they going to become a safe equipment, you know, or are they going to retain the beauty of what I see of what they do, which is, you know, a bunch of patterns that they explain particularly well. That means I can look at the pattern and go, oh, I get it. I think let's have a go.

Right, yeah, I'm a big fan of that as well because I think what they really allude to, what that really focuses on, is exactly that. Be explicit about the reasons why you're doing things the way that you're doing them them, know what the options are and make an informed decision about how you do it rather than just arbitrarily deciding that you're just going to follow this thing because a consultant told you that's the one you should buy. And you know, and I think that's. Yeah, we. Because once again, that's. We're also using discipline agile inside the unit and we're also using, you know, and we, we've got the ways of working book and we go through it and we look at the. Oh, we're going to do a vision. Well, here are the different ways we can create a vision. Which are these approaches are we going to take. How are we going to do this for the, this particular program and how do we construct it? And I find that's actually an incredibly useful tool for even just knowing which questions you need to be asking as you go.

Again, one of the customers I'm working with, we've been using information templates, product templates for a while, worked with them and we iterated on and came up with Information Product Canvas. And that's had a massive benefit, actually, for a whole lot of really unexpected reasons. And right now we're trialing and working through the idea of pressing releases. Y so again, right, there's less content than an IP canvas, but, you know, it's all useful information and we're just trying to figure out where it fits in the life cycle. And it'll either come out as having value for the way we're working there or it won't. Yeah, but the fact that you can find some information around, well, you know, for a press release for, you know, for a normal Amazon product, you know, it looks like this. How do you adapt that for data? Is it. They're much different. Not really.

Yeah, that's right.

Thinking of the words that go into it, that hurts. The boxes themselves aren't that much different. So again, it's that reuse of patterns.

It is. And it's that ability to actually know, even just.

It'S a fallacy to assume that everyone already knows the right way to get there, and knowing what the options are, knowing what the approaches are, knowing what other organizations have done, having resources you can go and look at to show you what all the different ways of doing it are, is incredibly useful because I don't think anyone could actually say hand on heart, they know the best way to do this job. Data warehousing, data analytics, BI. You know, there's actually 100 different ways to do it correctly or more. And it is important that you're explicit about each step that you take and how you do it. We're using the canvas approach for our continuous improvement framework that we run. So alongside our program increment schedule and our agenda, we also run a CI canvas and board. And we are running a number of continuous improvement initiatives at a business unit level and at a team level and at a squad level and at a management team level. So we're looking at the big things that we need to make shifts on from a continuous improvement standpoint. And we're looking at small incremental changes as well through that framework. And yet we've used the DAD practices for that and it's been really successful. Yeah, yeah.

So let's hope that it carries on living and being open and doesn't get locked away in a safe and prescriptive. Because I'd hate for that to happen.

Well, I think actually, if I look at what PMI has done there, I think that's about PMI adapting to the new world rather than trying to pull something back into the old world, so to speak.

And the benefit of that sale actually was I'm seeing a lot more noise in terms of training available.

Yes.

So one of the problems was unless Scott or Mark was in your area, there was no way of getting access apart from reading it yourself.

No, exactly.

And now it seems to be a lot more investment and scaling out the ability to learn from the people who know. So that was a good benefit. Yeah, I'm happy about that one.

I hate to say this as well, but actually the other thing that helps is something like pmi, which has brand recognition with a lot of leaders and executives having that label on it actually helps a great deal as well.

Even TWI runs agile courses now.

Yeah, exactly.

Things have been around a long time in data space. Mason didn't move for a while. They've changed as well.

No, exactly.

It's good to see.

Yeah, so it is good to see that shift. And this is really about what we as practitioners for years have seen coming. Finally getting it into the limelight. And into the leadership headspace has actually been really powerful.

Okay, good to see you making that change. One thing we might do is get you back because we had Agile Pete on last year and he was doing some work for treasurer, I think around Agile funding models. So he said once he's kind of.

Worked through that, well, that'll be as a public servant, treasury, getting their heads around that will be really helpful for us.

Well, so I'm not sure he has or he hasn't or where they've got to because he's helping them. But if he's got some leverage, might get you back with him and then be interesting to have a conversation around.

Well, that'd be quite good because that is the other challenge. You know, talking in Wellington, talking in the public sector, there is a whole nother dimension which we need to work through, which is how the government interfaces into the public sector and how they approach it and how we get that agile thinking into the, into the government level and what that looks like. I have no idea how to approach that yet.

Treat every election iteration of three years as a PI planning.

Yeah.

That'D be an interesting approach.

Yeah.

With many demos.

With many demos, yeah. That's just it because I mean, you know, that's, I call that a public sector problem. It's a private sector problem as well. Because every organization is, is for the private sector. They have shareholders, they have a board, and then they have their leadership team who are beholden to the objectives that those groups set. We have in the public sector. We have all of our citizens and then we have our elected government, which is our equivalent to the board, and they're the ones setting the strategic direction and setting the goals for the organization and then from there we adapt how we achieve those. So to some extent, and you don't need those groups to be too cognizant of Agile, but you do need them to ideally understand how we're going to try and get there and how we're going to give them what they want. Because the big piece of work for us is continuous legislative change. And those legislative changes tend to come with an end date before we've ever even conceived of how we're going to execute on them. So it's important for us to be in constant communication with the government around what the impacts of those decisions are and how those affect us and what the trade offs need to be.

I was doing some reading the other day around legislation as code and some of the stuff that had been happening in New Zealand with researching and experimenting around that and that got me excited because if government can actually publish their policies, their legislation as a business rule, as a piece of code that could be tested and executed. If you think about everybody else that wants to do something and using automation test whether that's going to meet or break a piece of legislation, that's.

Yeah, it's an interesting concept because I would actually say we've probably already got code that is legislation as code. We just haven't called it that. What we have is we have business rules which are code which were written to meet legislative requirements which effectively execute on that. Yeah, yeah.

Well if you read these, they open sourced white paper on their findings, it came out of the group in DIA that just got shut down, the Special Innovation Project or whatever it was. But one of the ones I referenced was Inland Revenue. And the fact that they've put all their rules, their legislative rules and the Oracle Rules product.

Yes.

So not so much about the product, but the fact that they had the.

Idea of a rules engine with that ability to essentially containerize the rule and make it a reusable artifact across the organization.

So we had one bit of legislation that basically governs our government department and that contains funding conditions which can easily be put into code. If you follow these, you get funded. If not, blah, blah, blah. So if you scale that up.

Yeah.

And make it open source and put an API on it so that I can call it with some inputs and I'll get an answer. I don't have to ring you up anymore. Right. I don't have to take a pun. I can gamify the hell out of it.

You have to tighten up your rules.

But that's okay.

Well that's the next access to our rules engine.

Yeah. But that also is the next step for us. We need better interoperability between agencies as well. There's a lot of work we do where our agency is dependent on what's happening in another agency for a key decision that needs to be made for a client or citizen. And at the moment those processes are not always well streamlined or well orchestrated and they often add long time delays into processes. And it's almost the idea of an open API across government to be able to ask questions and things like that. And the ability to do that would empower a lot more proactive service to the country and a lot more ability to execute successfully.

I think if you look at that and then you say what are the side effects of that? So what we end up doing is actually doing value stream mapping across all the government. So we actually understand where some of the wastages and that can be focused on if it needs to be innovative companies can. Chatbots are cool, Right. So you could put some form of chatbot technology on there where I can ask a question and get an answer relatively easily because it's now a known back end, right?

That's right. Yeah.

There's just so much that could happen if we got to that stage.

Yeah. There is a big caveat on that, which is we need to continue to consider the privacy impacts that that would have and the potential negative impacts that could have. We don't, you know, there is a consideration that we don't want to be linking all of the government agency data across all agencies, on all citizens. That is generally seen as a toxic outcome and we need to be careful about how we do those sorts of things. So as much as I think that there is a huge benefit in doing that, that's the other dimension we haven't really touched on today is the privacy, the ethics and the human rights considerations.

The sovereignty as well, and the introduction of bias based on previous behavior.

That's right.

Especially with the AI. Again, finger marks, AI models.

Right.

They're all trained on data that's biological biased, therefore you're going to get a biased model because that's what it's learned.

That's right. And that is something we're actually, we're conducting joint research with some academic institutions on at the moment, is looking at how we look at disparate impact in models and how we also look at the privacy, human rights and ethics impacts of the work that we're doing and the considerations. And sovereignty is an interesting one because sovereignty affects us from a cloud adoption standpoint, but it also has other implications, especially when you look at, at Maori data sovereignty and you look at the principles of data as a form of identity and what that means to different individuals and to different cultures.

Look, I'm still relatively unhappy that a government organization is taking my cell phone usage data off all the telcos and selling it. Right. Because for me, from a sovereignty point of view, I don't feel comfortable that that is happening to my data without my permission.

That's right. And that's, you know, so everybody's got.

A different view, right?

It does.

Ethics and bias, right?

It does. And you know, and there's no easy answers. But. There is a but. You have to go through a consultation and a discussion process and you need to be open and transparent about what you're doing and you need to be very careful and Then you know, we're passionate about that in our urgency. But you know, if we, if we see that this will have a negative impact to people's lives, to people's well being, or to their privacy or to their human rights, we won't go ahead, we'll pull back.

And also just to close it out, I suppose. Talking about seven year cycle, I'm seeing lots more of the data governance conversations.

Yes, that's coming back around again and.

Sometimes again using old patterns. I hear conversations of data stewards, data custodians, data governance committees and that, but hearing a lot more of catalogs and automated profiling and data fit for purpose based on Persona. And so some of the new ways of working, which hopefully this time will help us be a little bit better around governing the data.

Yeah, no, I know what you mean. And we're actually doing all of that. So we're deploying an enterprise data catalog, we're implementing better lineage and profiling capabilities, but we're also setting up steward networks. And our governance committees around data governance are a little bit stalled at the moment because my experience with governance in this space is it's the same challenge we've always had, which is if you want effective data governance, have a problem to solve and then apply the governance to that problem rather than doing governance for the sake of governance.

Yeah, yeah. I remember one of the largest data governance funded projects in New Zealand. The government for a while years ago was acc. And taking your point, I remember there was a data breach in terms of not somebody hacked him, but information was publicly sent out when it shouldn't be. And the chief executive said that's a problem that needs to be solved now. And therefore the data governance framework and program of work was formed to solve that problem.

Yes.

Not sure they ever did, but there was a problem to solve. Right.

Well, they probably made some progress against that problem and that's kind of how it works. And to be honest, that's not just true for data governance, that's true for all governments. If you've got governance and you don't have a clear reason for it, then you shouldn't have it. Yeah, yep. Yeah.

Which, you know, means a lot of steering committees should be a lot shorter than they are.

Yes, they should be.

We'll get on to that next time. All right, look, I think we've done a, a good session there. We've definitely bounced around lots of really interesting subjects. So thanks for coming in.

No problem.

And we'll probably get you back in 12 months time and do a bit more of a state of the market.

Sounds fantastic.

Excellent.

All right, all right. Thank you.

Thank.

You. Foreign.

You've been listening to another podcast from Blair and Shane where we discuss all things agile bi. For more podcasts and resources, please go to agilebi Guru.