AgileData #26 - AgileBI - Raphael Branger

Welcome to the Agile Data Podcast, where we talk about the merging of agile and data ways of working in a simply magical way. Welcome to the Agile Data Podcast. I'm Shane Gibson.

Hi, and I'm Raphael Branger.

Welcome, welcome. It's been, uh, I've been following you on LinkedIn for a long, long time. It's typically hard for me to find people that are working in the world of agile and data BI, so So it's good to finally have you on the podcast. The reason I asked you to come along is you're writing a book called Agile BI. So I thought it would be great to have a chat about, you know, what's going to be in the book, how you see agile and data blending together. But before we rip into that, why don't you give us a little bit of a background about yourself, just so the listeners know how you got into this world of agile and data?

Yeah, so thanks. It's a great pleasure to be here, and thanks for inviting me. So, I'm in BI and data since nearly 20 years. So when I was 19, I started my career at ITLogix, the same company I'm still working for. And back then, I started really with just creating reports with a tool, now a dinosaur, called Crystal Reports. Maybe you know that. And basically from there, it took me everywhere to system engineering and then into bigger data warehousing projects as a developer. And suddenly, let's say, I recognized that requirements engineering is a crucial topic in business intelligence projects, of course. And that, especially coming from a waterfall world in my, let's say, the first few years of practicing business intelligence, I noticed that there are several disadvantages in gathering requirements just upfront or before the projects because typically, you know, as little as at the beginning of a project. This was basically what me brought to agile, thinking about, okay, how can we adapt agile methods from a software engineering world into the business intelligence world? Then we started to practice that first internally at the company ITLogix where I work. After a few internal learning cycles, we went then out to first customer projects. And so, basically, now it's about 8 to 10 years as we, let's say, working in this agile space. And this was basically the reason for us to decide that we want to write a book and consolidate all our findings until today. But it's just, let's say, a point, an immediate point, what we know today, and we are still on a journey going on.

Yeah, well, that's the agile mindset, right? Kind of inspect and then experiment and then adapt. So, yeah, if we ever get to a stage where we think we're done, we're done done.

Absolutely.

I got a funny feeling we're not kind of following the playbook. So, that's really interesting because it's almost the same as my journey, slightly different in some ways. I started off with forests and trees 30 years ago rather than Christmas. But that whole idea of the time we take to get these requirements up front and the old way of working. And, you know, they were guesses at best. And it was really, we spent too much time up front. And then we had bugger all time to be able to change it at the end, let alone test it. And like you, I mean, the problem I was trying to solve was I was frustrated in the kind of the data modeling, the ETL back then, phase of the development. That's the one I found was if you had a person who was an expert, it was just beautiful. It just worked, right? They knew what they were doing. They'd done it for years. We were always, well, pretty much always successful. But if you had anybody who hadn't done it before or they're relatively new, it was a nightmare. But I struggled to find any patterns in that space at the time. So ended up focusing on requirements as well as being the first thing to work in Agile. And same thing, worked internally in my company, got some ideas and practice, and then lucky enough to be able to experiment with a customer and just took on from there. So, yeah, it's kind of a similar journey. So talking about requirements and requirements engineering, you were scaring me there. I thought we're going to go— there's the whole hyper-specialization happening in the market at the moment with, you know, data engineer, analytics engineer, MLOps engineer. Let's hope we don't get a requirements engineer. But if we think about requirements as engineering, How does that work when you're working with a customer?

So, the term engineering was chosen very specifically because we wanted to express it's not just something, let's say, it's something which you can really learn. It's something which follows certain rules. And this is what brought us to frame a kind of, or define a framework where we said, okay, we don't want to think about what are typical requirements again and again. And learning it by heart. So, we started with having, let's say, 4 major areas, for example. So, we start with asking people about requirements coming, let's say, from the, we call it the environment. So, what is the overall purpose of a project, for example? What are the business goals and processes to be supported? So, really, what you typically would write into a project charter, for example. As well, some limitations you might have, like legal aspects where we— I wouldn't say it's a requirement because it's nothing which someone inside of the project can choose to follow. It's just something which is imposed onto you. So that's the first thing. Then the second thing we tackle is the question about, okay, what are requirements towards the organization and processes of your upcoming BI or analytics system? So this, is the idea to already think about, okay, who will operate the system, for example, or to what degree does a customer want to be involved in the development process, for example. Because this is actual, or actually as well a requirement to say, okay, we want to be involved in the process of developing the system. And as well having, let's say, the requirement to learn together with an external service provider as we are at ITLogix. These are the two first categories. Then we have the two others, which is about data requirements, and the other one is BI application requirements. Basically, what you see or what you get as reports, etc. We do actually vary depending, of course, on the overall scope of a project to then define or decide whether we need to tackle data requirements in what level of detail, for example, because it's a difference, of course, whether you are just building a few new reports on top of an existing, for example, data warehouse, or whether you are building the data warehouse and then maybe you do not even yet know what reports or what concrete reports you want to build after all. So that's basically the 4 main categories we have. In terms of requirements. But we don't— let's say we don't gather them upfront, all, let's say, at the beginning of the project. It's just a structure which we can reuse all the time. So of course, at the beginning of a project, probably the first two categories with the environment, processes, and organization are more center stage. And the more we are getting into the development, the more detailed questions coming up in terms of what are the data requirements and the BA application requirements. And of course, there are many subcategories as well to guide us in the project to ask the right questions and also give some inspiration to an organization of what actually could we want.

So do you find that you tend to start with a customer when they're at the beginning of their journey, you know, a greenfield type environment where you're working to do the data and the visualization? Or do you find at the moment you tend to be coming in when the data is in place to a degree and then you're focusing more on the last mile, the visualization, the dashboards, the BI stuff?

No, currently we are facing a lot of, we will call it, data warehouse modernization situations because we are now in the, let's say, in the age where, let's say, the typical first iteration or cycle systems, which now maybe are between 8 to 10 years old, are coming to an end of life, and either because of hardware and software deadlines where they would have to renew hardware and software licenses. And that basically leads to cloud adoption because a lot of our customers are now moving the BI stack into the cloud. And on the other hand, we see especially in the small and medium size businesses actually just entering a more enterprise-like business intelligence way of working, I would say. They started out maybe with some individual reporting and Excel spread marketing and so on. And now seeing that their business is growing and that they come, let's say, to an upper limit of what is feasible for them in a manual or more ad hoc way of working, Therefore, even though there are new buzzwords all the time, like data mesh, I just consider it the last few weeks where we had a few discussions. But basically, at least for, let's say, the average organization, it's still the classical data warehouse, bringing together data from various sources and have pretty, let's say, standardized reports on top of it to have a good start.

Yeah, I think, yeah, one of the things we do in the IT domain, which we should be embarrassed about, is, you know, new lamps for old. We take a technique or a pattern that's been around for years, and then we bring out a slightly new variation of technology and vendor wash it to make it the new cool. Or even worse, we find a set of technology that has a really good fit in a specific use case and then try and broadly apply it. And the one that that gets my goat the most on that is real-time, you know, real-time streaming of data into dashboards. It's required for some specific use cases, but to try and apply that pattern across every information product or data product is ridiculous.

And just one thought about the technology. I mean, even though it's constantly evolving and it helps to certainly solve some problems more elegantly or more quickly, in the end, when we have struggles in our projects, typically not about the technology. It's, it's really about understanding your business domain, the business processes. Let's say how these processes are depicted in your source systems and having there a lot of gaps between what people think how the data looks like in the systems or how the processes are implemented in their systems to what it is in reality. So this is where the agile thought comes into play once more, because you start with a, let's say, an assumption at the beginning, both from a content or a requirement perspective as well from a solution architecture perspective. And you just need to get access to the data as soon as possible to validate this hypothesis. And from there it starts iterating because once you see the data the first time, you recognize all these differences in your assumptions or compared to your assumption, and then you can narrow it down step by step to the concrete result or what you actually then will deliver.

Yeah. I think you used the word way of working or the term way of working earlier, which, you know, I love because, you know, at the beginning for me, I was focused on methodologies, right, on strict frameworks and structures, on a set of tasks that were repeatable, and we could follow because, you know, I came from a waterfall background. That's how we were meant to have done it. It's how all the big consulting firms pretended they do it. But after a while, I worked out that, you know, for me, there's a set of patterns that you apply, and those patterns may be technical patterns around technology, but they're also patterns around ways of working. How do the teams work? And that's dependent on how many people in the team, how many teams you've got. Also, that idea of a hypothesis, I love. The number of times we've been told how the data behaves or how the business process is executed. And in the old days, we'd believe it and we'd start designing everything around what we were told, then find out they're just urban legends. Nobody updates that field. Oh, no, that field doesn't actually just hold an identity for a person, it happens to hold organization as well because, you know, there was a change 2 years ago and they had no money, so they cheated. So that idea that, you know, you start off with everything's a hypothesis and you prove it as quickly as you can and then move on, I love that, that's something. And the third one that I kind of picked up there was this idea around domain or domain knowledge. And like you, I'm a little bit sick of, you know, new terms and vendor washing, and I agree data mesh will be the new big data. But under the covers, data mesh has some good principles.

Absolutely.

So the idea of being domain-focused, you know, having some domain expertise in the team, makes the team better, right? It's easier, quicker, faster for them, more fun for them to do the delivery.

What we see there is an interesting, I call it paradox, which we see on a large scale, like with a data mesh, but as well on a microscale in just a regular, let's say, small-scale or smaller-scale element, namely that the more flexibility or agility you want to achieve, the more you have to have a kind of standardization and governance on it. And I think that's very interesting to understand that sometimes you need to first limit the options you have in order to achieve higher flexibility in the area you actually want to have it. And I think that's Basically, what I took away from the whole data mesh discussion, you still need some kind of governance to coordinate the different, let's say, elements in the data mesh in order to make it somehow compatible to each other.

In my experience, you know, when I get invited in to help a team, you know, kind of start adopting an agile way of working with data, sometimes there's an expectation that the team are going to be more efficient, faster, deliver more on day one. And we had the conversation really early about actually we're doing complete change and change has implications. And the implications of when you adopt this new way of working is you're slower at the beginning, right? You fail more often and you learn from it. Do you find that? Do you find that people expect agile just to make the data process better on day one? Or do you find most people are open to the idea that actually they've got some learning to do, it'll take a while before the team get to a level of maturity that they're rocking it.

What helps me here is to distinguish between, or I use the analogy to, for example, a car factory. So, in the end, you want to get out thousands of cars per week, for example, and even with a high variability of things which you can choose from, et cetera. But until you get to that stage that you can, let's say, build so many cars each week, you need to invest quite a bit into your factory and build up the factory. And there is some initial effort. And I think that's something what I learned, or which is one lesson learned, let's say, from the past 10 years. There is no, let's say, red button to increase agility. And then you push it, and then agility is here. So basically, it's a quite challenging process, which starts, for example, we always say we start with professionalization. That means you need to have just people working in a professional way. So they need to know, for example, data modeling or certain skills, not only technical, but about requirement engineering. So it's about bringing together a good team, which has a certain level in what they do. And that's the start. And of course, you can have an iterative process like Scrum with Sprints, etc., if you don't have the good people in it, in the team, then you're just failing faster but never succeeding. And the other thing, what you see there, if you start working with professional people, then something follows which I call professional laziness, because you don't want to reinvent the wheel all the time. So, you are looking for standardization, which is the next step on your way to agility. If you start to standardize things from design standards, like from in your data model to technical patterns, but as well for the requirements engineering process, this is why we use a framework, which is always the same. It's always the same categories in order to go through. We have checklists and templates, et cetera. So you are just, let's say, faster with standardizing things or when working with standardized elements in your project. And then a third step comes into play. When you have standardized elements in your project, you can start to automate them. And only automation brings you there that you can deliver, or let's say that you can work in shorter iterations. And with short, I really mean 1 week, 2 weeks at the maximum, where you can build what we call vertical increments or end-to-end increments. Only if you have these short iterations, you actually have a chance to deliver more frequently, like every 4 weeks, every 6 weeks, and so on. And this is this journey where we say, okay, building up this professional way of working, establishing certain standards, implementing automation patterns, this is building the factory. And for, let's say, smaller companies or smaller projects and smaller scope, simpler data sources, etc., it's not that a big thing to build such a factory because again, you can do it in the same way all the time because the tools are there nowadays, etc. And a lot of standards are there anyway as well. But of course, if you have a very large organization, building the factory can be more challenging or more, let's say, time-consuming. Think about if you want to have fully automated deployment pipelines, continuous integration, and so on. So you can invest quite a bit of things if you want into your factory. But The more you invest into the factory, the more powerful are the outputs you can create in even much faster time.

Yeah, so, you know, again, we've taken that concept of automation in the data world and called it DataOps, right? So let's take DevOps and give it a new word. But that idea of factorization is one that intrigues me. So if I look at the way a data team typically works, for me, a flow-based model, a flow-based pattern from Agile tends to fit better. But what I find when I start working with a new team is if we start off with some flow patterns, they look so close to the pipelining and handoff process that they normally use. They'll end up with a BA that does requirements, who hands it over to a modeler who models. Exactly. I find they don't iterate as much, right? That there's not as much change, and therefore, like you said, they become professionally lazy, and not in a bad way, but it's comfortable for them. So what I find is if we move to a Scrum pattern, right, if we break things up into iterations, and that change in the way they work is actually enough to trigger a massive amount of iteration in terms of their focus. Focus on that way of working. And then they tend to then find ways of automation, and when they automate, they become back into that flow of factory process.

Is that what you see? To a certain degree, yes. It's true that we typically start with an iteration-based process as well because it gives you more safety in the beginning. It gives you certain— because think about In a waterfall world, having a plan and thoroughly thought-through concept in the beginning gives you safety as well. And you can be sure that you actually thought about everything and you can simply start developing. Of course, the issue is that the requirements might change while you are developing and that you misunderstood certain requirements. But except of this little detail, you are in a quite safe world because you have really a good concept. Now, in an agile world where you you do not have that much upfront design, it's much more scary, I would say, on an emotional level, but as well on an objective thing that you always need to think, okay, can we change this now without still being sustainable or producing a sustainable solution? Because that's, of course, a risk you are facing, that if you continuously evolve your data model, for example, that you end up in a mess and not a consistent data model, etc. And here, I think having the iterations, especially with certain iteration rituals, like an iteration planning, and then a dedicated iteration review. So this gives not only the developers, but also stakeholders, which probably are new to Agile, a certain safety or feeling of safety, so that they see what's going on, they have a fixed schedule, etc. So it's much easier for them. And for us, then typically in the first week, we call it a release project, consists, or is typically a release, which starts with some preparation work. So again, Agile doesn't mean that you just jump into development straight ahead. So you do some minimal upfront design, and then you go into the development phase, or we call it construction phase. And then we have a handover phase to bring something into operation, which we usually call the transition phase. And So this first release, typically between 2 to 3 months, there is nothing yet in production, especially if it's a greenfield approach for a new system. And their iteration-based working is absolutely fine. Once we are getting close to production of the first release, we are moving slightly away from the, not mainly the, let's say, the iteration rhythm, but we are more flexible in terms of the planning. I don't expect team then to, let's say, commit to what they want to achieve in the next 2 weeks in a very dogmatic way, because we cannot predict how many bug reports we will get or some other just operational things. Because typically, as you mentioned it before, the DevOps or DataOps thing, the developers, as well do the operation of the system, at least in the beginning, typically. And especially in this situation where you must be more flexible to react, the flow-based process, it's just more convenient. But it as well needs certain discipline and a maturity level, for example, of a product owner to constantly really writing their stories or keeping the backlog in shape. And if you don't have these kinds of rituals of strict, okay, now I have the deadline of the iteration workshop after tomorrow, then the flow-based thing can be dangerous, for new teams especially. Yeah.

So for me, it's about level maturity, right? As the team gets more confident, they've started to build in some of that automation that makes them more efficient and stuff. Safer, then change the way you work to be more flow-based if it fits you. And same thing is originally, we effectively had a development team and a BAU team or squads or whatever you want to call them because that's what tended to happen in the waterfall organizations. And so, you know, that whole, oh, there's a whole lot of unplanned features that came through, you know, how do the BAU team pick up and and fix them or go back to the development team. But the development team are in, you know, next iteration and they're now focusing on a new domain. So this DataOps idea or DevOps idea of, you know, you build it, you release it, you maintain it, I think it's a much better model, right? But it brings a whole lot more uncertainty in the planning because, you know, there's always smaller bits of work they're going to turn up out of a typical iteration cycle. The stakeholders don't want to wait, you know, 2 weeks, 3 weeks, 4 weeks for that small change. So on that, you know, I've experimented with customers on size of iteration, you know, time period. I've found that 3 weeks tends to be the best way of starting, right? So a 3-week iteration. And on day 1, it's really unlikely that the team will be able to go from ideation of, hey, We need a new information product to production in 3 weeks, right? They end up pipelining, so they break it down as we should, and it'll take multiple iterations before it's done or done, done. But they're striving to get that cycle time down into 3 weeks where possible. But everybody in the world always uses a 2-week iteration, right? Agile is kind of, with Scrum, Agile has kind of said 2-week iteration is now the de facto standard. I mean, I've tried 4. What I found is that last week was waste for some reason, right? The last week, we, you know, the team tended to waste that week for reasons. One was just a little bit too fast. We did one day for an experiment, one day just for a laugh, and that was funny. But, you know, getting data and transforming it and visualizing it in a day with a team is pretty hard. But what do you find? Do you find that there's a typical iteration length you start off with?

My, let's say, default is actually 2 weeks as well, but I have to distinguish here whether the factory is in place already. If the factory is in place, I tend to work in one-week iterations. So that's challenging. But if you have really automated the main processes from, for example, getting data into a persistent stage and then from there, for example, into a Core Data warehouse model, And if you really can focus on the actual requirements into the modeling, because that's what you can't automate, but then it's absolutely feasible. The main challenge we face is the availability of the product owner. So typically we go to 2 weeks because the product owner in a typical organization of our customer base does not have let's say, a full position or the full FTE for the project. They are typically available for 2 days a week. And that's why we then try to find out a good combination of what is their availability and then look what is the amount of development capacity which makes sense. And then it's as well experimenting. Once again, sometimes we start with a 1-week iteration and then we find, okay, it's too sporty, we just better do it in 2 weeks. In some other cases, we as well had the— or took a decision as a team to go temporarily to a 3-week cycle. But it really depends. And I really encourage you and as well our listeners to really experiment and do not just, let's say, follow some strict guideline in some way but really experiment and then and take a decision, okay, what did work best? And what works best today is perhaps not what works best in 3 months. So it really depends as well on the situation.

Yeah. And also, when you're working in an iterative model or an iterative pattern, they talk about it being, in Scrum terms, we use the term sprint. But, you know, for me, it's a marathon because it's continuous, it's constant, right? There is no downtime on that team. So for me, mixing it up with, say, a 1-day or 1-week iteration, which is really just a research one or, you know, a FedEx-style one where the team are focusing on something that's annoying them about the platform or the automation and they're just focusing on fixing it, changing that cadence from their standard iteration cycle to something different, again, gives them a change, right? And that change is treated as a brownie. Break. So when we go into greenfields, right, we've got to build the platform because as much as people talk about SaaS, you know, cloud data warehouses, the reality at the moment is we have some great cloud analytical databases, right, but they're not a data warehouse. They don't have everything we need. And so especially with the current modern data platform paradigm, we've got to go and grab a bunch of technologies and cobble them together to give us the capability we want. And that takes time and effort. And expertise. So, again, in the original days, I used to, you know, we used to look at iteration zeros, right? We used to look at a bunch of iterations that were just platform builds. And what we found was, you know, we weren't particularly good at guessing. So we knew what some of the core features were, right? You typically need some data storage, you need some way of running code, you need to transform the data, you need to visualize it. That was a no-brainer. But, you know, the details, the tricky stuff in the middle was where the real gold was. And so moved more to, you know, building the plane as you're flying, which is hard, right? But what it means for me is you're only building something when you need it. And so, and then you're testing it straight away, right? To prove that it's given you the value you want. Downside, you know, it's hard to build something when you actually need it on the day. And it makes it longer, right? It makes it perceived as longer to get, you know, information product out the door because there's all that platform engineering that needs to happen. What about you? Are you still seeing a lot of iteration zeros, or are you seeing organizations now kind of building it as they need it?

Again, it's both. I would say I would go with the 80/20 rule. So we invest, let's say, 20% of the time of our inception or iteration 0 to, let's say, trying to build 80% of the needed platform functionality. This we can do typically with a combination of data warehouse automation toolsets. So for example, the originally New Zealand-based tool from, from Webscape, and they already bring you quite a bit of, let's say, basic or foundational functionality, which you just need everywhere or again and again, and as well provides you with the basic pattern for various platforms, etc. So that's, that's the first thing. Now, what we did as a, as a company or service provider, we built on top certain patterns, which helps us to just, let's say, start very quickly with a customer. For example, we have a pattern for creating such a persistent staging area, because it's always the same thing. It's getting the data from resource system, let's say model-wise in a one-to-one fashion, maybe just a few data type mappings and adding historization just to having an archive or kind of an archive. We want to have physical delete detection. And this is always the same for whatever table you take. Now, given the fact, or let's say if we stay in the realm of table-based data, and So if we already have this, let's say after half of a day of configuration, then of course you can already start in the second half of this first day loading the first data. And of course there are some troubles connecting to the data source, but this is where you actually solve a concrete issue or problem. But just loading the data into this persistent stage, for example, that's it. And then we bring along a few patterns just to generate the whole data load for, for example, some dimensional or fact tables where we just need to build in the transformation rules to map the data from the, let's say, the source or the PSA system and map it to the fields you want to have in the core, in the core system. So you can already start modeling from day one because that's part of your requirements analysis. And this is where we work, for example, with the, with the BEAM methodology of Laurence Korn. To really collaboratively working together with business people and creating the data model and having now already the machinery in place that we just can generate the, let's say, the structural elements in the data warehouse in day 1. The only piece which is missing is then really again the mapping between source and target. This can be usually done in a few days. So having this automation already in place, both from a, let's say, platform perspective like the data warehouse tool provider, together with some predefined content, then you can actually achieve this kind of speed. And of course, if you don't have this, then maybe you will invest 1 or 2 weeks to build this kind of basic patterns on demand or based on what your current need is.

Yeah, and so, as I said in the beginning, finding people that apply agile and data together is difficult for me. I mean, there's lots of people that do agile. And one of the things I look for is this idea of an agile data coach. And for me, it's somebody who brings patterns that may be reusable with them. So, you just named a bunch of patterns that are core to a data team. When they're working. And 100% I'm with you with Laurence Kore and the BEAM approach. That for me is a pattern that most data teams should adopt for understanding the core business processes and the data that supports it. It's a pattern that's worked for me for 10 years and I love it. It's fun as well, right?

Absolutely.

Yeah, anybody who hasn't used it, go use it. So at risk of starting a religious war, the other pattern that that everybody tends to argue about is the modeling pattern. Which modeling paradigm do you tend to see customers using the most and which pattern do you tend to bring with you the most?

Now, I'm a bit biased because we are proposing dimensional modeling to our customers and therefore we do not have that many Data Vault customers because we are not that famous for having this or having the expertise in this modeling. One reason for that is we are still thinking or we are still convinced that tackling the BI challenge from a business perspective, which is tangible for people and which is basically understandable, is the most biggest benefit you can achieve so that you have an understandable data model that which is understandable by business people. And in most cases we are working for, and again, this depends again on the size of the company and the number of source systems, but the typical company we are serving where we are talking about maybe between 5 to 10 source systems typically in one, let's say, platform, then a dimensional model is definitely a good fit where you don't need the additional layer of a data for example. And we as well try to mitigate certain elements or certain risks which come along with a dimension model by always providing a persistent staging area as an archive where you can always reload the core if it's absolutely necessary. Of course, we try to omit that, but we have this kind of safety net. And now the question was, again, why was this it's possible to do so because earlier on, it was just too much effort to build, let's say, these dataflows into a persistent stage all the time for all the tables because you would have to create this manually for every table. Now, as we have all the patterns in place, it doesn't matter whether we have 5 tables, 50 tables, or 500 tables. You just drag them in into the tool and say, okay, now generate. And that's it. And then you go and grab a coffee and wait until the generation process and the deployment process— everything can be automated as long as there is no business logic there. And again, there, this agility as well brings us this flexibility that we can really work with a model which makes sense from a business perspective, even if it has some downsides from a technical perspective. But But let's say the overall package definitely works fine.

Yeah, I mean, as you probably know, I'm more of a Data Vault bigot than a dimensional one. But I take your point on complexity, you know, when you use Data Vault, you'd never expose the Data Vault models to the end consumers. You always need a consume layer or presentation or whatever you want to call it. And that may be dimensional, maybe denormalized, because under the covers, the model is is horrendous to look at, but incredibly easy to automate. And I'm with you on PSA, on what we call a history layer. You know, that's a bit of a heretic in the Data Vault world because in theory, you're meant to use the Vault modeling technique for a raw layer. But we don't, we recommend using persistent staging, which is mirroring the source system, just historizing the tables. For the same reason, you can reload from scratch, which you could do with a raw Vault. But I don't know, it's a pattern that's worked for me a lot. And so therefore, it's a pattern that, you know, we tend not to change, just like your experience in Dimensional works, and therefore, you get the value out of it. And so use the patterns that work for you.

I think one thing to add there is it's really depending on the, let's say, the customer organization, whether they have the people with the necessary skills to operate and understand and maintain a data vault. And this is where we see, especially for very large organizations, typically that works, or they have enough as well internal staff to do so. But for a bit smaller companies and where BI is mainly business-driven, you are just getting to some limitations in terms of, okay, understanding highly normalized model-like data of all these.

Whenever there's divergence in patterns, there's always very strong opinions. I remember the old Inman and Kimball arguments of many years ago. So, I'm with you, it's up to the organization and the teams we're coaching for them to pick the patterns. What we do is we encourage them where we've seen value and ways of working from other teams, that's something we bring boring. I have had teams that actually don't model. Now, I don't recommend it, right? My recommendation, based on my experience, is always pick a modeling technique. There is value in that. But I have had teams that, in the way they work, they don't model. They are more of a deploy and destroy, and that's their paradigm, and it works for them. But I don't see it often, and it has to be a conscious choice, right? The team have had to have experimented and know it works rather than just not modeling because 'cause it seems easier. So when we talk about that factory and that automation, and we've got the framework or the pattern in for loading new data sources into the persistence stage. So we point at it, we do the horrible roundabout of how do we connect, don't have credentials, all yada, yada, yada. But we finally get access to the beast of a source system. We start grabbing the data over, we've identified the keys on the table, We're all rocking persistent stage or areas hydrated. If we're moving into a new domain, what I find is, you know, the team come in and they're right, you know, they're kind of back at day one because unless there's a domain expert on the team, they don't understand the core business processes, they don't understand the source data, they don't understand the profile. They've got a whole lot of work, right, before they can start applying their normal patterns. And there's a real temptation to do that work early, right, to kind of pipeline it where another team does all that work and hands it over. And so there's this balance between the problem we have where a team walk into iteration planning on day 1, let's say it's a Monday, and they have no background information and they're trying to plan their work, right? So they're their rate of guessing, they're guessing with a whole lot of uncertainty. So that's kind of one end of the spectrum. And there's the other end of the spectrum where, you know, BA goes away and does a whole lot of analysis and insight, doesn't write up a requirements document, but all that knowledge is in one person, right? And that work's been done upfront, but it's not been done by the team, so it's not as consumable. And so for me, a kind of hybrid model, right, a balance between that work done upfront and the team turning up with no insight is something that that we focus on a lot, and it's a difficult problem. How do you deal with that?

So, you just so-called inception phase with some minimalistic upfront design comes into play because we often have the situation that we really start from scratch or on the greenfield, and not even the customer knows exactly what they want and how things work. And usually, it doesn't take that much time to just just get some first insights and do some first experiments. So typically, maybe 2 to 3 days of hackathon where you just look at, first of all, the goals you want to achieve, then having some source system guys in place where you can just have a direct look into these tables, etc., looking what's there, asking questions, doing some modeling canvas work together. Basically, it doesn't take that much until you get a first idea how things work and how things relate to each other. And I think one important thing here is establishing a common language. So modeling for me is not only having a data model in the end, but modeling is certainly defining this kind of common vocabulary so that everybody understands what a customer is, what is a product, what is a service, whatever entity you have in your organization or finally then in the data model, I think you can solve a lot of things within only 2 to 3 days. And I think it's worth taking this effort to, let's say, get a new team accustomed to each other, but as well to the domain and the topic they should work in. And then establish the plan after having these first few workshops. And then start with maybe, if it's a really, let's say, sophisticated new business domain, why not doing a proof of concept or a little pilot where you don't need even to build the fully-fledged data warehouse? What we often do is we just build the persistent stage with a few tables because, again, automation, let's assume, is already there. Then we just do either a view layer or go directly in a tool like, for example, Power BI, and then do just some some virtual data modeling where we just have a look how could the skeleton look like, and then already show some first data. It's not about having a report which is production-ready, but that you can show a first Cocky Read result even after the first 3 to 4 days. And this is typically as well where you can then identify certain misunderstandings in terms of, oh, I didn't mean it like this from a customer perspective or a business perspective. And so that's, let's say, I think a good step in between that you do this kind of virtual data model where you just have a first idea how this could look like. And once you have figured out the main issues during this experiment, then you can tackle these issues already in your backlog, for example, and say, okay, here we see we need to— for example, we have certain data quality issues. So you already know, okay, here we need to plan a little bit more time during the sprint or an upcoming sprint to address them, things like this.

Yeah, exactly the same technique that I, that Pat and I recommend. So the idea of a research spike, the idea of timeboxing it, and again, really important to timebox it because as data people, we love to go into detail. So, you know, when the team find out they'll need to match customer records from multiple systems That's enough, right? That's enough statement to know there's some complexity, but they always want to deep dive into how's the matching algorithm going to work. I'm a great fan of canvases. I find that canvases are a good way of getting enough information quickly that the team can look at it and nod together. They get understanding, but they're not going to write a whole lot of detail because at this time it has no value or minimal value. And definitely agree that the best way of getting feedback is put something in front of the customer. So, quick prototypes in front, but being very clear that they can never go to production. They don't meet our definition of done, right? So, as professionals, we will not push it to production because we have standards we hold ourselves to, and hacking a dashboard and a prototype and making it live for 1,000 users is not what we do as professionals. So, online there. So, two things I want to cover off just before we end is a quick conversation on scaling, which is never a quick conversation because that's one of the hardest problems. And a second conversation around the state of the BI and data tooling market for agile ways of working. So let's go on the scaling one. So what I find is working with a small team is not easy, but it's easier, right? They're dedicated, they're focused, they're iterating their ways of working, they're self-contained, self-organizing, you know, the lines of communication are small enough, they tend to just rock it, right? And it's amazing to watch. As soon as we scale to, you know, 10, 20, 30, 40 people, we now have to break things up. We know that a team of 40 working on the same thing is not efficient. So, we need to scale them, right? And we may scale them on a domain. Heavens forbid we scale them on the part of the process because, you know, not my favorite pattern. We may do it where we scale them to just pick up the next thing off the backlog, right? But now we have, we want to reuse patterns, we want to set standards across them and have a light federated governance model, those kinds of things. So what's your view? I mean, have you got any magic tips for when you want to scale from a team of 5 to a team of 30?

It's not magic, but the first thing I do is I typically split those who build the factory and those who use the factory. Because as well from a skilling perspective, Those who build the factory typically need to have much more insights in terms of what standards they want to see implemented. They need to maybe think a bit more in a more abstract way because they need always to find a generic kind of solution for which then can be turned back again into concrete solutions. So that's the first thing. And depending on how big your factory should become, you can put in quite a lot of people into the factory, which is then much more similar to working on a software engineering project, even though it's still in the data world, et cetera. And then again, the factory is a highly important aspect, I think, in order to then have multiple teams working on, for example, different domains, but always using the same patterns. And this you can't achieve without having, let's say, the kind of factory, let's say, powered by automation. Because the automation, again, as I told before, is always based on standardization because Otherwise, you can't automate. And I think that's, again, the foundation. Once you have this machinery in place, it's much easier then to have multiple teams on top working in always the same way because they don't have to care about naming conventions. They don't have to care about how certain code is implemented. They can purely focus on the business requirements, the modeling, and certain, let's say, transformation logic to map the source data to the target model. So that's, let's say, my main, let's say, direction I would use in order to scale up the teams. The other angle we could think of is when we talked about these end-to-end increments, it's always a question, what does end-to-end mean? And typically, or certain authors in the literature, they always say, okay, you always have to go from the source system up to the fully-fledged dashboard, etc., and only this is agile BI. And then, of course, there are other authors which say, okay, no, everything needs to be split. So, I mean, nearly not every layer in the data warehouse, but perhaps the persistent stage is one layer, and this is a user story, and then you do a user story for your core layer, etc. And we think that it's somewhere in between. We typically go with, let's say, until the core warehouse where we say the core warehouse is independent of the main business context. Of course, there is some business logic, for example, when it comes to data integration of various source systems. But usually on the level of a data mart and then the final reporting, this is typically driven by a specific use case. This is as well where we could split teams to say, Okay, one team is responsible for really building the data foundation and do the stuff which is rather generic in the context of the organization. And then you have individual either project or use case driven teams, which are then working on specific, really some soft business rules and the implementation of the, let's say, the data model reporting layer. And there you can as well scale based on this separation of resources.

Yeah, so I agree with you. Scaling's hard. Pick a pattern, experiment with it, adapt to where it doesn't work. A lot of the time, the teams I've worked with, we have moved to what I call platform as a product, right? So we have a squad or a team that are focused on the platform. We tend to introduce the idea of a data platform manager whose role it is to engage with the other teams or squads to understand what's coming. And so, they can serve them as their customer. One downside I found with that is is some of the fun stuff, some of the platform engineering, some of that hardcore automation, then gets taken out of the squads and given to another team that tends to do all the cool stuff. And, you know, the squads end up kind of being more factory of just move the data and visit. So, one of the teams I worked with, they experimented with the squads doing the initial cut of a piece of automation as part of their information product delivery. So, they solved their own problem with some help from the platform squad. So we might have parachuted a platform squad member into that team for a while, just help them. And then once it had some value, then it got moved to the platform squad to automate it and make it available as a product to the other teams so they could use that. And as you said, professional laziness, I think Spotify talked about, used different terms, but the same model, which was when something's repeatable and available and automated, a team will typically pick it up and use it because it's easy for them. So you don't need to force them, you just need to encourage them. So, yeah, and then, yeah, you can do it via domain, you can do it via stage of the development cycle. But the key thing is experiment and adapt. So if it's not working for you, change the model. So, yeah, but scaling is hard. So, last one, state of the data, the tooling market to support agile ways of What's your view on it at the moment?

Definitely better than 10 years ago. So when we started 10 years ago, I read about things like data warehouse automation, and definitely there were some vendors back then, but it was still on a rather rudimentary basis, I would say. And nowadays we have quite a vivid market of tool vendors in the area of automation. What I prefer having there, tools, and of course, again, WhereScape, where you have still a certain flexibility, where you can, let's say, influence what code is generated. And even though out-of-the-box stuff is great to start very quickly, you will always end up with certain, let's say, customer or organization-specific things which you need to adapt. And this is something where I would look for that you always— that you don't lose the flexibility to control what's actually generated, to be not that much dependent on a specific vendor. Second thing I would like to mention are data or data warehouse-specific testing tools, which is something which isn't on the market that long, I would say. Or at least there I see as well interesting development so that you can first of all automate test cases or running the test cases. And what's as well a very interesting combination is when you manage to combine the data warehouse automation tool with the test automation tool so that you can, let's say, that the test automation can, for example, use the metadata of a tool like WhereScape to derive what tables to be tested, having the full lineage, for example, to know, okay, if this is my PSA table, the source table in the source system is this and this table, for example. And I think this makes it very powerful in terms of as well making sure that the data quality is there and that you don't have regression issues. And the third thing I would like to mention is that agility or this tooling stuff is not only there in the backend, in the data backend, it's as well there in the frontend because the same is true in terms of standardization. So maybe you have heard of the IBCS, the International Business Communication Standard. Standards. And using such, we call it a notation standard, where you define or where it's somehow predefined how charts should look like, how tables should look like. Then there are as well tool providers, typically add-ins for existing tools like Power BI, so that you can apply this standard automatically or pretty much out of the box. And this again speeds up the whole process from requirements engineering because you don't need to rediscuss, oh, would I like to have now this chart in blue, yellow, orange, or whatever color? It's just predefined to a certain degree. And as well, the development is very quick because let's say when I used to invest multiple days, if not weeks, to create a fancy chart, you get much more out of the box, again, based on the standard. And I think I think that's why I said it's much better than 10 years ago throughout, through the whole stack from really getting the data in, testing the data. We have a lot more of automation aspects available nowadays. And of course, as well, the whole deployment pipelines, DevOps tools. So I think we did a good progress, but we are still behind the software industry or software engineering world. They are still still, let's say, 10 years before us.

Yeah, definitely. And especially when we talk about testing, right? You know, testing, automated testing of data is— we are so woeful. It's getting better, but I think we see waves and patterns in the technology space. And, you know, like the first tool you used, Crystal Reporting, got bought out and consumed into a big beast and then completely lost. I think we're going to see a wave now of all the little niche tools that add value to our lives getting merged into big behemoths until the next wave goes. So just To close out, I started my journey reading a couple of books. I read some of the agile data or BI books from Ralph Hughes and Ken Collier. They really illuminated my journey for me. And then there were a bunch of other books that came out, Lawrence's book around Beam, Hans's book around Data Vault, that were more specialized, right? They were dealing with one small part. Yes, they used the word agile, but they weren't really talking about the end-to-end process and how it fitted. They were just talking about some really good patterns that you could use in there. And then we kind of had a gap, right? There's not a lot came out. And then we got a bunch of what I would call whitewash books, right? We got a bunch of books that had the word agile in it and data, but really, they were technology books that talked step by step how to implement, you know, SQL Server Analysis Services, or, you know, a bunch of tools and technologies. And look, they were valuable if you hadn't done your platform build before, but they were all focused on technology, not ways of working. So you're writing a book, right? I know books take a while to write, but, and obviously, based on what we're talking today, I'm assuming your book is going to be about the end-to-end process, about that way of working, and patterns you can leverage. So, is that true? And when can I start reading it?

You can already start reading it now because we are, of course, writing the book in an agile way. So, we are doing iterations chapter by chapter, and we are publishing, let's say, beta releases of the book. So, maybe we can post the link on the show notes. But you can easily register there and become a beta reader. So, currently, we have the first 6 chapters already available. And then you can as well give feedback if you want, what you would like to see in addition or where something didn't make sense for you, for example. So that's the current process. So we plan to have a first draft ready this spring somewhere, and then hopefully the book can be published somewhere in 2022 already. Now, in terms of how do we want to be different to already existing books is basically, first of all, we addressing non-IT people in that book. So, because our main stakeholders we are currently working with are typically controllers, for example, or marketing analysts, etc. So, really business people. And there we see that they didn't want or they don't want to read too much of a technical book. They are more interested in the overall process, for example. And that's why we are currently— or the book is designed as a city guide. So, we depicted the city called Agile BI City, and we are taking our readers onto a sightseeing tour through this city where we show them around various districts. And we start, for example, with a district called Inception Beach. And Inception Beach, you learn how you do this minimal upfront design. Then we go further to Vision Hills. So, this is more about, okay, what What role does a BI strategy play, for example, to enable agility in BI? Then we go to Westside Requirements, for example, where it's all about gathering requirements. Then we go to, I guess we have Contracting House. So how about agile contracting? Downtown Patterns. Then we go to Technologyland or a district called Technologyland, I think. So we are really walking the reader through the various steps, the various, I I would say, building blocks you typically need. And of course, you do not have to visit or, let's say, then adapt this into your whole, in your own company to have all the different areas. But I think it gives you a good overview, what are the elements you need to think of, and then adapt accordingly to your own situation.

That's cool. I mean, I love the way that it's telling, giving the information by telling a story in a different way. I mean, One of the books I did enjoy that came out around agile but not around data was The Phoenix Project. And I enjoyed it because it told a good story. And then as part of telling that good story, you learn some things. So, yeah, that idea of a city guide, that's cool. That's something I haven't seen before. So I look forward to reading it this year. So before we close out, anything else top of mind for you about agile and BI and data that you want to talk about?

So I think what's for me again and again the most important part, show the value as early as possible. Even though we talked about building a factory, et cetera, don't invest too much into the factory before showing some concrete value, maybe just having a simple report or a quick win where you can show the value to your stakeholders. I think that's the most important point. Once they know that there is value in their data, then you can start focusing on building the necessary foundation to have a really sustainable solution, et cetera. But don't forget about the value and to show as well and make this value visible to your stakeholders.

Yeah, no, I can't agree more. So look, thanks for your time. It's been an awesome talk. We've covered a lot of bullet points, but they were all things that people should think about when they're doing agile BI or agile data with your team. So thank you for your time again, and we'll catch you later.

Yes, thanks for having me. Bye.

And that, Data Magicians, was another Agile Data Podcast. If you'd like to learn more on applying an agile way of working to your data and analytics, Head over to agiledata.io.