AgileData #26 - AgileBI - Raphael Branger

Welcome to the Agile Data Podcast, where we talk about the merging of agile and data ways of working in a simply magical way.

Welcome to the Agile Data Podcast. I'm Shane Gibson.

Hi. And I'm Rafael Branger.

Welcome. Welcome. It's been following you on LinkedIn for a long, long time. It's typically hard for me to find people that are working in the world of agile and data bi. So it's good to finally have you on the podcast. The reason I asked you to come along is you're writing a book called Agile bi. I thought it would be great to have a chat about what's going to be in the book, how you see agile and data blending together. Before we rip into that, why don't you give us a little bit of a background about yourself, just so the listeners know how you got into this world of agile and data.

Yeah, thanks. It's a great pleasure to be here and thanks for inviting me.

I'm in BI and data since nearly 20 years. So when I was 19, I started my career at IT Logics, the same company I'm still working for. And back then I started really with just creating reports with a tool now at Dinosaur called Crystal Reports, maybe you know that. And basically from there it took me everywhere to system engineering and then into bigger data warehousing projects as a developer. And suddenly, let's say, I recognized that requirements engineering is a crucial topic in business intelligence projects, of course, and that especially coming from a waterfall world. In my, let's say, the first few years of practicing business intelligence, I noticed that there are several disadvantages in gathering requirements just upfront or before the projects. Because typically, you know, as.

At the beginning of a project, this was basically what we brought to Agile thinking about, okay, how can we adapt Agile methods from a software engineering world into the business intelligence world?

Then we started to practice that first internally at the company. IT Logics where I work. And after a few internal learning cycles, we went then out to first customer projects. And so basically now it's about eight to ten years as we, let's say, working in this agile space. And this was basically the reason for us to decide that we want to write a book and consolidate all our findings until today. But it's just, let's say, a point, an immediate point, what we know today and we are still on a journey going on.

Yeah, well, that's the agile mindset, right? Kind of inspect and then experiment and then adapt. So yeah, if we ever get to a stage where we think we're done, we're Done.

Done. Absolutely.

I get a funny feeling we're not kind of following the playbook. So that's really interesting because it's almost the same as my journey. Slightly different in some ways. I started off with forest and trees 30 years ago rather than crystal. But that whole idea of the time we take to get these requirements up front and the old way of working and you know, they were guesses at best and, and it was really, we spent too much time up front and then we had bugger all time to be able to change it at the end, let alone test it. And like you, I mean the problem I was trying to solve was I was frustrated in the kind of the data modeling, the ETL back then phase of the development. That's the one I found was if you had a person who was an expert, it was just beautiful, it just worked right. They knew what they were doing, they'd done it for years. We were always, well, we're pretty much always successful but if you had anybody who hadn't done it before or they're relatively new, it was a nightmare. But I struggled to find any patterns in that space at the time, so ended up focusing on requirements as well as being the first thing to work in in Agile and same thing worked internally in company, got some ideas and practice and then lucky enough to be able to experiment with a customer and just took on from there. So yeah, it's kind of a similar journey. So talking about requirements and requirements engineering, you were scaring me there. I thought we're going to go, there's the whole hyper specialization happening in the market at the moment with, you know, data engineer, analytics engineer, mo ops engineer, let's hope we don't get a requirements engineer. But if we think about requirements as engineering. Yeah, how does that work when you're working with a customer?

So the term engineering was chosen very specifically because we wanted to express it's not just something.

Let'S say it's something which you can really learn, it's something which follows certain rules. And this is what brought us to frame or defined a framework where we said, okay, we don't want to think about what are typical requirements again and again and learning it by heart. So we started with having, let's say four major areas. For example, so we start with asking people about requirements coming let's say from the, we call it the environment. So what is the overall purpose of a project? For example, what are the business goals and processes to be supported? So really what you typically would write into a project charter for example.

As well some.

Limitations you might have, like legal aspects where we. I wouldn't say it's. It's a requirement because it's nothing which someone inside of the project can choose to, to follow. It's. It's just something which is imposed onto you before. And so that's the first thing. Then the second thing we tackle is the question about, okay, what are requirements towards the organization and processes of your upcoming BI or analytics system? So this is the idea to already think about, okay, who will operate the system, for example, or to what degree does a customer want to be involved in the development process, for example, because this actual or actually as well a requirement to say, okay, we want to be involved in the process of developing the system and as well having, let's say the requirement to learn together with an external service provider as we are at IT logics. So these are the two first categories and then we have the two others which is about data requirements and the other one is BI application requirements. So basically what you see or what you get as reports, etc. And we, we do actually vary depending of course on the overall scope of a project to then define or decide whether we need to tackle data requirements in what level of detail, for example, because it's a difference, of course, whether you are just building a few new reports on top of an existing, for example data warehouse or whether you are building the data warehouse. And then maybe you do not even yet know what reports or what concrete reports you want to build, after all. So that's basically the four main categories we have in terms of requirements. But we don't, let's say we don't gather them up front all, let's say at the beginning of the project, it's just a structure which we can reuse all the time. So of course at the beginning of a project, probably the first two categories with the environment, processes and organization are more center stage. And the more we are getting into the development, the more detailed questions coming up in terms of what are the data requirements and the BA application requirements. And of course there are many subcategories as well to guide us in the project to ask the right questions and also give some inspiration to organization or what actually could we want.

So do you find that you tend to start with a customer when they're at the beginning of their journey, a greenfields type environment where you're working to do the data and the visualization, or do you find at the moment you tend to be coming in when the data is in place to A degree and then you're focusing more on the last mile, the visualization, the dashboards, the BI stuff.

No, currently we are facing a lot of, we would call it data warehouse modernization kind of situations because we are now in the, let's say in the age where, let's say the typical.

First iteration or cycle systems, which now Maybe are between 8 to 10 years old, are coming to an end of life. And either because of hardware and software deadlines where they would have to renew hardware and software licenses, that basically leads to cloud adoption because a lot of our customers are now moving the BI stack into the cloud. And on the other hand we see especially in the small and medium size businesses actually just entering a more enterprise like business intelligence way of working. I would say they started out maybe with some individual reporting and Excel spread marketing and so on and now seeing that their business is growing and that they come, let's say, to kind of an upper limit of what is feasible for them in a manual or more ad hoc way of working. And therefore, even though there are new buzzwords all the time like data mesh, I just consider it the last few weeks or where we had a few discussions. But basically, at least for let's say the average organization, it's still the classical data warehouse, bringing together data from various sources and have pretty, let's say standardized reports on top of it to have a good start.

Yeah, I think one of the things we do in the IT domain which we should be embarrassed about is, you know, new lamps for old. We take a technique or a patent that's been around for years and then we bring out a slightly new variation of technology and vendor wash it to make it the new call. Or even worse, we find a set of technology that has a really good fit in a specific use case and then try and broadly apply it. And the one that that gets my goat the most on that is real time, real time streaming of data into dashboards. It's required for some specific use cases. But to try and apply that pattern across every information product or data product is ridiculous.

Just one thought about the technology. Even though it's constantly evolving and it helps to certainly solve some problems more elegantly or more quickly in the end, when we have struggles in our projects, typically not about the technology, it's really about understanding your business domain, the business processes, let's say, how these processes are depicted in your source systems and having there a lot of gaps between what people think how the data looks like in the systems or how the processes are implemented in their systems. To what it is in reality. So this is where the Agile thought comes into play once more. Because you start with, let's say, an assumption at the beginning, both from a content or requirement perspective as well from a solution architecture perspective. And you just need to get access to the data as soon as possible to validate this hypothesis. And from there it starts iterating because once you see the data the first time, you recognize all these differences in your assumptions or compared to your assumption, and then you can narrow it down step by step to the concrete result or what you actually then will deliver.

Yeah, I think you used the word way of working or the term way of working earlier, which I love, because at the beginning for me I was focused on methodologies, on strict frameworks and structures, on a set of tasks that were repeatable and we could follow because, you know, I came from a waterfall background. That's how we were meant to have done it. It's how all the big consulting firms pretended they do it. But after a while I worked out that, you know, for me, there's a set of patterns that you apply. And those patterns may be technological, technical patterns around technology, but they're also patterns around ways of working. How do the teams work. And that's dependent on how many people in the team, how many teams you've got. Also that idea of a hypothesis. I love the number of times we've been told how the data behaves or how the business process is executed.

And in the old days we'd believe it and we'd start designing everything around what we were told, then find out they're just urban legends. Nobody updates that field. Oh no, that field doesn't actually just hold a identity for a person. It happens to hold organization as well because there was a change two years ago and they had no money, so they cheated. So that idea that you start off with everything's a hypothesis and you prove it as quickly as you can and then they move on. I love that. That's something. And the third one that I kind of picked up, there was this idea around domain or domain knowledge. And like you, I'm a little bit sick of new terms and vendor washing. And I agree, Data Mesh will be the new big data. But under the covers, Data Mesh has some good principles.

Absolutely.

The idea of being domain focused, having some domain expertise in the team, makes the team better. Right. It's, it's easier, quicker, faster for them, more fun for them to do the delivery.

What we see there is, is an interesting, I call it paradox, which we see in, on large scale, like with a data mesh, and but as well on a micro scale, in just a regular, let's say small scale or smaller scale element, namely that the more flexibility or agility you want to achieve, the more you have to.

Have a kind of standardization and governance on it. And I think that's very interesting to understand that sometimes you need to first limit the options you have in order to achieve higher flexibility in the area you actually want to have it. And I think that's basically what I took away from the whole data mesh discussion. You still need some kind of governance to coordinate the different, let's say, elements in the data mesh in order to make it somehow compatible to each other.

In my experience, when I get invited in to help a team kind of start adopting an agile way of working with data, sometimes there's an expectation that the team are going to be more efficient, faster, deliver more on day one and, and we had the conversation really early about actually we're doing complete change. And change has implications. And the implications of when you adopt this new way of working is you're slower at the beginning, you fail more often, and you learn from it. Do you find that, do you find that people expect agile just to make the data process better on day one, or do you find most people are open to the idea that actually they've got some learning to do, It'll take a while before the team get to a level of maturity that they're rocking it.

What helps me here is to distinguish between, or I use the analogy to, for example, a car factory. So in the end you want to get out thousands of cars per week, for example, and even with a high variability of things which you can choose from, etc. But until you get to that stage that you can, let's say, build so so many cars each week, you need to invest quite a bit into your factory and build up the factory and there is some initial effort. And I think that's something what I learned or which.

Is one lesson learned, let's say from the past, past 10 years. There is no, let's say red button to increase agility and then you push it and then agility is here. So basically it's a quite challenging process which starts, for example, we always say we start with professionalization. That means you need to have just people working in a professional way. So they need to know, for example, data modeling or certain skills, not only technical, but about requirement engineering. So it's about bringing together a good team which has a certain level in what they do, and.

That'S the start. And of course you can have an iterative process like scrum with sprints, etc. If you don't have the good people in it, in the team, then you are just failing faster, but never succeeding. And the other thing, what you see there, if you start working with professional people, then something follows which I call professional laziness, because you don't want to reinvent the wheel all the time. So you are looking for standardization, which is the next step on your way to agility. And if you start to standardize things from design standards, like from in your data model to technical patterns, but as well for the requirements engineering process. This is why we use a framework which is always the same, it's always the same categories in order to go through, we have checklists and templates, etc. So you're just, let's say faster with standardizing things or when working with standardized elements in your project. And then a third step comes into play. When you have standardized elements in your project, you can start to automate them and only automation brings you there that you can deliver. Or let's say that you can work in shorter iterations and with short, I really mean one week, two week at the maximum, where you can build what we call vertical increments or end to end increments. Only if you have these short iterations, you actually have a chance to deliver more frequently, like every four weeks, every six weeks, and so on. And this is this journey where we say, okay, building up this professional way of working, establishing certain standards, implementing automation patterns, this is building the factory. And for, let's say smaller companies or smaller projects and smaller scope, simpler data sources, etc. It's not that a big thing to build such a factory, because again, you can do it in the same way all the time, because the tools are there nowadays, et cetera, and a lot of standards are there anyway as well. But of course, if you have a very large organization, building the factory can be more challenging or more, let's say, time consuming. Think about if you want to have fully automated deployment pipelines, continuous integration and so on. So you can invest quite a bit of things, if you want, into your factory. But the more you invest into the factory, the more powerful are the outputs you can create in even much faster time.

Yeah, so again, we've taken that concept of automation in the data world and called it data ops, right? So let's take DevOps and give it a new word. But that idea of factorization is one that intrigues Me. So if I look at the way a data team typically works, for me, a flow based model, a flow based pattern from Agile tends to fit better. But what I find when I start working with a new team is if we start off with some flow patterns, they look so close to the pipelining and handoff process that they normally use, they end up with a BA that does requirements. Who hands it over to a model model.

Exactly.

I find they don't iterate as much, right, that there's not as much change and therefore, like you said, they become professionally lazy and not in a bad way, but it's comfortable for them. So what I find is if we move to a scrum pattern, right, if we break things up into iterations and that change in the way they work is actually enough to trigger a massive amount of iteration in terms of their focus on that way of working and then they tend to then find ways of automation and when they automate, they become back into that flow of factory process. Is that what you see?

To a certain degree, yes. It's true that we typically start with an iteration based process as well because it gives you more safety in the beginning. It gives you certain. Because think about in a waterfall world, having a plan and a thoroughly thought through concept in the beginning gives you safety as well. And you can be sure that you actually thought about everything and you can simply start developing. Of course the issue is that the requirements might change while you are developing and that you misunderstood certain requirements. But except of this little detail.

You are in a quite safe world because you have really a good concept now in an agile world where you do not have that much upfront design, it's much more scary I would say on an emotional level, but as well on an objective thing that you always need to think, okay, can we change this now without or still being sustainable or producing a sustainable solution. Because that's of course.

A risk you are facing that if you continuously evolve your data model, for example, that you end up in a mess, not a consistent data model, etc. And here I think having the iterations, especially with certain iteration rituals like an iteration planning and then a dedicated iteration review. So this gives not only the developers, but also stakeholders which probably are new to agile, a certain safety or feeling of safety so that they see what's going on, they have a fixed schedule, etc. So it's much easier for them to. And for us then typically in the first week, we call it a release project consists or is typically a release which starts with some preparation work so again, agile doesn't mean that you just jump into development straight ahead. So you do some minimal upfront design and then you go into the development phase, or we call it construction phase and then we have a handover phase to bring something into operation which we usually call the transition phase. And so this first release, typically between two to three months, there is nothing yet in production, especially if it's a greenfield approach for a new system and their iteration based working is absolutely fine. Once we are getting close to production of the first release, we are moving slightly away from not mainly, let's say the iteration rhythm, but we are more flexible in terms of the planning. I don't expect team then to, let's say commit to what they want to achieve in the next two weeks in a very dogmatic way because we cannot predict how many bug reports we will get or some other just operational things because typically, as you mentioned it before, the DevOps or data op thing, the developers as well do the operation of the system at least in the beginning, typically.

Especially in this situation where you must be more flexible to react the flow based process, it's just more convenient. But it as well needs certain discipline and a maturity level, for example of a product owner to constantly really writing their stories or keeping the backlog in shape. And if you don't have these kind of rituals of strict, okay, now I have the deadline of the iteration workshop after tomorrow, then.

The flow wave thing can be dangerous for new teams especially.

Yeah, so for me it's about level maturity. Right. As the team get more confident, they've started to build in some of that automation that makes them more efficient and, and stuff safer. Then change the way you work to be more flow based if it fits you. And same thing is originally.

We effectively had a development team and a BAU team or squads or whatever you want to call them because that's what tended to happen in the waterfall organizations. And so that whole there's a whole lot of unplanned features that came through. How did the BAU team pick up and, and fix them or go back to the development team. But the development team are in next iteration and they are now focusing on a new domain. So this data Ops idea or DevOps idea of you build it, you release it, you maintain it, I think it's a much better model but it brings a whole lot more uncertainty in the planning because there's always smaller bits of work they're going to turn up out of a typical iteration cycle and the stakeholders don't want to wait Two weeks, three weeks, four weeks for that small change.

So on that.

I've experimented with customers on size of iteration time period. I found that three weeks tends to be the best way of starting. So a three week iteration.

In day one, it's really unlikely that the team will be able to go from ideation of hey, we need a new information product to production in three weeks, right. They end up pipelining so they, they break it down as we should and it'll take multiple iterations before it's done or done done. But you know, they're striving to get that cycle time down into three weeks where possible. But everybody in the world always uses a two week iteration, right? Agile is kind of, with Scrum, Agile's kind of said two weeks iteration is now the de facto standard. I mean I've, I've tried four. What I found is that last week was waste for some reason, right? The last week we, you know, the team tended to waste that week for reasons. One was, was just a little bit too fast. We did one day for an experiment, one, one day just for, for a laugh. And that was funny. But you know, getting data and transforming it and visualizing in a day with a team is pretty hard. But what do you find? Do you find that there's a typical iteration lengths?

You start off with my, let's say default is actually two weeks as well. But I have to distinguish here whether the factory is in place already. If the factory is in place, I tend to work in one way in one week iterations, that's challenging. But if you have really automated the main processes from for example getting data into a persistent stage and then from there, for example into a core data warehouse model.

And if you really can focus on the actual requirements onto the modeling because that's what you can't automate. But then it's absolutely feasible. The main challenge we face is the availability of the product owner. So typically we go to two weeks because the product owner in a typical organization of our customer base does not have say a full.

Position or the full FTE for the project. They are typically available for two days a week. And that's why we then try to find out a good combination of what is their availability and then look what is the amount of development capacity, which makes sense. And then it's as well experimenting once again, sometimes we start with a one week iteration and then we find okay, it's too sporty, we just better do it in two weeks. In some other cases we as well had.

Or took a decision as a team to go temporarily to a three week cycle. But it really depends and I really encourage you and as well our listeners to really experiment and do not just, let's say follow some strict guideline in some way, but really experiment and tend to then take a decision, okay, what did work best? And what works best today is perhaps not what works best in three months. So it really depends as well on the situation.

Yeah. And also when you're working in an iterative model or an iterative pattern, they talk about it being in scrum terms. We use the term sprint, but for me it's a marathon because.

It'S continuous, it's constant, there is no downtime on that team.

So for me, mixing it up with say a one day or one week iteration, which is really just a research one or you know, a FedEx style one where the team are focusing on something that's annoying them about the platform or the automation and they're just focusing on fixing it. Changing that cadence from their standard iteration cycle to something different again gives them a change, right. And that change is treated as a brand break. So when we go into greenfields, right, we've got to build the platform because as much as people talk about sas, you know, cloud data warehouses, the reality at the moment is we have some great cloud analytical databases, right? But they're not a data warehouse. They don't have everything we need. And so especially with the current modern data platform paradigm, we've got to go and grab a bunch of technologies and cobble them together to give us the capability we want. And that takes time and effort and expertise. So again, in the original days, I used to, you know, we used to look at iteration zeros, right? We used to look at a bunch of iterations that were just platform builds. And what we found was, you know, we weren't particularly good at guessing, so we knew what some of the core features were. Right. You typically need some data storage, you need some way of running code, you need to transform the data, you need to visualize it. That was a no brainer. But yeah, the details, the tricky stuff in the middle was where the real gold was and so moved more to, you know, building the, the plane as you're flying, which is hard, right? But what it means for, for me is you're only building something when you need it. And so, and then you're testing it straight away, right, to, to prove that it's giving you the value you want. Downside, you know, it's hard to build something when you actually need it. On the day and it makes it longer. Right? It's perceived as longer to get information product out the door because there's all that platform engineering that needs to happen. What about you? Are you still seeing a lot of iteration zeros or are you seeing organizations now kind of building it as they need it?

Again, it's both. I would say, I would go with the 8020 rule. So we invest let's say 20 time or 20% of the time of our inception or iteration zero to let's say trying to build 80% of the needed platform functionality. This we can do typically with a combination of data warehouse automation tool set. So for example the originally New Zealand based tool from Warescape and they already bring you quite a bit of let's say basic or foundational functionality which you just need everywhere or again and again and as well provides you with the basic pattern for various platforms, etc. So that's the first thing. Now what we did as a company or service provider, we build on top certain patterns which helps us to just let's say start very quickly with a customer. For example, we have a pattern for creating such a persistent staging area because it's always the same thing. It's getting the data from a system, let's say model wise in a one to one fashion, maybe just a few data type mappings and adding historization just to having an archive or kind of an archive. We want to have physical delete detection and this is always the same for whatever table you take. Now given the fact or let's say if we stay in the realm of table based data and if we already have this, let's say after half of a day of configuration, then of course you can already start in the second half of this first day loading the first data. And of course there are some troubles connecting to the data source, but this is where you actually solve a concrete issue or problem. But just loading the data into this persistent stage, for example, that's it. Then we bring along a few patterns just to generate a whole data load for for example some dimensional or fact tables where we just need to build in the transformation rules to map the data from the let's say the source or the PSA system and map it to the fields you want to have in the core in the core system. So you can already start modeling from day one because that's part of your requirements analysis. And this is where we work for example with the beam methodology of Lawrence Core.

To really collaboratively working together with business people and creating the data Model. And having now already the machinery in place that we just can generate, let's say, the structural elements in the data warehouse in day one.

The only piece which is missing is then really again the mapping between source and target. And this can be usually done in a few days. So having this automation already in place, both from a, let's say platform perspective, like the data warehouse tool provider, together with some predefined content, then you can actually achieve this kind of speed. And of course, if you don't have this, then maybe you will invest one or two weeks to build this kind of basic patterns on demand or based on what your current need is. So.

Yeah, and so as I said in the beginning, finding people that apply agile and data together is difficult for me. I mean, there's lots of people that do agile.

And one of the things I look for is this idea of an agile data coach. And for me it's somebody who brings patterns that may be reusable with them. So you've just named a bunch of patterns that are core to a data team when they're working and 100% on with you with Lawrence Core and the, and the beam approach, you know, that for me is a pattern that most data teams should adopt for understanding the core business processes and the data that supports it. You know, it's a pattern that's worked for me for 10 years and I love it. It's fun as well. Right, so absolutely, yeah. Anybody hasn't used it, go use it. So at risk of starting a religious war, the other pattern that everybody tends to argue about is the modeling pattern.

Which modeling paradigm do you tend to see customers using the most? And which pattern do you tend to bring with you the most?

Now I'm a bit biased because we are proposing dimensional modeling to our customers and therefore we do not have that many data vault customers because we are not that famous for having this or having the expertise in this model. One reason for that is.

We are still thinking or we are still convinced that tackling the.

BI challenge from a business perspective, which is tangible for people and which is basically understandable, is the biggest benefit you can achieve. So that you have an understandable data model which is understandable by business people, people and in most cases we are working for. And again, this depends again on the size of the company and the number of source systems. But the typical company we are serving, where we are talking about maybe between five to 10 source systems, typically in one, let's say, platform, then a dimensional model is definitely a good fit where you don't need the additional layer of a data vault, for example. And we as well try to mitigate certain elements or certain risk which come along with a dimension model by always providing a persistent staging area as an archive where you can always reload the core if it's absolutely necessary. Of course, we try to omit that, but we have this kind of safety net. And now the question was again, why was this possible to do so? Because earlier on it was just too much effort to build, let's say, these data flows into a persistent stage all the time for all the tables, because you would have to create this manually for every table. Now, as we have all the patterns in place, it doesn't matter whether we have five table, 50 tables or 500 table, you just drag them in, into the tool and say, okay, now generate and that's it. And then you go and grab a coffee and wait until the generation process and the deployment process. Everything can be automated as long as there is no business logic there. And again there. This agility as well brings us this flexibility that we can really work with a model which makes sense from a business perspective, even if it has some downsides from a technical perspective. But, but let's say the overall package definitely works fine.

Yeah, I mean, as you probably know, I'm more of a data vault bigot than a dimensional one. But I take your point on complexity. When you use data vault, you'd never expose the data vault models to the end consumers. You always need a consume layer or presentation or whatever you want to call it. And that may be dimensional, maybe denormalized, because under the covers the model is, is horrendous to look at, but incredibly easy to automate. And I'm with you on PSA on what we call a history layer. That's a bit of a heretic in the data vault world because in theory you're meant to use the vault modeling technique for a raw layer, but we don't. We recommend using a persistent staging, which is mirroring the source system, just historizing the tables. For the same reason, you can reload from scratch, which you could do with a raw vault. But I don't know, it just, it's a pattern that's worked for me a lot. And so therefore it's a pattern that, you know, we tend not to change, just like your experience in dimensional works. And therefore you get the value out of it and so use the patterns that work for you.

I think one, one thing to add there is, it's really, it's really depending on the, let's say the customer organization, whether they have the people with the necessary skills to operate and understand and maintain a data vault. And this is where we see especially for very large organizations, typically that works or they have enough as well internal stuff to do so. But for a bit smaller companies and where BI is mainly business driven, you are just getting to some limitations in terms of okay, understanding highly normalized model like data. Wow.

Whenever there's a divergence in patterns, there's always very strong opinions. I remember the old Inman and Kimball arguments of many years ago. So on with you. It's up to the organization and the teams we're coaching for them to pick the patterns. What we do is we encourage them where we've seen value and ways of working from other teams. That's something we bring.

You know, I have had teams that actually don't model. Now I don't recommend it. Right. My recommendation based on my experience is always pick a modeling technique. There is value in that. But you know, I have had teams that in the way they work, they don't model. They are more of a deploy and destroy and that's their paradigm and it works for them. But I don't see it often and it has to be a conscious choice. Right. The team have had to have experimented and know it works rather than just not modeling because it seems easier. So when we talk about that factory and that automation and we've got the framework or the pattern in for loading new data sources into the persistent stage. So we point at it, we do the horrible roundabout of how do we connect, don't have credentials or yada yada, yada. But we finally get access to the beast of a source system. We start grabbing the data over. We've identified the keys on the table, we're all rocking. Persistent stage or area is hydrated. If we're moving into a new domain, what I find is the team come in and they're kind of back at day one because unless there's a domain expert on the team, they don't understand the core business processes, they don't understand the source data, they don't understand the profile. They've got a whole lot of work before they can start applying their normal patterns. And there's a real temptation to do that work early. Right to, to kind of pipeline it where another team does all that work and hands it over.

And so there's this balance between the, the problem we have where a team walk into iteration planning on day one. Let's say it's a Monday and they have no background information and they're trying to plan their work, right? So they're their, their rate of guessing, they're guessing with a whole lot of uncertainty. So that, that's kind of one end of the spectrum and there's the other end of the spectrum where you know, BA goes away and, and does a whole lot of analysis and insight, doesn't write up a requirements document, but all that knowledge is in one person, right? And, and that work's been done up front, but it's not been done by the team. So it's not as, as consumable. And so for me, a kind of hybrid model, right, A balance between those that work done up front and the team turning up with no insight is that we focus on a lot and it's a difficult problem. How do you deal with that?

So you so called inception phase with some minimalistic upfront design comes into play because we often have the situation that we really start from scratch or on the green field. Not even the customer knows exactly what they want and how things work. And usually it doesn't take that much time to just get some first insights and do some first experiments. So typically maybe two to three days of kind of a hackathon where you just look at first of all the goals you want to achieve, then having some source system guys in place where you can just have a direct look into these tables, etc. Looking what's there, asking questions, doing some modeling canvas work together. So basically it doesn't take that much until you get a first idea how things work and how things relate to each other. And I think one important thing here is establishing a common language. So modeling for me is not only having a data model in the end, but modeling is certainly defining this kind of common vocabulary so that everybody understands what a customer is, what is a product, what is a service, whatever entity you have in your organization or finally then in the data model, I think you can solve a lot of things within only two to three days. And I think it's worth taking this effort to, let's say, get a new team accustomed to each other, but as well to the domain and the topic they should work in. And then establish the plan after having these first few workshops and then start with maybe if it's a really, let's say sophisticated new business domain, why not doing kind of a proof of concept or a little pilot where you don't need even to build the fully fledged data warehouse? What we often do is we just build the persistent stage with a Few tables. Because again, automation, let's assume is already there. Then we just do either a view layer or go directly in a tool like for example, Power bi and then do just some virtual data modeling where we just have a look how could the skeleton look like. And then already show some first data. It's not about having a report which is production ready, but that you can show a first concrete result even after the first three to four days. And this is typically as well, where you can then identify certain misunderstandings in terms of, oh, I didn't mean it like this from a customer perspective or a business perspective.

That'S, let's say, I think a good step in between that you do this kind of virtual data model where you just have a first idea how this could look like. And once you have figured out the main issues during this experiment, then you can tackle these issues already in your backlog, for example, and say, okay, here we see we need to, for example, we have certain data quality issues. So you already know, okay, here we need to. To plan a little bit more time during the Sprint or an upcoming Sprint to address them. Things like this.

Yeah, exactly the same technique that Pat and I recommend. So the idea of a research spike, the idea of timeboxing it. And again, really important to timebox it because as data people we love to go in the detail. So when the team find out they'll need to match customer records from multiple systems. That's enough, right? That's enough statement to know there's some complexity. But they always want to deep dive into how's the matching algorithm going to work. I'm a great fan of canvases. I find that canvases are a good way of getting enough information quickly that the team can look at it and nod together. They get understanding, but they're not going to write a whole lot of detail because at this time it has no value or minimal value. And definitely agree that the best way of getting feedback is put something in front of the customer. Customer. So quick prototypes in front. But being very clear that they can never go to production, they don't meet our definition of done right. So as professionals, we will not push it to production because we have standards we hold ourselves to. And hacking a dashboard and a prototype and making it live for a thousand users is not what we do as professionals. So online there. So two things I want to cover off just before we end is a quick conversation on scaling, which is never a quick conversation cause that's one of the hardest problems. And a second conversation around the state of The BI and data tooling market for agile ways of working heavily. So let's go on the scaling one. So what I find is working with a small team is not easy, but it's easier, right? They're dedicated, they're focused, they're iterating their ways of working, they're self contained, self organizing. The lines of communication are small enough, they tend to just rock it. And it's amazing to watch. As soon as we scale to 10, 20, 30, 40 people, we now have to break things up. We know that a team of 40 working on the same thing is not efficient. So we need to scale them. And we may scale them on a domain, heavens forbid we scale them on the part of the process because not my favorite pattern, we may do it where we scale them that pick up the next thing off the backlog. But now.

We want to reuse patterns, we want to set standards across them and have a light federated governance model, those kind of things. So what's your view? I mean, have you got any magic tips for when you want to scale from a team of five to a team of 30?

It's not magic, but the first thing I do is I typically split those who build the factory and those who use the factory because as well from a skilling perspective.

Those who build the factory typically need to have much more insights in terms of what standards they want to see implemented. They need to maybe think a bit more in a more abstract way because they need always to find a generic kind of solution which then can be turned back again into concrete solutions. So that's the first thing. And depending on how big your factory should become, you can put in quite a lot of people into the factory, which is then much more similar to working on a software engineering project, even though it's still in the data world, etc. And then again, the factory is a highly important aspect, I think in order to then have multiple teams working on, for example, different domains, but always using the same patterns. And this you can't achieve without having let's say the kind of factory.

Let'S say powered by automation, because the automation again as I told before, is always based on standardization because otherwise, otherwise you can't automate. And I think that's again the foundation. Once you have this machinery in place, it's much easier than to have multiple teams on top working in always the same way because they don't have to care about naming conventions, they don't have to care about how certain code is implemented, they can purely focus on the business Requirements, the modeling and let's say transformation logic to map the source data to the target model. So that's, let's say my main, let's say direction I would, would use in order to, to scale up the teams. The other angle we could think of is when we talked about these end to end increments, it's always a question what does end to end mean? And typically or certain authors in the literature, they always say, okay, you always have to go from the source system up to the fully fledged dashboard, et cetera. And only this is agile bi. And then of course there are other authors which say, okay, now everything needs to be splitted. So nearly not every layer in the data warehouse, but perhaps the persistent stage is one layer and this is a user story and then you do a user story for your core layer, etc. And we think that it's somewhere in between. We typically go with let's say.

The core warehouse, where we say the core warehouse is independent of the main business context. Of course there is some business logic, for example when it comes to data integration of various source systems, but usually.

On the level of a data mart and then the final reporting, this is typically driven by a specific use case. And this is as well where we could split teams to say, okay, okay, one team is responsible for really building the data foundation and do the stuff which is rather generic in the context of the organization. And then you have individual, either project or use case driven teams which are then working on really some soft business rules and the implementation of let's say the data model and reporting layer. And there you can as well scale based on this separation of resources.

Yeah, so I agree with you, scaling's hard. You know, pick a pattern, experiment with it, adapt to where it doesn't work a lot of the time. The teams I've worked with, we have moved to what I call platform as a product, right? So we have a squad or a team that are focused on the platform. We tend to introduce the idea of a data platform manager whose role it is is to engage with the other teams or squads to understand what's coming and so you know, they can serve them as their customer. One downside I found with that is some of the fun stuff, some of the platform engineering, some of that hardcore automation then gets taken out of the squads and given to another team that tend to do all the cool stuff and the squads end up kind of being more factory of just move the data and visit. So one of the teams I worked with, they experimented with the squads doing the initial cut of a piece of automation as part of their information product delivery. So they solved their own problem problem with some help from the Platform Squad. So we might have parachuted a Platform Squad member into that that team for a while just help them and then once it had some value then it got moved to the Platform Squad to automate it and make it available as a product to the other teams so they could use that and as you said professional lazy ism. I think Shop Spotify talked about use different terms but the same model which was when something's repeatable and available and automated, a team will typically pick it up and use it because it's easy for them. So you don't need to force them, you just need to encourage them. So yeah, and then you can do it via domain, you can do it via stage of the development cycle. But the key thing is experiment and adapt. So if it's not working for you, change the model. So yeah, but scaling is hard. So last one, state of the data the tooling market to support agile ways of working. What's your view on it at the moment?

Definitely better than 10 years ago.

So when we started 10 years ago I read about things like data warehouse automation and definitely there were some vendors back then, but it was still on a rather rudimentary basis I would say. And nowadays we have have quite a vivid market of tool vendors in the area of automation. What I prefer having their tools and of course again whereascape where you have still a certain flexibility where you can let's say influence what code is generated and even though out of the box stuff is great to start very quickly, you will always end up with certain let's say customer organization specific things which you need to adapt. This is something where I would look for that you don't lose the flexibility to control what's actually generated to be not that much dependent on a specific vendor. Second thing I would like to mention are data or data warehouse specific testing tools which is something which isn't on the market that long I would say or at least there I see as well.

Interesting development so that you can first of all automate test cases or running the test cases. And what's as well a very interesting combination is when you manage to combine the data warehouse automation tool with the test automation tool so that you can let's say that the test automation can for example use the metadata of like to derive what tables to be tested having the full lineage for example to know okay, if this is my PSA table, the source table in the source system is this and this table for example. And I think this makes it very powerful in terms of as well making sure that the data quality is there and that you don't have regression issues. And the third thing I would like to mention is that that agility or this tooling stuff is not only there in the backend, in the data backend, it's as well there in the front end because the same is true in terms of standardization. So maybe you have heard of the IBCs, the International Business Communication Standards and using such we call it a notation standard where you define or where it's somehow predefined how chart should look like, how tables should look like. There's then there are as well tool providers typically add ins for existing tools like Power bi so that you can apply this standard automatically or pretty much out of the box. And this again speeds up the whole process from requirements engineering because you don't need to rediscover would I like to have now this chart in blue, yellow, orange or whatever color. It's just predefined to a certain degree. And as well the development is very quick because. Because let's say when I used to invest multiple days, if not weeks to create a fancy chart, you get much more out of the box again based on the standard. And I think that's why I said it's much better than 10 years ago through the whole stack from really getting the data in testing the data we have a lot more of automation aspects away available nowadays and of course as well the whole deployment pipelines, devop tools. So I think we did good progress but we are still behind the software industry or software engineering world. They are still, let's say, 10 years before us.

Yeah, definitely. And especially when we talk about testing. Right. You know, automated testing of data is. We are so woeful, it's getting better. But I think we see waves and patterns in the technology space and like the first tool you use, crystal reporting got bought out and consumed into a big beast and then completely lost. I think we're going to see a wave now of all the little niche tools that add value to our lives getting merged into big behemoths until the next wave goes. So just to close out, I started my journey reading a couple of books. I read some of the Agile Data or BI books from Ralph Hughes and Ken College earlier.

And they really illuminated my journey for me. And then there were a bunch of other books that came out. Lawrence's book around Beam, Hans's book around Data Vault that were more specialized, they were dealing with one small part. And yes, they use the word agile, but they weren't really talking about the end to end process and how it fitted. They were just talking about some really good patterns that you could use in there. And then we kind of had a gap, right? I couldn't, there's not a lot came out. And then we got a bunch of what I would call whitewash books, right? We got a bunch of books that had the word agile in IT and data, but really they were technology books that talked step by step how to implement, you know, SQL Server analysis services or you know, a bunch of tools and technologies. And look, they were valuable if you hadn't done your platform build before. But they were all focused on technology, not ways of working. So you're writing a book, right?

I know books take a while to write and obviously based on what we're talking today, I'm assuming your book is going to be about the end to end process, about that way of working on patterns you can leverage. So is that true and when can I start reading it?

You can already start reading it now because we are of course right now the book in an agile way. So we are doing iterations chapter by chapter and we are publishing, let's say better releases of the book. So maybe we can post the link on the show notes, but you can easily register there and become a better reader. So currently we have the first six chapters already available and then you can as well give feedback if you want, what you would like to see in addition or where something didn't make sense for you, for example. So that's the current process. So we plan to have a first draft ready this spring somewhere and then hopefully the book can be published somewhere in 2022 already. Now in terms of how do we want to be different to already existing books is basically first of all we are addressing non IT people in that book. So because our main stakeholders we are currently working with are typically controllers for example, or marketing analysts, etc. So really business people. And there we see that there is they didn't want or they don't want to read too much of a technical book, they are more interested in the overall process, for example. And that's why we are currently or the book is designed as a city guide. So we depicted the city called Agile BI City and we are taking our readers onto a sightseeing tour through this city where we show them around various districts and we start for example with a district called Inception beach and Inception beach you learn how you do this minimal upfront design. Then we go further to Wishing Hills. So this is more about, okay, what role does a BI strategy play? For example, to enable agility in bi. Then we go to west site requirements, for example, where it's all about gathering requirements. Then we go to, I guess we have contracting south, so how about Agile contracting downtown patterns. Then we go to Technology Land or a district called Technology Land, I think. So we are really walking the reader through the various steps, the various, I would say building blocks you typically need. And of course you do not have to visit or let's say then adapt this into your whole in your own company to have all the different areas. But I think it gives you a good overview. What are the elements you need to think of and then adapt accordingly to your own situation.

That's cool. I mean I love the way that it's giving the information by telling a story in a different way. I mean one of the, my, one of the books I did enjoy that came out around Agile but not around data was the Phoenix Project. And I enjoyed it because it told a good story and then as part of telling that good story, you learn some things. So yeah, that idea of a city guide, that's cool. That's something I haven't seen before, so I look forward to reading it this year. So before we close out anything else, top of mind for you about Agile and BI and data that you want to talk about about.

So I think what's for me again and again, the most important part, show the value as early as possible.

Even though we talked about building a factory, etc. Don't invest too much into the factory before showing some concrete value. Maybe just having a simple report or a quick win where you can show the value to your stakeholders. I think that's the most important point. Once, once they know that there is value in their data, then you can start focusing on building the necessary foundation to have a really sustainable solution, et cetera. But don't forget about the value to show as well and make this value visible to your stakeholders.

Yeah, no, I can't agree more. Thanks for your time. It's been an awesome talk. We've covered a lot of bullet points, but they were all things that people should think about when they're doing Agile BI or Agile Data with your team. So thank you for your time again and we'll catch you later.

Yes, thanks for having me. Bye.

Data Magicians was another Agile data podcast. If you'd like to learn more on applying an agile way of working to your data and analytics, head over to agiledata IO.