Data Engine Thinking Patterns with Roelant Vos - Episode #66

Welcome to the Agile Data Podcast, where we talk about the merging of agile product and data patterns in a simply magical way. My first book, An Agile Data Guide to Information Product Canvas, is published on Amazon. If you hear your stakeholders say, "Hey, that's not what I wanted," and then you hear your team reply with, but that's what you asked for, then you probably have a problem capturing data requirements. The Information Product Canvas is a pattern template that is a quick and simple way to solve that problem. Head over to agiledataguides.com and click on the green book link to buy a copy now. Welcome to the Agile Data Podcast. I'm Shane Gibson.

Hey, I'm Roland Vos. Thanks for having me.

Hey, Roland. Thank you for coming on the show. Today, we're going to talk about the patterns that are in your book, Data Engine Thinking. But before we do that, why don't you give the audience a bit of background about yourself?

Yeah, sure. Thanks, Shane. I'm Roland Vos. I'm a Dutch national, but I've been living in Australia for almost 17 years now. So I've got this weird blend of accents and I'll try to not bother too much with it, but I've been working in the data space for more than 25 years, so pretty technical guy. I've started with computer science and then got my first job as a Cognos consultant, so building reports and cubes and everything, and quickly realized that building these reports is— that's good fun and it's important, and obviously it's what people often consume their data with, but the issues that you find in the data, you can't easily or should easily fix them in the reporting side. Slowly but surely descending into the depths of data management and ETL, data integration, all all the way down to database administration, more the real hardcore technical, low-level stuff. And then slowly making my way back into the light. Took a management position at Allianz for a long time. And now I'm back to more the technical programming area of things. Because there is this pendulum always, I find, right? So as a data professional, you see these problems in organizations and you want to fix it, make it better, make the data more easy to consume. And to an extent you can with your tools and skills and all this stuff. But problems are always often partly organizational as well. So then you find yourself going into more management and trying to change your organization to work better with data, but you lose the technical aspect of it a little bit. So it is always this back and forth between I want to do something to improve the organization and also want to keep up to date with technical skills. So that's been my story all along.

And it'd be fair to say that over those 25 years, you've focused a lot on automation. Before the term DataOps came out, I always remember a lot of your writing and thinking was around how to automate some of the tasks so that you don't have to do them manually.

Absolutely. Yep. So I started that in 2000, actually, with TCL, which was part of Oracle, Oracle Database at the moment. So when we were working with Oracle Warehouse Builder, may it rest in peace, we used this TCL language to try to automate things. And we came up with ways to change the physical modeling to align better with automation because obviously refactoring takes a lot of time and want to speed it up. And that's also when we ran into more hybrid modeling techniques, such as Data Vault and things like that, because that was already more mature than we were doing, but it still aligns with the same mindset of modular patterns that are easier to automate, quote unquote. So that whole approach towards making life easier with automation, that was, it started out as we can do this and it's awesome, saves a bit of time, but it evolved into this mindset of working with data is based on a series of assumptions which are translated into a model and translated into an interpretation of the world, right? Literally, what a model is. And those interpretations are always going to be flawed because you start with a certain background that you may not have the full awareness of understanding of everything is in place and the history of why things are now where they are. That idea that your interpretation of reality, your model, is always going to be flawed hooks into that automation concept because you want to change your interpretations the more you understand the data, the organization, the history, the knowledge that is in people's minds, social memory. And the more you learn, the more you want to refine that model. And then you have to go back every time to update your codebase. And that's why automation is such a critical thing. So automation and model-driven engineering, pattern-based design, those kinds of terms have always been the red thread through, through the work. Yeah.

And for me, it's, it's like idea that if the cost of change is lower, then we're more willing to change.

100%.

We can manage more change more often. We can iterate more often. We can take more risks. We can make earlier guesses because we know the cost of refining that guess in the future is lower than if everything was manual, everything's hard-coded, everything involves human effort that's large. We just won't take the risk because we want to do a lot of design up front. Because we don't want to invoke that high cost of change. I'm with you there. The ability to automate, it gives us the ability to experiment.

Absolutely. And then that links into things like data warehouse automation and the tooling that exists and the patterns that exist for it. So then you think about what kind of components do we need? How can we really specifically define them so they are reusable and they tick all the right boxes? And I'll get to what those boxes are in a bit. But my experience with the tooling back in the day was not always super great, right? They always felt there's a couple of gaps that aren't addressed and sometimes made to be looked too easy because certain things were missing. And at the same time, it shouldn't be that hard because once you figure it out and you embed it once, then you can use it as many times as you want, right? So there is this feeling that always lingering that the patterns that we were using weren't as correct as they should be. That pattern and that's, I guess, that implementation approach and put it into something that is truly generic, truly ticks all the boxes, then we can park it and leave it behind. Because once we figured it once and for all, then we're done. But all these different tools and different frameworks that many people use, there was always some issue with it that sort of made it incompatible. And the way I look at it, these systems, these data solutions as I call them, they're not complicated, but they're complex, right? There's a lot of tiny moving bits and pieces that need to align. And this is what the book is about, the Data Engineer Thinking book, because Once we talk through end-to-end what it means to set up an architecture that is designed for change and what these components are that need to work together, what kind of frameworks you need to think about, the message is that correct viable pathways to implement things, but you need to understand if you change one of the components on the left-hand side, then it has downstream impacts on the components on the right-hand side. So all these components, they need to be aligned, need to be in sync with each other so they deliver the correct functional result. And that's really what it is. So there's a couple of options, but you need to understand what the consequences are of those options and make a decision that you then capture in your design. And then you move on. Working through that allows us to then build code generation templates and automate everything from there.

Cool. So what is the book about?

So Data Engine Thinking is an end-to-end overview of what it means to create a data solution that is designed for change. So what's What kind of frameworks do we need? What kind of steps do we need to think about? What kind of decisions do we need to make along the way? And it's split into two general sort of storylines. So one is that the theory, say, how do we treat with key distribution, bitemporal data, late-arriving changes? How do we manage time? How do we manage information models versus physical models? All this stuff, right? So that's bulk of the content. And then on the other hand, you've got a fictitious company called FastChangeCo. That we follow from the start all the way to the implementation of that concept. Because there's a couple of reasons for that. One is that we feel like some of the implementation choices that you have are really opinionated, and that's in some cases rightfully, in some cases it's— we try to make the point it doesn't really matter, but we try to take that emotional aspect and put it into the book storyline so we can have that conversation about pros and cons and what it means and how you get to decision outside of the standards, if you will. So it's not like you have to do this, but these are your options. This is what we would recommend. And then we keep the emotional aspect of it out of it. And at the same time, it follows typical discussions and typical dynamics that you have in a given company, right? So how do you get to an implementation of a new data warehouse or data solution or anything? So it's that first thing, let's define a vision. What are the current problems? How do we have the conversation about let's buy a tool versus we need to spend time on information modeling. How do we translate from the information model to a physical implementation? What kind of technology shell or methodology should we select? And then all the way to how do we introduce automation? Why does it matter? How do we implement testing? How do we roll it out into DevOps? All the way to the end result. Okay.

So, did I hear that you pick up the pattern for the book out of the Goal and the Phoenix Project, where you tell a story to reinforce some of the messaging? That you've also, rather than having somebody have to guess what the patterns are, because if you go for the Phoenix Project and you read it, after a while you go, "Oh, holy shit, this is about DevOps. I can start seeing it." But it comes as a bit of a shock, right, if you haven't read it before.

Yeah.

So I think what you're saying is you're kind of calling out the patterns and then using a quasi-real-life example story to then give a reinforcement about how you decide between the options, because they're all good options. But given the situation, certain patterns are more valuable given the context and others become anti-patterns. So combining both, describing the pattern and then telling a story to give us a reinforcing model of how you choose and how you use. Is that right?

Yep. That's right.

Yeah.

And that goes hand in hand all the way through the book. And yeah, we hope that people can use this to implement a fully automated solution themselves. Should be really practical.

Okay. And then the other thing is I tend to talk about information value stream and information factory, and I'll give you the difference. So information value stream is when a team work out How are they going to identify a problem in the organization? How are they going to ideate the possible choices they have of solving that problem? How are they going to discover which option looks like it's the most viable, feasible, valuable? How are they then going to get the work prioritized so it can go into their delivery queue? So it's more product thinking on the left-hand side. I'm using my hands here. And then on the right-hand side, once it's been prioritized, they pick it up and they do some light design. They do some building, some deployment. They release that value back to the stakeholders, they may maintain or decommission over time, and then they go back and it's like a continuum. They can aiming for speed through that so they can find a problem, deliver back quickly.

Yep.

An information factory is more about thinking about the way we build the platforms that support that work and the way we move data through it all the way from collection through to consumption. And where we can look at automation, where we can use platforms and tools to automate work that has value when it's automated and try and turn that work into as much of a factory as possible, still realizing that it's half art, half science, because no data ever survives our first engagement. So using those two models, I get the feeling that data engine thinking's about that information factory, about how you make decisions around the vision for the platform, the components you need to build for the platform, the way you make decisions on the trade-offs that you have on every component. How you automate it, how you deploy it, all that engineering craft and practice. Is that right?

Yeah, that's spot on, right? Definitely more in the information factory area. So as part of the modeling and the, I guess, information gathering and the interactions with the quote unquote business side of the company, the interface with that is the information model. So we use a couple of examples, but stick to FCOAM mostly, where we say you need to have an understandable way of communicating with everybody in a way that is unambiguous. We feel like verbalizations and FCOAM concepts in general are really suited for that. but we don't really talk about how you workshop that. We just say, look, you need to do that, but then you feed in the results of that into your engine, and then everything will reflect automatically and look the same way, or however you want it, not the same way, but yeah, the result of that decision.

Okay. And so when you talk about information model and the pattern that you're adopting, are you talking about a business model, a conceptual model of the data for the organization?

Yep. 100%. Yeah. So we make the case that the, that is true IP and you want to use things that are available in the market at the moment for that. But definitely at the physical, pretty much automatically. So that's also a pattern that you can decide on and codify and then run. So definitely the, the conceptual model is the input and then everything downstream will be generated.

And then often I talk about hydration. So I talk about if you get your conceptual model right and then you understand the attributes, which is a form of logical modeling, then you should be able to hydrate your physical model using any modeling pattern you want. You should be able to deploy it as anchor or data vault or dimensional or relational because actually it's like you said, it's that conceptual understanding of the data that's more important. And then the physical modeling can change depending on the technology, the user, the tools, all those constraints that you have that make one physical modeling pattern seem better for you in that situation versus another, versus the old days where we were dogmatic and we were Data Vault for everything or Dimensional or not quite Inman versus Kimball, but definitely Data Vault versus Dimensional was a key scrap a few years ago from memory.

100%. In that context, we define the term pattern into a design pattern and a solution pattern. So the design pattern is more the, what do we need to do? How should it work? What are the sort of conceptual boxes that we're trying to tick? And then the solution pattern is the, specific implementation on a given technology. So the design pattern itself is holistic, generic, technology agnostic, all that stuff. And that, that includes translating the logical model to the physical model, for example. So you have a specific idea on how this should work in technology X, then you've got to switch over into something else when you need to. So the technology becomes less of a, of a thing. Yes, you need to optimize it, but it's also not really where the IP resides.

Okay. And so if I play that back, 'cause I was thinking about the complexity of the paths. So I can go and say there's a design pattern of historized data. So where data changes in a source system, we want to be able to see every historical view instantiation of that data over time. So that's like a big pattern in my head. Was that what you call a design pattern? Yeah. And then I can go, if I deploy SCD Type 2 dimension, that gives me a way of storing historized data, or I could do a satellite satellite off Data Vault Hub, that gives me the ability to store historization. I can reconcile change records in a single table. There's a whole lot of technical patterns for the physical modeling. And then I could do that using dbt, or I could use that doing Oracle Warehouse Builder if I still had it. I could do it in SQL code, I could do it in Python, heaven forbid, I could do it in Java or Scala, if I'm back on the Hadoop stage. And then actually, I could physically store it in some form of Databricks, some form of Fabric, some form of Snowflake, S3, and they just all— Or both. Yeah. Yeah, or as we always do in all of them. So they're just solution patterns. They're just different ways. But people often struggle with the complexity because we're almost in an OLAP cube now, which is, hey, I've got this one simple design pattern, historize data. I know, sorry, I didn't bring in my data layers. So I could be persistent staging, EDW, presentation, or I could be bronze, gold, silver. I have a whole lot of different language just for the layers. So do you cover the way you should think about it when you have all these complexities to take it from a design pattern to a solution pattern?

Yeah, and it's actually one of the more difficult things we found to explain and to write, and we ended up moving text and content around quite a bit to try to explain that the best way we can. So first of all, the solution architecture can partly be the collection of all the design patterns. So you've got specific patterns applied to specific architectures is another way of looking at it. So we start saying these are the high-level architectures you can think about, right? The high-level building blocks. DSA was mentioned, Précision was mentioned. And we say that at that level, you need to make a call and include one of these design patterns in your solution design, because that will have a lot of downstream impact. And we try and explain there, if you don't do this, then you need to think about this. And if you do this, then you don't do this. But if you do that, then— so it's almost like a decision tree of options that you need to think about. And that's a critical first part because Once that is in place, it changes the way how the next layer or downstream processes will work. For example, if you don't have PSA, your refactoring, your reloading of historical data in a different mode will look different. But you can still do it, it will just be different. So the complexity is shifting that way a little bit. If you use an insert-only pattern instead, or also allow updates in the data, it shifts the way how you interpret the timelines when you deliver data for information. So we need to take a couple of high-level decisions and understand what the implications are.

And if I think about it, the way we've dealt with this in the past is we have blueprints or reference architecture. So if you look at any experienced data architect, that's both a data architect and a data platform architect, they effectively have a small number of blueprints in their head that have been crafted for certain styles of organization, large versus small, distributed versus not, complex data, fast-moving data, big data where we talk about terabytes, and then a whole lot of other patterns. And they then bucket them into blueprints that are templates that they bring first, and then they iterate it for the things that don't fit for that context of that organization. I think what you're telling me is you've taken that upper level to say, actually, here's the building blocks of those blueprints and reference architectures. Here's the choices that you need to make that will tell you which block to adopt. And then the consequence later or earlier in your information factory that decision will make.

Yeah, exactly. And so that's a good way to, to put it at that level. And what we also feel is that once we define that reference architecture and we do select one, right? So we've got 12 different blueprints, if you will, and then one that we say, this is our reference, we'll just continue with that, but keep those in mind and refer to the other ones throughout the book. But we also feel that this complexity is something that can be understood and managed encoded, codified once to the point that we then don't have to worry about it as much. And that's sometimes a controversial topic, right? So basically what we're saying is it doesn't matter who you are. If you use that approach, it does everything you need and just making sure that one solution will always work. And again, not talking about the technical level, the solution pattern, because the infrastructure and technology will be slightly different from case, but the architect, the design of the platform itself should be the same for everyone.

So let me ask a couple of questions to clarify what you've just said. So I can interpret that in what I call an anti-pattern, which is whenever you're modeling Party role is the only table you need because it covers every use case for every organization. And I'm not personally a great fan of that pattern. Or I can infer it as you saying, historize data by default because you're gonna need it. and therefore, if you don't do it, you're gonna have to do it at another stage. It always has to be done. Just decide which pattern you're gonna use to do it, and here's your choices. It's a design block. We do need to historize data. Now we just need to figure out which solution design we're gonna use to create the historization.

Yeah, pretty much. Yeah, it's definitely second. So the partial thing, that's more at the modeling side, and there's actually a chapter on that, how that would behave if you played it out and why it's not a good idea, but I guess the point I'm trying to make is in a given architecture to make it do what we want it to do, which is being endlessly refactorable and all that stuff, right? So you can morph your model and move with organization, all that good stuff. You have to have a couple of things. You have to have this relationship point. You have to specify what we call an assertion timeline and state timelines, like a technical and a business equivalent, right? Because you have to be able to handle late-arriving transactions, future dates, and data stuff. So one of these blocks is to have these two timelines in place at all times and solve the problems associated with it, like the bitemporal. Like, so these are complex things, but we, we argue you have to have all of them—historization, bitemporality, a way of delivering information where you can make some decisions here and there. But there's a couple of these core things to, to make it work. Having business keys is one. Defining historization at column level is one. Or attribute level. If you don't have that, then it will not work. But we also argue to truly work with data in your organization, there's no avoiding them, right? Sooner or later you will run into these problems, so you might as well tackle it up front. Embrace that complexity, make sure it's understood and codified once and once and for all, and you don't have to worry about it as much because at that point it becomes pattern-based, right? Because you know that you need two timelines, you know that The way to fix timeline gaps or issues in date math, all that stuff is the same stuff over and over again. The pattern for delivering data is gonna be the same no matter what you do. So these are complex things by themselves.

I'm gonna drill down on that in a second. I just wanna go back and get a couple more examples, cuz bitemporality is quite a deep thing to understand as a data person. So if I go back to a couple of examples that I think are a little bit more simpler to understand that I always see in a data platform. Platform. So one is reference data, the ability to have a bunch of data with some lookups where it's not created in a source system. It's in Excel, Google Sheets, or in a Word document. And we need to be able to capture that data and iterate it in the data platform. So I see that one a lot. The ability to file drop, the ability to grab a CSV file, a JSON file, an XML file, or heavens forbid, an Excel file. And manually drop it into the platform for an ad hoc task is one I see all the time. Okay. So those are design pattern blocks. They're things that you don't have to build it on day one, but you're going to build it. If your team and your data platform survive for a couple of years, you will create those features because everybody wants them.

Do that, then make sure that these boxes are ticked because it needs to adhere to the fundamental principles everything else has to.

Okay. And then if we look at an example of ECD2 and hubsets and links, So the value of those patterns is you can write SCD2 code and it's the same code. It is a known solution pattern. You write that code, you can harden that code. And if you want to create an SCD2 type dimension, that code becomes bulletproof. It's the same one of the reasons I like Vault, because you can create the code that says create hub, create link, create set, load hub, load link, load set. You do all the exceptions ones, of course, but those 6 pieces of code become bulletproof. They become reusable. They are used so often. That actually every bug in every use case gets solved, that you can just trust that code. And I think what you're saying is we take that and apply it to the design patterns and say, okay, let's come up with solution patterns that get used time and time again. Then we're effectively iterating those solution patterns, giving it more test rounds, more coverage, that they become bulletproof. And they become easier to implement. And therefore, Why wouldn't we do historization if we could just drop a solution pattern in and know it's going to work? We don't have to build it ourselves from scratch. Is that what you're saying?

Absolutely. And extend that to more complicated things like the bitemporality you mentioned. Okay.

And so when you get these complex use cases, then what you're looking for is a proven solution pattern so you don't have to invent it yourself. And you're effectively dropping that Lego block and the problem's taken away to a degree.

And that's controversial, right? It's controversial, but I don't know why.

So let's discuss that because in the DevOps world, you create Terraform templates to deploy your servers as cattle. You protect the Terraform code like nothing else. You deploy your servers, your server away, you redeploy it. It's a proven pattern. In data, we seem to like to handcraft things.

Oddly. Yeah, I'm with you. There's a couple of answers. The first one is that there's no set of patterns that has been sufficiently enough described to tick all the boxes that actually work in, in all scenarios. So what works for one methodology doesn't work for some other one, or there is a couple of gaps that a couple of shortcuts are taken to avoid some of the nasty things that may happen, and then you run into them and then you have to make changes. So it's not matured enough yet. It's almost there, but not quite yet. So that's also a reason why a lot of handcrafting is, is in place, but The other thing is also that data professionals seem to be a bit shy of exposing too much complexity if they can avoid it. So they would opt for simpler patterns if that ticks the box for a little while longer, and then don't worry about the things that will go wrong in the next 6 months or so. So let's not make it too complicated because what will people think? And I sense that quite a bit. And I'm like, yeah, look, Apple satellites is a great example, the Data Vault satellites, right? So yes, historization, that's easy. We argue you need to set up your satellites for bitemporality always, basically, because if you don't, you will run into the problem and then you have to go back and invent the wheel and change your patterns. And, and there's no proper tool that fully supports that. And you go into hand coding and it gets all kinds of weird. So it's that combination of not everything has been fully thought through yet, in my opinion, combined with let's be a bit careful with how complex we want to make things, because what will people think necessary?

So Yeah, we've talked about this off and off over the years, and I struggle to find a template or a pattern to describe patterns. So it's one of those things that I can write a name, I can write a description, I can provide some context about where it has value or not. I can provide one example, but then it just starts getting hard. And for somebody to be able to pick up a pattern and read it and actually understand how to apply it is something that I've always struggled with. With your design patterns, how far down that path did you get with the book?

We have defined templates, one for the design patterns that we follow throughout the book. By the way, the book will come when we launch it. We'll also launch a GitHub which has all these code examples and patterns and things you can run yourself and templates and everything. So everything we've used in the past, obviously in our projects, and then add it to the book, will be publicly available also with a view on collaborating on it further. So these templates will be there and it has a couple of things, but having a template with a couple of headers is never going to fully cut it because you need more of a dynamic, interactive sort of thing. So the way I look at it is that the sections that we've defined in the book will go through all the options and considerations as part of that pattern, to use the word again, error exception handling reference data. So we say this is what you need to think about. And then at these points, we say in your template, include this and explain to Serity why you are doing this. Why am I excluding or including this? Why am I making this decision? So we basically provided this checklist of stuff that you then need to include in the template, and that will be part of your solution design. Acknowledging that it's almost like a graph model, right? Like how these things hang together. So also these pattern definitions. But we think that's a good start because it, it it elevates it to a higher level concept and then breaks it up into smaller pieces that you can include or exclude.

Okay. So it becomes a pattern library effectively. And then some golden paths or guided paths of the way you would typically put the patterns together. Because there are, like you said, there are a common set that we all do. And then there are the exceptions that we end up doing because the organization context for some reason is so different to everybody else that we go, ah, actually, We have to deal with that one differently here for this reason. But we only deal with that as an exception because I don't know about you, but the number of times I've worked with an organization that said they're special and unique, and I'm sitting there going, uh, I've seen this before a few times. I'm not quite sure you are special or unique.

Still special, not unique, right?

Yeah, true. And then we end up spending 6 to 12 months building something they never use because they said they wanted it and they never actually use it.

Yeah. And the second part is always an interesting thing because the more complexity you add, the more the more time, the more maintenance, all that stuff. So again, automation is key to take that concern away, right? Because we feel you need that complexity, but we also understand you don't want to build like the Jaguar for when you also only need a Mazda 2 or whatever to, to get where you need to be initially. But then at least take a couple of options to prepare you for the future that you will get to if you ever get to there. But automation is essential there. So once you have that foundation, in order, then you do have some options to scale out. On that note, and also the notion of, I guess, the complexity that we've attributed, there's this ideal that I have that I've been talking about for years and years and years. And it's also one of the red threads in the book, even though I don't really make it as explicit often. It's the virtualization idea, right? So my dream is still to have this virtual representation of data that doesn't require persisting of data through all these layers and areas, all the complexity. You just have the data, you need to historize it in this most fundamental way. But after that, all the things that you put on top of that are derived the same way from the information model as you would traditionally do by creating physical model, creating through data logistics, like quote-unquote ETL processes and all that data movement, right? So all that stuff can be replaced with the virtual interpretation. Funny thing is it still needs to tick the same boxes. You still need bitemporality, historization, reference data, all the things we've talked about. But it's such a powerful idea, and I think in the book we try to always come back to that and say, if you do this, think about how it would behave in this virtual world. Can we still do this? Can I still go to one version of the data and then to another version of data, back to the original version of data, and make it look the exact same way, right? Deterministic processing. If the answer is yes, then you know your solution works. If the answer is no, you're going to get in trouble. Similarly for things like, do I Select an insert-only approach, or do I support updates in my data? Right. So if you do an SCD2 as your example earlier, do I update the end dates? Can I derive that? Why is it necessary? Does it matter? How do we do things like driving key link satellites in an insert-only approach? For example, how do we do bitemporality in insert-only approach? The funny thing is you can do all that. There's no limitation on doing that, but it will have some impacts. Your consumption of data is going to work. So that kind of idea is something that continues to fascinate me.

I'm the same. We've experimented with fully virtualized in our platform. Form as a core concept. And yeah, we always end up regressing back to physicalization of data at some points because we had a constraint either in cost or performance, but more often we had a constraint in physicalization in our thinking. When we bring in the next pattern and it's all virtualized, we lose the ability to trace the physical data flow to understand whether we've got their patent applied properly or not. And I'll give you an example. It's that graph problem. There's just too many things going on for our little brains to, little brains thinks. So for example, we run a form of Data Vault modeling for our physical structure, but not for our conceptual structures. And we treat linked tables as event tables effectively. So we can say, customer ordered product from this store and this employee is an event. And we hydrate our consume layer. So you can see all the data denormalized into one big table. So for that event, you can see the customer and all their details, the order and all their details, the product and all their details, the store and all its details, and the employee, right, in one table. So that's, that was first cut, easy. And then you go, okay, now we want to make sure that when we hydrate that, we're picking up the right version of the data. That existed at the point in time that event happened. Okay, now we've got to go and take our Hubs and Sets or our DIMMs, and we've got to find the right point in time to hydrate it so that it was looking like the time that event happened. And again, we can physicalize that and we can then virtualize it. And then we go, well, our ETL, our transformation code, we call rules, the code or the logic that's applying some change to that data. Again, we can virtualize it. But now if we wanted to say, actually, We're gonna virtualize it. We gotta say this event happened at this point in time. This was the raw data for each of those core concepts at that point in time. And this was the business rule that was being applied at exactly that point in time, and then switch it out. While for reasonable volumes of data, the platform could probably handle it, we can't see that graph. It becomes so complex that to test every each case and make sure the pattern's hardened becomes so intensive that we go, is it worth it right now? And what you are saying is if that pattern was available to us and we could just apply it. Then hell yeah, we would, because we know it's a proven brand, we can trust it's already been tested. We don't have to craft it ourselves, and then we can trust it to be virtualized. So I think that's part of it is when you have to get that graph in your head and every piece of complexity you add, because it does become complex, makes it harder to prove that your code is doing what it should for every use case. That's the problem that we need to solve.

Yeah, absolutely. And there's so many to unpack there, right? To start, I always say we're in the business of establishing trust. So we always are delivering the data. So we have to prove it's correct. And by correct, I don't mean according to the expectation of the business, but we need to be able to audit to the original data at any point in time, because if we don't, we're done. Trust is easy to get, quick to lose, all of that. So having that auditability and that lineage is essential. And it ties into versioning that we have to version control the metadata and the code generation templates hand in hand with data, which is built in, but those things need to be versioned together to make that work. And the third is the virtualization. There's too many practical barriers at the moment still to make that realistic, but I see it as a business test to make sure that the patterns that you do have are working as intended. So if you can do it, you can also physicalize and/or materialize and do all these other things. But if you can't virtualize it, then something's wrong with the patterns. So it's more like a test to make sure everything is correct. And then when performance and those bottlenecks are resolved potentially, or the datasets are manageable enough, The ideal that I have to mitigate those kinds of concerns that you mentioned is that almost the same as how API versions work, in my view. So you have this data solution that acts as a service for data consumers to extract information, and they use that data as per a specific version of availability, like a release, right? You've got the model looks like this, data following these rules, etc. So you get the data, and then if you change something, anything in your design, you could theoretically, again, spin up a second version that is version 1.1 next to the original version, give people some time to cut over and to understand what's happening. And all those lineages and everything will still be captured in your version control history and visible. So that's how I look at it. And again, this is ideal world, but I do think thinking about your solution in that terms helps to make it as robust as it can be.

And I think technology has moved on, yeah, because we use Google BigQuery under the covers and we can virtualize all the way through.

There's always that constraint that you have to work with.

Yeah, I remember the old days of Oracle servers and we used to have to get them ordered in Singapore and shipped over and more memory put in. And then we got really expensive Teradata boxes that could handle the volume but cost a fortune. The guys from Otherduck talk about it's not a big data problem for many people. And so I think that's true. So I think we're a lot closer of the technology being able to support virtualization. I just don't think as data professionals we are. And this idea of having reasonable patterns that then have solution model patterns to support it, that then have proven code, gives us one step towards being safer, being able to virtualize all the way through. Especially with things like zero-copy cloning, where we can actually, don't have to take a full copy of the data back to test things out. So maybe the first step is whenever you're doing any development, do it in an end-to-end virtualized pattern. To test that it works and then physicalize when you have a cost or performance constraint.

That's the final chapter of the book. So the engine concept is where we start the book with, and it explains all these sort of building blocks of what constructs a fully automated machine. And one of the components is optimizer. And we say exactly that, right? So start virtual, but depending on which optimization directive you want to give it, like cost, storage, I/O, latency, whatever it is, it can calculate based on usage patterns what the best combination of physical virtual objects are to deliver that, right? So if you don't want cost because you're running on Snowflake, for example, or something like that, then, you know, physicalize some parts because it will help you in optimizing for that directive. But if you want latency, maybe just don't do that. Just, just physicalize the end result, keep everything virtual. So it becomes a little playground for making things work the best way you can. It's really fun.

And then you're going to have to bring in the segmentation or domain-based or business unit-based or process-based. So you say these data flows need to be low latency.

Exactly. How cool is that, right? I want this mark to be refreshed every minute and this one every hour, right? And this is literally always saying the book that becomes a requirement because as a data team, you have limited resources no matter how you look at it. If you want to optimize that, it becomes more of a management decision and then you get more resources to deliver that requirement and you just update the directions and go.

So if I took that future state of virtualize everything and it actually worked, I could also see a future state where you basically pull up the design patterns, you tick some boxes for them, fill out the constraints around which technologies it's allowed to use, and it actually writes the solution patterns and the code and deploys itself as a fully config-driven end-to-end platform from a couple of— there'd be a lot of checkboxes though, a lot of complexity.

In our definition, the design patterns would remain the same, but the different solution patterns will be selected by the engine to deliver the updated physical implementation.

So are you saying that the design patterns become mandatory in your opinion?

Yeah, yeah, because that defines how your solution operates, right? The historization, how to do reference data, all those things we've discussed earlier. So you need to make a decision how you want to treat all those big blocks. But for each of design patterns, you've got a number of solution patterns for different technologies or concepts or anything So you can switch between those and then let the solution refactor.

Okay. So let's explore that a little bit because again, it's just bringing this idea of agility, which is ability to change at low cost and with speed, not Scrum. And so there's this balance about work done upfront. So there's times where you do work upfront where you don't need it right now, cuz you know it's gonna save you time in the future, but it is waste cuz you're doing it before you need it. And in fact, it's a really interesting conversation with Patrick Lager at the moment on LinkedIn where he's saying he actually doesn't do any work upfront anymore. It's if he doesn't need to build it, he doesn't build it. And so I'm going to have to have a whole chat with him now about, hold on, doesn't that just become a whole bunch of ad hoc? Now he's got enough experience to probably know how to retrofit the patterns after the fact. Okay. But it is, it's not go off on that tangent. So let's go a couple of examples. So I would say you always want to bring in a historization pattern into your platform on day one, especially if you can just pull it out of the pattern library because you are gonna need it. I could then say, if I was looking for feature flags for analytics for machine learning, so I wanted to be able to create very wide, very long table that was keyed by customer that had a whole lot of feature flags, their age bucket, their usage bucket, their income bucket that I was gonna put into a machine learning model to do some clustering segmentation. I would probably not do that work until I had an information product that needed that work to be done. That would be my standard approach right now. What I think you're saying is if the, as a design pattern, and it is, and if the solution patterns and the implementation code was readily available and proven, that box will be ticked by default because at some stage I would need it and the cost of implementing it is almost nothing. Is that what you're saying or did I misunderstand that?

Partly. So in this particular example, I would argue that every delivery of data uses the same pattern. So no matter how you deliver, like everything normalized or one big flat table, that's all the same pattern you use. You just select different columns, but conceptually, yes. So the way I look at it, you would have a couple of solution patterns readily available because we provide them with the book and the GitHub and all that stuff, but it might not be fitting your technology and all that kind of stuff. So you would then add them as you need them, I would think. So good looks like having a library of patterns that we can all collaborate on that is somewhat centrally available and we can work together as community to improve them. That's a bit of an ideal world dream, but in the smaller groups, that would certainly be possible. But I think that if your solution pattern doesn't exist for the specific technology that you're working with, yeah, you would have to add it.

And in that case, I would only add it when I need to add But that idea of a pattern library we can all share and collect, that happens in the software domain. Our software brethrens have that already. They've done the work to make that happen and got the value out of it. In the data domain, we don't, right? We go and implement a new piece of technology or a new vendor. We go, oh, SQL Server, I'm going to do Snowflake now. And we reinvent all the patterns again. And it's, oh, we're going to go Databricks. Oh, let me reinvent all the patterns again from scratch. Because I'm changing the technology stack.

And that's just crazy. That's just— yes. So much to learn from our friends in software development. Yeah.

But I still see large platform builds. I still see projects where it's 6 months to a year before the platform is available to the team to actually build something of value. That has changed a little bit, but not a lot in my view. Not a lot.

So there's room for growth there. And I think having a library like that to use and contribute to So hopefully it will help get it moving to that direction. We've talked about this before, right? And those plans, they're still there.

I've got to say, at least you've been following this dream for a while. So I can see that actually to write this book, there are at least 3 challenges, 3 big challenges that just come to mind. So one is building out this mental framework of design patterns and what they are, solution patterns and code that implement those solutions and how you describe them, how you link them, the kind of meta-model for that model. That's the first problem. The second one is then taking your 25 years of experience. Dirk's been in, in the industry about as long. Yep. So taking 25 years of experience from two people and then distilling that down into those patterns, like just picking them out of your brain and writing them down. So that you get this library of all the patterns that you've ever implemented and that are valuable. And then the third one is figuring out how the hell you tell that complex story in a book.

100%. Yep. And it's taken us almost 7 years, right? So that's something. And I do find, as you point out, the Data Engine Thinking is co-written by Dirk and myself, and we've, we both worked very, very hard on it. And the interesting thing is that there has never been a case where we didn't agree on something, right? So it's like this convergent evolution kind of thing. You work towards something that you know works and people get there. And not just us, but other people as well. You slowly move towards this approach that makes sense in the present. So it's been very harmonious is probably the right word to use. It's— we never had any sort of disagreement on how things should be done. There were a couple of cases where we didn't know something yet, right? So a couple of errors that we run into and say, hey, actually I never thought about that, or I wonder what would happen if that, that happens, and then sort of explore that and started coding and wrote a couple of examples. And then certain cases where timelines or time periods, right, from start date, end date, that kind of thing, were missing in, um, not in an expected pattern or something like that, right? Or what is the consequences if the source system, operational system, doesn't have that time validity and So to really explain what happens then, oh yeah, that's interesting, but never that we weren't on the same page. So I think that's pretty, pretty special.

And I also think the fact that you are creating code and applying it against data scenarios to prove that both the solution pattern and the design pattern have legs, right? Are real things that you can describe. Actually probably helped in terms of validating the book as you go. Here's the design pattern, here's what we think the solution pattern is, let's create some code and if it survives and does what the patent says it should do, then we know it's a real thing. Yes, absolutely. Because there seems to be a few books that have come out over the last couple of years. I read them and I go, that's great theory, but where's the meat and potatoes, right? How do I do that? Because that seems great. It seems like nirvana and we've been trying to get to that for the 35 years I've been in data, but we've never made it. So just tell me how to do it for once. Yeah. And this book seems to be taking us to that state, right?

Where there is an example. Certainly.

And so into June, you reckon? Yep.

We got the Global Data Summit coming up in Iceland. So Dirk will be there to present the book and everything. So that's when the physical copy should be available and everything should be online as well. So that's going to happen. Yeah. But events. Yeah.

I was hoping to be at Iceland this year, but for a lot of personal reasons, it's not going to happen. And then I was hoping to be at the Helsinki Data Week and then they moved it. To when I wasn't going to be in the UK. So I missed that one as well. I'm happy that 2025 seems to be the year of the books. There's a hell of a lot of really good data books coming out. So I'm hoping 2026 will be the year of good conferences for me.

Yeah. See you there.

One of the interesting questions Joe Reese asks on his podcast just lately, and I'm going to go steal it, thanks Joe, is this idea of how did you write the book? What was the process you used to write this book? You just Talk me through how the two of you wrote it together.

I'm not sure if you actually follow the process. So Dirk and I both have been giving classroom training for a long time. Then our focus areas are slightly different, but it's similar stuff. And the idea of having a storyline that supports the more quote-unquote dry materials, the theoretical parts, we agreed on that really early. Back in the day, we thought, ah, maybe we don't want to make it like a data cookbook or anything like that, right? There are some data components, but we want to make it a bit more than that. So I started to write the main theoretical chapters and Dirk started to write the main storyline chapters, and then we would peer review our work. And then revising that, sort of, then we switched, swapped roles a little bit, saying, now I'm going to update that storyline with the theoretical explanation that I have in mind and then tweak a little bit. And then Dirk would do the same and review. Change some of the explanations or, you know, add what he wants to say. So it's like that, right? So we both write a chapter, storyline versus theory, and then we switch roles and then we do it again.

So effectively, you storyboarded the chapter and then wrote in isolation and then swapped to iterate and align. And then it would be interesting to see which chapters you had close alignment after the first iteration and which chapters you deviated from each other more than some of the other chapters would be interesting. Yeah.

Yeah. I think especially on the biggest work we've done is on the definition of what the architecture would be, the different options, the big boxes that we've talked about earlier. That took us the most time to really nail that. And the other one is the different ways of looking at temporality and data. That also took a long time. Those are the real big things. Yeah.

Seems simple, right? As well as actually, it really is. Seems until you, again, until you engage with the first bit of data. And one of the things I found, and it's only after I've done the retro of publishing my book that I realized how valuable it was, but actually creating a course for the content in the book before you write the book and then actually training people for me was something I hadn't realized how valuable it was because by creating the content, you're actually drafting the book. You're just not writing the words. You're drafting images and you talk it and then you write the hands-on that people do to reinforce it. and then as you're writing the book, you're flicking back between iterating the course and the book because you're starting to learn a different way of articulating that seems better, or you're getting feedback on the course that you need to bring in. And so that gives you a safe kind of cool testing ground to actually learn and do at the same time.

Is that what you found? 100%. Yep. Yeah. So at some point I started recording the trainings for that purpose. And then after the training sessions, I was updating the notes in the slide decks to find the right words. And that all made it into the book. And at the end, I just opened training again and started removing slides from the deck if I was happy enough that it was sufficiently covered in the book until there weren't any to resonate.

Oh, I hadn't thought about doing that actually. Yeah. Picking up the course material and deleting it when you know the book's covered it to prove you've got the coverage on it. Aren't slipping stuff in. Yeah. What I actually found was I ended up writing more stuff in the book that I haven't put back in the course yet. Yes. Because then I've got to extend the course, make it longer, and I'm like, I'm not sure I want to make it any longer. Exactly. Yeah. Yeah. So it's that trade-off decision. Yeah. All right. If people want to get hold of you, what's the best way for them to find you?

Yeah, find me on rolandvoort.com. That's my website with the blog that I have ignored slightly because I was so busy writing the book. But that's where you can find me. And also dataenginethinking.com, which has the book website, and it has some examples, and it has some preview chapters and some of the personas. So definitely find us there.

Excellent. And join It'll be out on Amazon, no doubt. For sure.

Yeah. And we're also exploring some other platforms, including our own website, but yeah, definitely Amazon to start with. Excellent.

Right. I look forward to buying that one and finally getting a design pattern template that I can use repeatedly and stop hating the ones that I use currently. So thank you for that. At least that's going to be the best value that I've had for a long time.

I generally hope so. Excellent.

Hey, look, thanks for coming on the podcast and I hope everybody has a magical day. And that, data magicians, was another Agile Data podcast episode. If you want to learn new patterns that you can apply to craft your own Agile Data way of working, head over to agiledata.io.