Data Engine Thinking Patterns with Roelant Vos - Episode #66

Foreign.

Welcome to the Agile Data Podcast where we talk about the merging of Agile Pro products and data patterns in a simply magical way.

My first book, An Agile Data Guide to Information Product Canvas, is published on Amazon. If you hear your stakeholders say hey, that's not what I wanted, and then you hear your team reply with but that's what you ask for, then you probably have a problem capturing data requirements. The Information Product Canvas is a pattern template that is a quick and simple way to solve that problem. Head over to agiledataguides.com and click on the Green Book link to buy a copy now.

Welcome to the Agile Data Podcast. I'm Shane Gibson.

Hey, I'm Rolands Voss. Thanks for having me.

Hey Roland, thank you for coming on the show. Today we're going to talk about the patterns that are in your book, Data Engine Thinking. But before we do that, why don't you give the audience a bit of background about yourself?

Yeah, sure. Thanks Shane. I'm Roland Voss. I'm a Dutch national but I've been living in Australia for almost 17 years now. So I've got this weird blend of accents and I'll try to not bother too much with it, but I've been working in the data space for more than 25 years, so pretty technical guy. I've started with computer science and then got my first job as a cognos consultant. So building reports and cubes and everything and quickly realized that building these reports is good fun and it's important and obviously it's what people often consume their data with. But the issues that you find in the data, you can't easily or should easily fix them in the reporting site. Slowly but surely descending into the depths of data management and ETL data integration all, all the way down to database administration, more the real hardcore technical low level stuff and then slowly making my way back into light. Took a management position at alliance for a long time and now I'm back to more the technical programming area of things because there is this pendulum always I find, right. So as a data professional you see these problems in organizations and you want to fix it, make it better, make the data more easy to consume and to an extent you can with your tools and skills and all this stuff. But problems are always often partly organizational as well. So then you find yourself going into more management and trying to change your organization to work better with data, but you lose the technical aspect of it a little bit. So it is always this back and forth between I want to do something to improve the organization and also Want to keep up to date with technical skills. So that's been my story all along.

And it'd be fair to say that over those 25 years you're focused a lot on automation. Before the term data ops came out. I always remember a lot of your writing and thinking was around how to automate some of the tasks so that you don't have to do them manually.

Absolutely, yep. So I started that at 2000 actually with TCL, which was part of Oracle Oracle Metadatabase at the moment. So when we're working with Oracle Warehouse Builder made rest in peace. We use this TCL language to try to automate things. And we came up with ways to change the physical modeling to align better with automation because obviously refactoring it takes a lot of time and want to speed it up. And that's also when we run into more hybrid modeling techniques such as data vaults and things like that, because that was already more mature than we were doing. But it still aligns with the same mindset of modular patterns that are easier to automate, quote, unquote. So that whole approach towards making life easier with automation, that was started out as we can do this and it's awesome, saves a bit of time, but it evolved into this mindset of working with data is based on a series of assumptions which are translated into a model and translated into a. An interpretation of the world. Right. Literally what a model is. And those interpretations are always going to be flawed because you start with a certain background that you may not have the full awareness of understanding of everything is in place and the history of why things are now where they are. That idea that your interpretation of reality, your model is always going to be flawed hooks into that automation concept because you want to change your interpretations. The more you understand the data, the organization, the history, the knowledge that is in people's minds to show memory. And the more you learn, the more you want to refine that model and then you have to go back every time to update your code base. And that's why automation is such a critical thing. So automation and model driven engineering, pattern based design, those kinds of terms have always been the red thread through, through the work. Yeah.

And for me it's idea that if the cost of change is lower, then we're more willing to change 100%. We can manage more change more often, we can iterate more often, we can take more risks, we can make earlier guesses because we know the cost of refining that case in the future is lower than if everything was manual, everything's Hard coded. Everything involves human effort that's large. We just won't take the risk because we want to do a lot of design up front because we don't want to invoke that high cost of change. I'm with you. That the ability to automate gives us the ability to experiment.

Absolutely. And then that links into things like data warehouse automation and the tooling that exists and the patterns that exist for it. So then you think about what kind of components do we need? How can we really specifically define them so they are reusable and they tick all the right boxes. And I'll get to what those boxes are in a bit. But my experience with the tooling back in the day was not always super great. Right. They always felt there is a couple of gaps that aren't addressed and sometimes made to be looked too easy because certain things were missing. And at the same time it shouldn't be that hard because once you figure it out and you embed it once, then you can use it as many times as you want. Right. So there is this feeling that always lingering that the patterns that we were using weren't as correct as they should be. That pattern. And that's, I guess, that implementation approach and put it into something that is truly generic, truly ticks all the boxes, then we can park it and leave it behind. Because once we figured it once and for all, then we're done. But all these different tools and different frameworks that many people use, there was always some issue with it that sort of made it incompatible. And the way I look at it, these systems, these data solutions, as I call them, they're not complicated, but they're complex. Right. There's a lot of tiny moving bits and pieces that need to align. And this is what the book is about, the data engine thinking book. Because once we talk through end to end, what it means to set up an architecture that is designed for change and what these components are that need to work together, what kind of frameworks need to think about the message is that correct, viable pathways to implement things. But you need to understand if you change one of the components on the left hand side, then it has downstream impacts on the components on the right hand side. So all these components, they need to be aligned, need to be in sync with each other so they deliver the correct functional result. And that's really what it is. There's a couple of options, but you need to understand what the consequences are of those options and make a decision that you then capture in your design and then you move on working through that allows us to then build co generation templates and automate everything from there.

Cool. So what is the book about?

So data engine thinking is an end to end overview of what it means to create a data solution that is designed for change. So what frameworks do we need? What kind of steps do we need to think about? What kind of decisions do we need to make along the way? And it's split into two general sort of storylines. So one is that the theory say, how do we treat with key distribution by temporal data late driving changes, how do we manage time, how do we manage information models versus physical models? All this stuff, right? So that's bulk of the content. And then on the other hand you've got a fictitious company called Fast ChangeCo that we follow from the start all the way to the implementation of that concept. Because there's a couple of reasons for that. One is that we feel like some of the implementation choices that you have are really opinionated and that's in some cases rightfully in some cases it's. We try to make the point, it doesn't really matter, but we try to take that emotional aspect and put it into the book storyline so we can have that conversation about pros and cons and what it means and how you get to decision outside of the standards, if you will. So it's not like you have to do this, but these are your options, this is what we would recommend. And then we keep the emotional aspect of it out of it. And at the same time it follows typical discussions and dynamics that you have in a given company. Right? So how do you get to an implementation of a new data warehouse or data solution or anything? So it's that first thing, let's define a vision. What are the current problems? How do we have the conversation about let's buy a tool versus we need to spend time in information modeling. How do we translate from the information model to a physical implementation, what kind of technology or methodology should be select? And then all the way to how do we introduce automation? Why does it matter? How do we implement testing? How do we roll it out into DevOps all the way to the end result?

Okay, so did I hear that? You pick up the pattern for the book out of the goal and the Phoenix project, where you tell a story to reinforce some of the messaging that you've also rather than having somebody have to guess what the patterns are. Because if you go for the Phoenix project and you read it, after a while you go, oh, holy shit, this is about DevOps. I can start seeing it, but it comes as a bit of a shock. Right. If you haven't read it before.

Yeah, yeah.

So I think what you're saying is you're kind of calling out the patterns and then using a quasi real life example story to then give a reinforcement about how you decide between the options. Because they're all good options, but given the situation, certain patterns are more valuable given the context and others become anti patterns. So combining both describing the pattern and then telling a story to give us a reinforcing model of how you choose and how you use. Is that right?

Yep, that's right, yeah. And that goes hand in hand all the way through the book. And yeah, we hope that people can use this to implement a fully automated solution themselves. Should be really practical.

Okay. And then the other thing is, I tend to talk about Information Value Stream and Information Factory and I'll give you the difference. So Information Value Stream is when a team work out how they're going to identify a problem in the organization, how they're going to ideate the possible choices they have of solving that problem, how they're going to discover which option looks like it's the most viable, feasible, valuable, how they're then going to get the work prioritized that can go into their delivery queue. So it's more product thinking on the left hand side, I'm using my hands here. And then on the right hand side, once it's been prioritized, they pick it up and they do some light design, they do some building, some deployment, they, they release that value back to stakeholders. They may maintain or decommission over time and then they go back and it's like a continuum. They can aiming for speed through that so they can find a problem, deliver back quickly.

Yep.

An information factory is more about thinking about the way we build the platforms that support that work and the way we move data through it all the way from collection through to consumption. And where we can look at automation, where we can use platforms and tools to automate work that has value when it's automated and try and turn that work into as much of a factory as possible. Still realizing that it's half art, half science, because no data ever survives our first engagement. So using those two models, I get the feeling that data engine thinking's about that information factory, about how you make decisions around the vision for the platform, the components you need to build for the platform, the way you make decisions on the trade offs that you have on every component, how you automate it, how you deploy it all that engineering craft and practice, is that right?

Yeah, that's spot on. Right. Definitely more in the information factory area. So as part of the modeling and the, I guess, information gathering and the interactions with quote unquote, business side of the company, the interface with that is the information model. So we use a couple of examples, but stick to FCOAM mostly, where we say you need to have a understandable way of communicate with everybody in a way that is unambiguous. We feel like verbalizations and FCOM concepts in general are really suited for that. But we don't really talk about how you workshop that. We just say, look, you need to do that, but then you feed in the results of that into your engine and then everything will refactor automatically and look the same way or however you want it. Not the same way, but yeah, the result of that decision.

Okay, and so when you talk about information model and the pattern that you're adopting, are you talking about business model, a conceptual model of the data for the organization?

Yep, 100%. Yeah. So we make the case that that is true IP and you want to use things that are available in the market at the moment for that, but definitely at physical, pretty much automatically. So that's also a pattern that you can decide on and codify and then run. So definitely the conception model is the input and then everything downstream will be generated.

And then often I talk about hydration. So I talk about if you get your conceptual model right and then you understand the attributes, which is a form of logical modeling, then you should be able to hydrate your physical mod using any modeling pattern you want. You should be able to deploy it as anchor or data vault or dimensional or relational. Because actually it's like you said, it's that conceptual understanding of the data that's more important. And then the physical modeling can change depending on the technology, the user, the tools, all those constraints that you have that make one physical modeling pattern seem better for you in that situation versus another versus the old days where we were dogmatic and we were data vault for everything or dimensional or not quite Inman versus gimbal. But definitely data vault versus dimensional was a key scrap a few years ago from memory.

A hundred percent. In that context, we define the term pattern into a design pattern and a solution pattern. So the design pattern is more the what do we need to do? How should it work? What are the sort of conceptual boxes that we try to take? And then the solution pattern is the specific implementation on a given technology. So the design pattern itself is Holistic, generic technology agnostic, all that stuff. And that, that includes translating the logical model to the physical model, for example. So you have a specific idea on how this should work in technology X, then you've got to switch over into something else when you need to. So the technology becomes less of a, of a thing. Yes, you need to optimize it, but it's also not really where the ip.

Okay. And so if I play that back because I was thinking about the complexity of the path, so I can go and say there's a design pattern of historized data. So where data changes in the source system, we want to be able to see every historical view instantiation of that data over time. So that's like a big pattern in my head. Is that what you call a design pattern?

Yeah.

And then I can go, If I deploy SCD Type 2 dimension, that gives me a way of storing historized data. Or I could do a satell light off data vault hub that gives me the ability to store historization. I can recon stan change records in a single table. There's a whole lot of technical patterns for the physical modeling. And then I could do that using DVT or I could use that doing Oracle Warehouse Builder if I still had it. I could do it in SQL code, I could do it in Python, heaven forbid, I could do it in Java or Scala if I'm back on the HADOOPI stage. And then actually I could physically store it in some form of data bricks, some form of fabric, some form of snowflake history. And they just all.

Or both.

Yeah, yeah. Or as we always do in all of them. So they're just solution patterns, they're just different ways. But people often struggle with the complexity because we're almost in an OLAP cube now, which is, hey, I've got this one simple design pattern historize data. I know, sorry. I didn't bring in my data layers. So I could be persistent staging EDW presentation or I could be bronze, gold, silver. I have a whole lot of different language just for the layers. So do you cover the way you should think about it when you have all these complexities to take it from a design pattern to a solution pattern.

Yeah. And it's actually one of the more difficult things we found to explain and to write and we ended up moving text and content around quite a bit to try to explain that the best way we can. So first of all, the solution architecture can partly as the collection of auto design patterns. So you've got specific patterns Applied to specific architectures is another way of looking at it. So we start saying these are the high level architectures you can think about, right? The high level building blocks. PSA was mentioned, precision was mentioned. And we say that at that level you need to make a call and include one of these design patterns in your solution design because that will have a lot of downstream impact. And we try to explain there, if you don't do this, then you need to think about this, and if you do this, then you don't do this. But if you do that then. So it's almost like a decision tree of options that you need to think about. And that's a critical first part because once that is in place, it changes the way how the next layer or downstream processes will work. For example, if you don't have psa, you're refactoring, you're reloading of historical data in a different mode will look different, you can still do it, it will just be different. So the complexity is shifting that way a little bit. If you use an insert only pattern or also allow updates in the data, it shifts the way how you interpret the timelines when you deliver data for information. So we need to take a couple of high level decisions and understand what the implications are.

And if I think about it, the way we've dealt with this in the past is we have blueprints or reference architectures. So if you look at any experience, data architect, that's both a data architect and a data platform architect, they effectively have a small number of blueprints in their head that have been crafted for certain styles of organization, large versus small, distributed versus not complex data, fast moving data, big data, where we talk about terabytes and then a whole lot of other patterns and they then bucket them into blueprints that are templates that they bring first and then they iterate it for the things that don't fit for that context of that organization. I think what you're telling me is you've taken that up a level to say actually here's the building blocks of those blueprints and reference architectures, here's the choices that you need to make that will tell you which block to adopt and then the consequence later or earlier in your information factory that decision will make.

Yeah, exactly. And so that's a good way to put it at that level. And what we also feel is that once we define that reference architecture and we do select one, right, so we've got 12 different blueprints, if you will, and then one that we say this is our reference, we'll just continue with that, but keep those in mind and refer to the other ones throughout the book. But we also feel that this complexity is something that can be understood and managed and encoded, codified ones to the point that we then don't have to worry about it as much. And that's sometimes a controversial topic. Right. So basically what we're saying is it doesn't matter who you are, if you use that approach, it does everything you need and just making sure that one solution will always work. And again, not talking about the technical level, the solution pattern, because the infrastructure and technology will be slightly different from case, but the architect, the design of the platform itself should be the same for everyone.

So let me ask a couple of questions to clarify what you've just said. So I can interpret that in what I call an anti pattern, which is whenever your modeling party role is the only table you need because it covers every use case for every organization and I'm not personally a great fan of that pattern or I can infer it as you saying, historize data by default because you're going to need it and therefore if you don't do it, you're going to have to do it at another stage. It always has to be done. Just decide which pattern you're going to use to do it and here's your choices. It's a design block we do need to historize data. Now we just need to figure out which solution design we're going to use to create the historization.

Yeah, pretty much. That's definitely second. So the particular thing that's more at the modeling side, and there's actually a chapter on that, how that would behave if you played it out and why it's not a good idea. But I guess the point I'm trying to make is in a given architecture, to make it do what we want it to do, which is being endlessly refactorable and all that stuff. Right? So you can morph your model and move with organization, all that good stuff. You have to have a couple of things. You have to have the historic relationship point, you have to specify what we call a assertion timeline and state timelines, like a technical one, a business equivalent, right. Because you have to be able to handle late arriving transaction, future dates and data stuff. So one of these blocks is to have these two timelines in place at all times and solve the problems associated with it, like the bi temporal text. So these are complex things, but we, we argue you have to have all of them. Historization, bi temporality, a way of delivering information. You can make some decisions here and there, but there's a couple of these core things to to make it work. Having business cases. One defining historization at column level is one or attribute level. If you don't have that, then it will not work. But we also argue to truly work with data in your organization, there's no avoiding them. Right? Sooner or later you will run into these problems. So you might as well tackle it up front. Embrace that complexity, make sure it's understood and codified once and once and for all. And you don't have to worry about it as much. Because at that point it becomes pattern based. Because you know that you need two timelines. You know that the way to fix timeline gaps or issues in date matter. All that stuff is the same stuff over and over again. The pattern for delivering data is going to be the same no matter what you do. So these are complex things by themselves.

I'm going to drill down on that in a second. I just want to go back and get a couple more examples because itemporeality is quite a deep thing to understand as a data person. So if I go back to a couple of examples that I think are a little bit more simpler to understand that I always see in a data platform. So one is reference data. The ability to have a bunch of data with some lookups where it's not created in a source system, it's in Excel, Google sheets or in a Word document. And we need to be able to capture that data and iterate it in the data platform. So I see that one a lot. The ability to file drop, the ability to grab a CSV file, a JSON file, an XML file, heavens forbid, an Excel file and manually drop it into the platform for an ad hoc task is one I see all the time. Okay, so those are design pattern blocks, the things that you don't have to build it on day one, but you're going to build it. If your team and your data platform survive for a couple of years, you will create those features because everybody wants them do that.

Then make sure that these boxes are ticked. Because it needs to adhere to the fundamental principles. Everything else has to.

Okay. And then if we look at an example of ECD2 and hub sets and links. So the value of those patterns is you can write SCD2 code and it's the same code. It is a known solution pattern. You write that code, you can harden that code. And if you want to create an SCD2 type, dimension, that code becomes bulletproof. It's the same. One of the reasons I like Vault, because you can create the code that says create hub, create link, create set, load, hub, load, link, load, set. You do the exceptions ones, of course, but those six pieces of code become bulletproof, they become reusable. They are used so often that actually every bug and every use case gets solved that you can just trust that code. And I think what you're saying is we take that and apply it to the design patterns and say, okay, let's come up with solution patterns that get used time and time again, then we're effectively iterating those solution patterns, giving it more test rounds, more coverage, that they become bulletproof and they become easier to implement. And therefore, why wouldn't we do historization if we could just drop a solution pattern in and know it's going to work, we don't have to build it ourselves from scratch, Is that what you're saying?

Absolutely. And extend that to more complicated things like the BI temporality you mentioned.

Okay. And so when you get these complex use cases, then what you're looking for is a proven solution pattern so you don't have to invent it yourself and you're effectively dropping that Lego block and the problem's taken away to a degree.

And that's controversial, right?

It's controversial, but I don't know why. So let's discuss that. Because in the DevOps world, you create terraform templates to deploy your servers as cattle. You protect the terraform code like nothing else. You deploy your servers, your server away, you redeploy it. It's a proven pattern in data, we seem to like to handcraft things oddly.

Yeah, I'm with you. There's a couple of answers that. The first one is that there's no set of patterns that has been sufficiently enough described to tick all the boxes that actually work in in all scenarios. So what works for one methodology doesn't work for some other one. Or there is a couple of gaps that a couple shortcuts are taken to avoid some of the nasty things that may happen and then you run into them and then you have to make changes. So it's not matured enough yet. It's almost there, but not quite yet. So that's also a reason why a lot of handcrafting is in place. But the other thing is also that data professionals seem to be a bit shy of exposing too much complexity if they can avoid it. So they would opt for simpler patterns if that ticks the box. For a little while longer and then don't worry about the things that will go wrong in the next six months or so. So let's not make it too complicated because what will people think? And I sense it quite a bit. And I'm like, yeah, for satellites is a great example. The data for satellites, right? So, yes, historization, that's easy. But we argue you need to set up your satellites for bi temporality always, basically, because if you don't, you will run into the problem and then you have to go back and invent the wheel and change your patterns and there's no proper tool that fully supports that and you go into hand coding and it gets all kinds of weird. So it's that combination of not everything has been fully thought through yet, in my opinion, combined with let's be a bit careful with how complex we want to make things, because what will people think necessary.

So, yeah, we've talked about this off and off over the years, and I struggle to find a template or a pattern to describe patterns. So it's one of those things that I can write a name, I can write a description, I can provide some context about where it has value or not. I can provide one example. But then it just starts getting hard. And for somebody to be able to pick up a pattern and read it and actually understand how to apply it is something that I've always struggled with. With your design patterns, how far down that path did you get with the book?

We have defined two templates, one for the design that we follow throughout the book. By the way, the book will come when we launch it. We'll also launch a GitHub which has all these code examples and patterns and things you can run yourself and templates and everything. So everything we've used in the past, obviously in our projects and then added to the book will be publicly available also with a view on collaborating on it further. So these templates will be there and it has a couple of things, but having a template with a couple of headers is never going to fully cut it because you need more of a dynamic, interactive sort of thing. So the way I look at it is that the sections that we've defined in the book will go through all the options and considerations as part of that pattern, to use the word again, error, exception, handling reference data. So we say this is what you need to think about. And then at these points, we say in your template, include this and explain to 30 why you are doing this. Why am I excluding or including this? Why am I making this decision. So we basically provided this checklist of stuff that you then need to include in the template and that will be part of your solution design. Acknowledging that it's almost like a graph model. Right. Like how these things hang together. So also these pattern definitions. But we think that's a good start because it's. It elevates it to a higher level concept and then breaks it up into smaller pieces that you can include or exclude. Okay.

So it becomes a patent library effectively and then some golden paths or guided paths of the way you would typically put the patterns together. Because there are, like you said, there are a common set that we all do. And then there are the exceptions that we end up doing because the organization context for some reason is so different to everybody else that we go, actually we have to deal with that one differently here for this reason. But we only deal with that as an exception because I don't know about you, but the number of times I've worked with an organization that said they're special and unique and I'm sitting there going, I've seen this before a few times, I'm not quite sure you are.

Special, unique, still special and not unique, right?

Yeah, true. And then we end up spending six to 12 months building something they never use because they said they wanted it and they never actually use it.

Yeah. And the second part is always an interesting thing because the more complexity you add, the more overhead, the more time, the more maintenance, all that stuff. So again, automation is key to take that concern away. Right. Because we feel you need that complexity, but we also understand you don't want to build like the Jaguar for when you also only need a Mazda 2 or whatever to. To get where you need to be initially. But then at least take a couple of options to prepare you for the future that you will get to, if you ever get to there. But automation is essential there. So once you have that foundation in order, then you do have some options to scale out. On that note, and also the notion of, I guess, the complexity that we've attributed, there's this ideal that I have that I've been talking about for years and years and years and it's also one of the red threads in the book, even though I don't really make it as explicit often. It's a virtualization idea. Right. So my dream is still to have this virtual representation of data that it doesn't require persisting of data through all these layers and areas. All the complexity, you just have the data, you need to historize it in this most fundamental way. But after that, all the things that you put on top of that are derived the same way from the information model as you would traditionally do by trading physical model creating data logistics like ETL processes and all that data movement, Right? So all that stuff can be replaced with the virtual interpretation. Funny thing is it still needs to tick the same boxes. You still need by temporality, historization, reference data, all the things we've talked about. But it's such a powerful idea and I think in the book we try to always come back to that and say, if you do this, think about how it would behave in this virtual world. Can we still do this? Can I still go to one version of the data and then to another version of data, back to the original version of the data and make it look the exact same way? Right. Deterministic processing. If the answer is yes, then you know your solution works. If the answer is no, you're going to get in trouble. Similarly, for things like do I select an insert only approach or do I support updates in my data? Right? So if you do an std2 as your example earlier, do I update the end dates? Can I derive that? Why is it necessary? Does it matter? How do we do things like driving key link satellites in an insert only approach? For example, how do we do by temporality in insert only approach? The funny thing is you can do all that. There's no limitation on doing that, but it will have some impacts your consumption of data and is going to work. So that kind of idea is something that continues to fascinate me.

I'm the same we've experimented with fully virtualized in our plat as a core concept. And yeah, we always end up regressing back to physicalization of data at some points because we had a constraint either in cost or performance. But more often we had a constraint in physicalization in our thinking. When we bring in the next pattern and it's all virtualized, we lose the ability to trace the physical data flow to understand whether we've got their pattern applied properly or not. And I'll give you an example. It's that graph problem. There's just too many things going on for our little brains to little brains things. So for example, we run a form of data vault modeling for our physical structure, but not for our conceptual structures. And we treat link tables as event tables effectively. So we can say customer ordered product from this store and this employee is an event. And you hydrate our consume layer so you can see all the data denormalized into one big table. So for that event, you can see the customer and all their details, the order and all their details, the product and all their details, the store and all its details, and the employee right in one table. So that was first cut easy. And then you go, okay, now we want to make sure that when we hydrate that we're picking up the right version of the data that existed at the point in time that event happened. Okay, now we've got to go and take our hubs and sets or our dimms, and we've got to find the right point in time to hydrate it so that it was looking like the time that event happened. And again, we can physicalize that and we can then virtualize it. And then we go, well, our etl, our transformation code, we call rules. The code or the logic that's applying some change to that data. Again, we can virtualize it, but now if we wanted to say, actually we're going to virtualize it. So we've got to say this event happened at this point in time. This was the raw data for each of those core concepts at that point in time. And this was the business rule that was being applied at exactly that point in time. And then switch it out while the reasonable volumes of data, the platform could probably handle it. We can't see that graph. It becomes so complex that to test every edge case and make sure the pattern's hardened becomes so intensive that we go, is it worth it right now? And what you're saying is if that pattern was available to us and we could just apply it, then hell yeah, we would, because we know it's a proven pattern. We can trust it's already been tested, we don't have to craft it ourselves. And then we can trust it to be virtualized. So I think that's part of it is when you have to get that graph in your head and every piece of complexity you add, because it does become complex, makes it harder to prove that your code is doing what it should for every use case. That's the problem that we need to solve.

Yeah, absolutely. And there's so many to unpack there. Right. To start, I always say we're in the business of establishing trust, so we always are delivering the data. So we have to prove it's correct. And by correct, I don't mean according to explanation for the business, but we need to be able to audit to the original data at any point in time, because if we don't, we're done. Trust the easy to get, quick to lose all of that. So having that auditability and that lineage is essential. And it ties into versioning that we have to version control the metadata and the code generation templates hand in hand with data which is built in. But those things need to be versioned together to make that work. And the third is the virtualization. There's too many practical barriers at the moment still to make that realistic. But I see it as a business test to make sure that the patterns that you do have are working as intended. So if you can do it, you can also physicalize and, or materialize and do all these other things. But if you can't virtualize it, then something's wrong with patterns. So it's more like a test to make sure everything is correct. And then when performance and those bottlenecks are resolved potentially or the data sets are manageable enough. The ideal that I have to mitigate those kinds of concerns that you mentioned is that almost the same as how API versions work in my view. So you have this data solution that acts as a service for data consumers to extract their information and they use that data as per a specific version of availability, like a release, right? You've got the model looks like this data, follow these rules, et cetera. So you get the data and then if you change something, anything in your design, you could theoretically again spin up a second version that is version 1.1 next to the original version. Give people some time to cut over and to understand what's happening and all those lineages and everything will still be captured in your version control history and visible. So that's how I look at it. And again, this is ideal world, but I do think thinking about your solution in that terms helps to make it as robust as it can be.

And I think technology's moved on because we use Google bigquery under the covers and we can virtualize all the way through.

There's always that constraints that you have to work with.

Yeah, I remember the old days of Oracle servers and we used to have to get them ordered in Singapore and shipped over and more memory put in and then we got really expensive teradata boxes that could handle the volume but cost a fortune. The guys from other Duck talk about it's not a big data problem for many people. And so I think that's true. So I think we're a lot closer of the technology being able to support virtualization. I just don't think as data professionals we are. And this idea of having reasonable patterns that then have solution model patterns to support it, that Then have proven code gives us one step towards being safer being able to virtualize all the way through. Especially with things like zero copy cloning, where we can actually don't have to take a full copy of the data back to test things out. So maybe the first step is whenever you're doing any development, do it in a end to end virtualized pattern to test that it works and then physicalize when you have a cost or performance constraint.

That's the final chapter of the book. So the engine concept is where we start the book with and it explains all these sort of building blocks of what constructs a fully automated machine. And one of the components is optimizer and we say exactly that, right? So start virtual. But depending on which optimization directive you want to give it, like cost, storage, IO latency, whatever that is, it can calculate based on usage patterns what the best combination of physical virtual objects are to deliver that. Right. So if you don't want cost because you're running on Snowflake for example, or something like that, then you know, physicalize some parts because it will help you in optimizing for that directive. But if you want latency, maybe just don't do that, just physicalize the end result, keep everything virtual so it becomes a little playground for making things work the best way you can. It's really fun.

And then you're going to have to bring in the segmentation or domain based or business unit based or process based. So you say these data flows need to be low latency.

Exactly. How cool is that? Right? I want this mark to be refreshed every minute and this one every hour. Right. And this is literally always saying the book that becomes a requirement because as a data team you have limited resources. No matter how you look at, if you want to optimize that, it becomes more of a management decision and then you get more resources to deliver that requirement and you just update the directions and go.

So if I took that future state of virtualize everything and it actually worked, I could also see a future state where you basically pull up the design patterns, you tick some boxes for them, the constraints around which technologies it's allowed to use, and it actually writes the solution patterns and the code and deploys itself as a fully config driven, end to end platform from a couple. There'd be a lot of checkboxes though a lot of complexity.

In our definition, the design patterns would remain the same, but the differences solution patterns will be selected by the engine to deliver the updated physical implementation.

So are you saying that the Design patterns become mandatory, in your opinion?

Yeah, yeah, because that defines how your solution operates, right? The historization, how to do reference data, all those things we've discussed earlier. So you need to make a decision how you want to treat all those big blocks. But for each of design patterns, you've got a number of solution patterns for different technologies or concepts or anything. So you can switch between those and then let the solution refactor.

Okay, so let's explore that a little bit because again, it's just bringing this idea of agility, which is ability to change at low cost and with speed, not scrum. And so there's this balance about work done up front. So there's times where you do work up front where you don't need it right now because you know it's going to save you time in the future, but it is waste because you're doing it before you need it. And in fact, it's a really interesting conversation with Patrick Lager at the moment on LinkedIn where he's saying he actually doesn't do any work upfront anymore. It's if he doesn't need to build it, he doesn't build it. So I'm gonna have to have a whole chat with him now about doesn't that just become a whole bunch of ad hoc now? He's got enough experience to probably know how to retrofit the patterns after the fact. Okay, but let's knock off on that tangent. So let's go a couple examples. So I would say you always want to bring in a historization pattern into your platform on day one, especially if you can display, pull it out of the patent library because you're going to need it. I could then say if I was looking for feature flags for analytics for machine learning, so I wanted to be able to create very wide, very long table that was keyed by customer that had a whole lot of feature flags, their age bucket, their usage bucket, their income bucket that I was going to put into a machine learning model to do some clustering, segmentation. I would probably not do that work until I had an information product that needed that work to be done. That would be my standard approach right now. What I think you're saying is if there's a design pattern, and it is, and if the solution patterns and the implementation code was readily available and proven, that box will be ticked by default because at some stage I would need it and the cost of implementing it is almost nothing. Is that what you're saying or did I misunderstand that?

Partly. So. In this particular example, I would argue that every delivery of data uses the same pattern. So no matter how you deliver, like everything normalized or one big flat table, that's all the same pattern you use. You just select different columns. But conceptually, yes. So the way I look at it, you would have a couple of solution patterns readily available because we provide them with the book and the GitHub and all that stuff, but it might not be fitting your technology and all that kind of stuff. So you would then add them as you need them? I would think so. Good. Looks like having a library of patents that we can all collaborate on, that is somewhat centrally available and we can work together as a community to improve them. That's a bit of an ideal world dream. But in the smaller groups that would certainly be possible. But I think that if your solution pattern doesn't exist for the specific technology that you're working with, yeah, you, you would have to add it. And in that case I would only add it when I need to add it.

But that idea of a pattern library, we can all share, and that happens in the software domain. Our software Breverns have that already. They've done the work to make that happen and got the value out of it. In the data main, we don't.

Right.

We go and implement a new piece of technology or a new vendor. We go, oh, SQL Server, I'm going to do Snowflake now and we reinvent all the patterns again and we're going to go data bricks. Oh, let me reinvent all the patterns again from scratch because I'm changing the technology stack. And that's just crazy.

That's just so much to learn from our friends in software development.

Yeah, but I still see large platform builds. I still see projects where it's six months to a year before the platform's available to the team to actually build something of value. That has changed a little bit, but not a lot in my view.

Not a lot. So there's room for growth there. And I think having a library like that to use and contribute to, hopefully it will help get it move into that direction. We've talked about this before, right? And those plans, they, they're still there.

I got to say. At least you've been following this dream for a while, so I can see that. Actually, to write this book, there are at least three challenges, three big challenges that just come to mind. So one is building out this mental framework of design patterns and what they are, solution patterns and code that implement the solution and how you describe Them, how you link them, the kind of meta model for that model. That's the first problem. The second one is then taking your 25 years of experience. Dirk's been in the industry.

Yep.

So taking 25 years of experience from two people and then distilling that down into those patterns, like just picking them out of your brain and writing them down so that you get this library of all the patterns that you've ever implemented and that are valuable. And then the third one is figuring out how the hell you tell that complex story in a book.

100%. Yep. And it's taken us almost seven years. Right. So that's, that's something. And I do find, as you point out, the Data Engine Thinking is co written by Dirk and myself and we've, we both worked very, very hard on it. And the interesting thing is that there has never been a case where we didn't agree on something. Right. So it's like this convergent evolution kind of thing. You work towards something that, you know, works and people get there, and not just us, but other people as well. You slowly move towards this approach that makes sense in the present. So it's been very. Harmonious is probably the right word to use. It's never had any sort of disagreement on how things should be done. There were a couple of cases where we didn't know something yet. Right. So a couple of errors that we run into and say, hey, actually I never thought about that, or I wonder what would happen if that, that happens. And then sort of explore that and started coding and wrote a couple of examples. And then certain cases where timelines or time periods right from start date, end date, that kind of thing were missing in not in an expected pattern or something like that. Right. Or what is the consequences if the source system, operational system doesn't have that time validity? And so to really explain what happens then. Oh yeah, that's interesting. But never that we weren't on the same page. So I think that's pretty, pretty special.

And also think the fact that you're creating code and applying it against data scenarios to prove that both the solution pattern and the design pattern have legs. Right. Are real things that you can describe actually probably helped in terms of validating the book as you go. Here's the design pattern, here's what we think the solution pattern is. Let's create some code and if it's survives and does what the patent says it should do, then we know it's a real thing.

Absolutely.

Because there seems to be a few books that have come out over the last couple of years. I read them and I go, that's great theory, but where's the meat and potatoes?

Right.

How do I do that? Because that seems great. It seems like nirvana and we've been trying to get to that for the 35 years I've been in data, but we've never made it. So just tell me how to do it for once. And this book seems to be taking us to that state. Right.

Where there is an example, certainly the Internet.

And so into June, you reckon?

Yep. We got the Global Data Summit coming up in Iceland, so Derek will be there to present book and everything. So that's when the physical copy should be available and everything should be order online as well. So that's going to happen. Yeah. By the events.

Yeah. I was hoping to be at Iceland this year, but for a lot of personal reasons it's not going to happen. And then I was hoping to be at the Helsinki Data Week and then they moved it to when I wasn't going to be in the uk, so I missed that one as well. I'm happy that 2025 seems to be the year of the books. There's a hell of a lot of really good data books coming out. So I'm hoping 2026 will be the year of good conferences for me.

Yeah, see you there.

What interesting questions Joe Reese asked on his podcast just lately and I'm going to go steal it. Thanks, Joe. Is this idea of how did you write the book? What was the process you use to write this book? You just talk me through how the two of you wrote it together.

I'm not sure if we actually follow the process. So Dirk and I both have been giving classroom training for a long time. Then our focus areas are slightly different, but it's similar stuff. And the idea of having a storyline that supports the more quote unquote drier materials, the theoretical parts, we agreed on that really early. Back in the day we thought maybe we don't want to make it like a dataflow book or anything like that. Right. There are some data components, but we want to make it a bit more than that. So I started to write the main theoretical chapters and Dirk started to write the main storyline chapters and then we would peer review our work and then revising that sort of. Then we switch, swap roles a little bit, saying now I'm going to update that storyline with the theoretical explanation that I have in mind and then tweak a little bit and then therefore put the same and review and change some of the Explanations or add what he wants to say. So it's like that, right? So we both write a chapter storyline versus theory and then we switch roles and then we do it again.

So effectively you storyboarded the chapter and then wrote in isolation and then swapped to iterate and align. And then would be interesting to see which chapters you had close alignment after the first iteration and which chapters you you deviated from each other more than some of the other chapters would be interesting.

Yeah, yeah. I think especially on the biggest work we've done is on the definition of what the architecture would be. The different options that the big boxes that we've talked about earlier that took us the most time to really nail that. And the other one is the different ways of looking at temporality and data that also took a long time. Those are the real big things.

Yeah. Seems simple, right? It really is until you again until you engage with the first bit of data. And one of the things I found, and it's only after I'd done the retro of publishing my book that I realized how valuable it was. But actually creating a course for the content in the book before you write the book and then actually training people for me was something I hadn't realized how valuable it was because by creating the content you're actually drafting the book. You're just not writing the words, you're drafting images and you talk it and then you write the hands on that people do to reinforce it. And then as you're writing the book, you're flicking back between iterating the course and the book because you're starting to learn from way of articulating that seems better. Or you're getting feedback on the course that you need to bring in. And so that gives you a safe kind of cool testing ground to actually learn and do at the same time. Is that what you found?

A hundred percent. Yep. Yeah. So at some point I started recording the trainings for that purpose. And then after the training sessions I was updating the notes in the slide decks to find the right words and that all made it into the book. And at the end I just opened training again and started moving slides from the deck if I was happy enough that it was sufficiently covered in the book until the end resonated.

Oh. And thought about doing that actually. Yeah. Picking up the course material and deleting it when you know the books covered it to prove you've got the coverage on it, aren't slipping stuff in. Yeah. What I actually found was I ended up writing more stuff in the book. That I haven't put back in the course yet because then I've got to extend the course, make it longer and I'm like, I'm not sure I want to make it any longer.

Exactly.

Yeah. So it's that trade off decision. Yeah. All right, if people want to get hold of you, what's the best way for them to find you?

Yeah, find me on rulandvor.com that's my website with the blog that I have ignored slightly because I was so busy writing the book, but that's where you can find me. And also dataenginethinking.com which has the book website and it has some examples and it has some preview chapters and some of the Personas. So definitely find us there.

Excellent. And join. It'll be out on Amazon, no doubt.

For sure. Yeah. And we're also exploring some other platforms, including our own website. But yeah, definitely Amazon to start with.

Excellent. Right. I look forward to buying that one and finally getting a design patent template that I can use repeatably and stop hating the ones that I use currently. Thank you for that.

At least.

That's going to be the best value that I've had for a long time.

I generally hope so.

Excellent. Hey look, thanks for coming on the podcast and I hope everybody has a magical day.

And that Data Magicians was another Agile Data podcast episode. If you want to learn new patterns that you can apply to craft your own Agile Data way of working, head over to agiledata IO.

Sam.