AgileData #27 - 3 agile architecture things - Brian McMillian

Welcome to the Agile Data Podcast, where we talk about the merging of agile and data ways of working in a simply magical way. Welcome to the Agile Data Podcast. I'm Shane Gibson.

And I'm Brian McMillan.

Hey, Brian, good to see you today, and thank you for coming on the show. I think in this one, we're going to talk about 3 things. We're kind of going to go on a really interesting journey. So we're going to talk about "The Thing Called 4x," Data Vault and why it's the best data modeling process in the world for now. That's my view, anyway. And "Everything is Code." But before we rip into it, why don't you give us a bit of a background for people who don't know who you are or your journey into this agile and data world?

Thanks, Shane. My name is Brian McMillan, and I am a career enterprise architect, been I've been practicing enterprise architecture for about 15 years or so. My area of expertise is really in the data and business intelligence space. And I've worked primarily in large companies, everything from manufacturing to IT providers to the defense industry.

Excellent. And enterprise architects in agile and data is often seen as an oxymoron, right? We're seen as anti-pattern because, because, you know, a lot of enterprise architects tend to treat themselves as the cop at the end of the road, right? And all the work's done and they come in and go, "Ooh, bad work, bad work, go back to the beginning." So it's really great to actually talk to somebody that's in the enterprise architecture space from large organizations, but has more of an agile mindset. So what's your experience with that? Do you find that a lot of the people with the the role or the title enterprise architect is actually bound on a more waterfall-type process?

Well, I think so. Enterprise architecture has been around for a long time. And, you know, it has been around when, you know, as long as Agile has been around, enterprise architecture has been around. But, Most of the enterprise architecture, in particular, is geared towards large enterprise projects and products that are pretty well established or so large in scope that we think we've got so many moving parts, we need to coordinate them. And that lends itself well to trying to plan out as much as you can, even though we know that that seldom works. My experience with agile and architecture comes from way back, oh geez, probably late 2000s, 2008 or so, where I was on a team that was building a data warehouse to store server performance metrics, and ticketing metrics, and call center metrics for electronic data systems. And we had just started to do ticketing work, we had a very large enterprise customer that we were onboarding, and their requirements were changing so fast, we basically just had to say, "Wait, whoa, whoa, stop. We can't work the way you're expecting us to work, and we're never gonna be able to meet your needs." And that's where I got introduced to extreme programming, XP, Agile, and that just lit a fire. You know what? We can do this in an agile way. We can treat the architecture incrementally. And I think that's the big— If your architects that you're dealing with are not treating their work incrementally, then you have a problem. You can't start from scratch and start planning everything. You have to do a little bit of planning, do a little bit of implementation, a little bit of planning, a little bit of implementation, wind things back that don't work. So, that is really important. Now, a little bit about my background. I didn't come at the architecture job from the normal route, which is, you know, I'm a computer scientist, I'm a programmer, I come from the application space, you know, I'm a systems administrator, I'm a database person, you know, I'm a DBA. That's where most architects come from. I came from the business side. I've got a degree in economics. On that warehouse team, I was the business guy. I did kind of everything from sales support to working with the customer, to dealing with the requirements intake, to doing demos for people. So I came at it from that side. This idea of product lifecycle is critically important to figuring out how you should be behaving. So I'll make a controversial statement here. There's absolutely nothing wrong with waterfall. There's absolutely nothing wrong with agile. There's absolutely nothing wrong with, you know, somewhere in between. It's wrong if you're applying it at the wrong stage. And that's where this idea of 4X comes. And 4X is a little bit of a misnomer. Kent Beck, who was one of the original signers of the Manifesto for Agile Software Development, you know, wrote the original test, unit test framework, has been very active in the IT community forever. He worked at Facebook a while ago, and one of his jobs was to evangelize the way that Facebook does their work. To hear him talk— and we can put something in the notes here, some links to this— but to hear him talk about it, there was this idea that he came into Facebook and they didn't operate anywhere like he expected them to be operating. Set out some time to try to learn how they were operating. And what they were doing was they were operating under this idea of a product lifecycle, which has been around forever. Product lifecycle is nothing new. But he coined this idea called 3X. So product lifecycle falls in 3 stages. And it's a sigmoid curve, an S-curve. Down at the bottom, low value. Basically everything is unknown, and this is what he calls the explore phase. You're exploring, you're trying to figure out what's going to work, you're trying to figure out who your customer is, you're trying to figure out what they need. That's that explore phase, and that's where all these projects start. And then once you find a customer and you start to get some traction, you start to go into expand, and that, that's where the curve starts to, you know, ramp up exponentially. If you're lucky. Well, if you're lucky or unlucky, depends on, depends on how you feel that day. Um, in that expand phase, you're getting more customers, you're providing more value, and things are breaking all the time, so it's constant firefighting mode. And then eventually that will take— you'll get a, you'll get a handle on your product, and that will taper off, and it will start to level off. And that's what he calls the extract phase, where you're extracting the value out of the system you have. You've reached a comfortable level of customers, comfortable level of value that you're delivering to your customers. And then the 4X piece is one that I think is critically important, which is you need to exit from there. You need to think about how you shut things down. And that isn't part of the original 3X model, It's an addition, but it's critically important because everything, every product you build is going to have to be turned off at some point. You know, think about, I was thinking about what's a good example of that. I think the best one is a car manufacturer. You know, they have 2 to 3 year cycles where they completely retool the car. If you bought a Honda Accord 2 years ago, or let's say 4 years ago, and you were looking to buy a new one now, it's a very different car. It has a lot of new features. It is not the same car. Lots of hard product companies, big capital-intensive product companies are really good at this. The IT world, particularly in the enterprise world, we're not very good at this for some reason. We have things that run forever. They get ramped up, they get turned on, they work, they're stable, and we are terrified to turn them off. In the IT world, we need to start thinking more clearly about that.

Yeah, I like that. So, the exit was the one that tweaked my interest because, you know, we've all worked for organizations that have replaced or refreshed their data platform every 5 to 10 years because, you know, technology change, right? They're on old, you know, on-prem servers or big expensive hardware and they're moving to the cloud, or before that, you know, they did the client server wave. But often you'll see an organization that only ever moves 80% of their workload off the old platform, and the last 20%, which is incredibly hard, it still lives there, right? And so now they're maintaining two platforms and getting, you know, benefits from the new one, but they just don't want to spend the money to kill the old one. Or, you know, the number of times we don't monitor the usage of the information products we produce. So the dashboards, the reports, the APIs, you know, we just, we build it, and we leave it, and we let it go stagnant, we never monitor if it's being used, and we never kill it, right? We never exit those products. And yes, they're probably low effort, low cost to maintain, but there's still a cost, there's still an effort, right? The mess that they create, the dependencies. So we need to do that tidy up, right? We need to think about that exit. So the other reason, apart from this idea of, well, how do we plan to exit the things we're building at the right time? The other thing that I find really interesting is if I think about applying the 4 Xs with teams that are kind of adopting a new agile way of working, data and analytics teams, it's almost some kind of similar to the forming, norming, storming kind of idea, right? So it's—

Yeah, absolutely.

Exploring ways of working, we're seeing what works for us and what doesn't, we then get Good at it and we want to scale, either scale the team or scale the processes or scale what we're delivering. Then we get to the stage where we want to automate it, you know, the whole DevOps, DataOps, just to extract the value and get more value than the effort we're putting in. And then hopefully we don't exit, I was thinking, as we get rid of the team, you know, the magical software takes over. I wasn't quite sure how to apply the last X to a team politely.

I think, you know, my recommendation is, so when you start getting into expand, and you start ramping things up, somewhere in the middle— so let's wind it back. So think about processes and ways the team works. The people all work differently in each of these different phases. You have people who are, let's say, more comfortable in one phase versus another. So let's just think about like project management, you know, trying to figure out what should we be working on. So initially in Explore, you're in what I call pre-agile. You're, you know, business model canvas, lean canvas type things. You're just trying to find a customer who has something that you're interested in and then work with them. Then at some point in later Explore, You can start to apply more agile ways of working. You know, you can start to do a teeny bit of, you know, a teeny bit of planning of whether, you know, it's a sprint or however you do that. Or if you're lucky and you are just moving straight into just straight flow, that's great. And that will carry you up into expand. Somewhere in expand, you're going to start to get a handle on things, and you'll be able to prioritize what you need to work on. You're looking for the constraint, you're fixing constraints. In the middle of that. And you can start to switch over to things like critical chain project management, where you are planning out the constraint, working around the constraint, and planning the work you have to do around that constraint. Everything else is much looser because you know what's expected, and you know how important it is that you deliver this feature at this particular time. And whether you're on track or not, it's not so much did you hit the date, it's are you on track? At that middle point, in the middle of expand, is when you should start thinking about, okay, we've got a handle on this, what would the next version of this look like? And you need to peel off some of the team and start them back in explore. And they start thinking about features and functionality that would be in the next platform. And they start doing their work. So you've got one S-curve that comes up, and then you have another S-curve that's starting in the middle and will eventually replace. Now, ideally, what would happen is when the top of the extract hits the expand on the next big thing, you can start to transition your customers to the next big thing. And then that exit now becomes a matter of winding those customers down, winding that infrastructure down. And it's the same process as in explore— I'm sorry, in expand, but in reverse. You're figuring out where your constraints are, and you're working the things that are not the constraints and turning those off. So you wind things down that way. But it's the same team, the work is just starting to change, and you need to start planning for that. And that's how you grow one version after another version after another version. So that it's not a problem.

So, the other way I kind of just, as you're talking, thought about it, and taking it back to that enterprise architecture, right, and this idea of agile architecture. And so, one of the podcasts I follow that I love is Catalog and Cocktails from data.world. And one of the guests, or one of the listeners on there, kind of came up with a term a while ago that they use quite often, I really love, and they talk about brakes on a car aren't there to stop you moving, right, or slow you down. Brakes are on the car, actually there to make you safe when you go faster. And so from an agile architecture point of view, you know, it's the idea that the architecture skillset is not there to put barriers in place, right, and to stop and put gates, right? They're there to put guardrails, right? They actually have to be in front of the work and go, okay, if we're going to go into that area and that's new, right, we're exploring. Or we've done some work and we're expanding and then going into extracting. What are the policies and processes we need to be in place to be safe, right? And so, from that point of view, if we think about it as an architect, our goal should be to make ourselves redundant, to not needed by the team for that piece of work, right? That we've kind of gone, worked with them, we've gone, the guardrails are in, you understand the things you need to do, the brakes you need to build to make yourself safe to go faster and go for it, right? And then, as you said, it'll be another iteration where there's a whole lot of new unknowns. And then, you know, the architect comes in to help the team or the skills in the team are there that they can figure out what the new guardrails are, right? So, for me, that's how we should be thinking as architects when we're working with teams that are, you know, data and analytics-focused and moving into that agile way of working.

Yeah, and each of those phases, again, here's where these phases come into play. Each of those phases has different sets of guardrails. Guardrails with different strengths. Like, um, you know, I think another analogy, if you don't like the guardrail one, that I, that I happen to like is channels and buoys. So, you know, you're, you're on a boat, you know, a big huge boat needs to stay in the channel. If you're in a kayak, you can go wherever you want. You know, what, what's going to be able to carry the most people? Well, clearly the bigger boat. And as that value increases, one of the ways you manage the risk, so you start to bring those guardrails in, those buoys in, stay within the buoys, and dig the channel a little bit deeper so that you have the capabilities you need. And I think that one of the things that's really important for architects to keep in mind is you need to be loose and explore, you need to be very strategic in expand, You know, there will be, you know, you may have some set of guardrails that you want to, you know, say a set of technologies you want to work within, you know, a set of technologies are going to work. You need to be very critical about where the constraints are that are popping up and be ready to shift if needed. You know, I, I've, uh, one of the, one of my sayings, I'm, I'm I'm rather fond of is all software sucks. Your job is to figure out where it's going to fail first and work around it. You make a decision. It doesn't matter what product you pick or what strategy you pick, or what pattern you're following. At some point, that pattern is going to break down for you, and you need to be ready to recognize it, and decide what to do about it. But as you move up that curve, though, it's a little bit— you need to be much more strict. Data governance is a huge deal when you've got a product that's in the extract phase. Everybody's counting on you. It better— the data better be there, it better have a good, solid interface, you should be paying attention to it. Observability is critically important. Important. In the middle, yeah, you may have to loosen it up a little bit because things are changing so quickly.

There's a term out there that people kind of adopt around professional laziness, but, you know, for me, I tend to like positive words, so I call it the happy path. And it's this idea that if there are things a team can leverage that are fit for purpose, do the job they need it to do, and are readily available, they're going to adopt it, right? It's a happy path. It's like, well, that works. I don't need to worry about it. I can go solve some of the problems that haven't been solved. So, again, from an agile architecture point of view, what we want to do is help them build these things that, as you said, they're buoys that keep them in their lane, you know, they're guardrails, and the teams will adopt it because it's easier for them, right? It makes sense. They're not going to go and reinvent it. We'll rephrase that. Most people won't go and reinvent it, you know, I have worked with developers whose first job is, you know, they typically say, it doesn't matter what the code looks like, I'm going to rewrite 70% of it because that's the mentality. But, you know, we just vote them off the island. So, you know, this idea that if things are in place and they're reusable, people will reuse them because it just makes sense. And everybody is a good person, right? And they will do what's seen as right for the organization and their teams. Kind of moving on from that one, because I was tempted to deep dive into data governance, but that's That's a whole podcast on its own, which I'm happy to have because for me, data governance is definitely not— it's before the explore stage, right? We're still at decisions by committees and I hate that. But before we deep dive, let's not deep dive into data governance, let's move on to Data Vault, something I love, right? So for me, there's been a wave of modeling techniques that have come out over the years. For me, Data Vault is currently the best technique I can find for managing data for a whole raft of reasons. It has some problems and, you know, hopefully somebody will come out with some new modeling techniques that give us the flexibility of Data Vault and solve some of the problems that it presents. But from your point of view, you know, Data Vault, why, why not? What's your view?

Data Vault modeling technique, I just love. I think it solves a lot of really difficult problems. And the first one is that it gives you a framework to think about the data problem you have, like the lifecycle of your data, from acquiring the data to actually making it, you know, fully functional and usable to whoever the end user is. So, it provides a bunch of natural layers to cut the problem, the, um, to cut the cognitive load for a developer down to bite-sized pieces. And it does this a couple of ways. You know, the, the first one is, and the thing that if you're, if you're familiar with Data Vault, probably the thing that you think of first are, you know, the, the hubs, links, and satellites, you know, the 3 basic table types. The hubs represent your entities. Those are your, you know, customers, products, you know, sites, whatever, whatever the entities that are important to your business. They get expressed in a hub, and it's every customer gets a record in the hub, and you can look at the hub table and you can, you know, count star and see how many customers you have. Which is really useful metadata about the system. If you come into a Data Vault, the first thing you do is you look at all the hub tables. Okay, now I know the domains that comprise this warehouse. Then you have the satellite— going on the other end, you have the satellite tables, which are the details. And that varies. Each domain is gonna be different, and it's not very prescriptive at all. Hub table structure is very prescriptive. Satellite table is not, except for how you link and a couple of other neat little fields that they require you have. And then the link table, it's just a place for you to express the relationships. So one that happens all the time, those customers, I've got two customer records, but they're the same customer. Well, I can express that same-as relationship in a link table. So that when I go to query the information, I can say, oh, I want the standardized customer name from the link table to join up with the data, the detailed attributes that are in my satellite table. And that's the basic structure. And one of the great things for people who are developing in the Data Vault model or consuming the Data Vault model is they only have to learn those three. If you can learn what the capabilities are and the purposes are of those 3 tables, you're good to go. It's very easy for you to maneuver around the warehouse then. You may have some extended joins to do, which is a problem with the Data Vault. You know, it's not like a fully, you know, third normal form system where you may have lots of data. Lots of tables that you have to join across, you basically have 3 basic tables. But you get the advantage of combining those tables up like you, you know, had a more denormalized system. So you can bridge different satellite tables across the link table or the hub tables. But there are lots of joins, which is a valid complaint about the Data Vault is that I've got lots of joins. But if you're doing anything complicated, you're probably gonna have lots of joins anyways. And the traditional star schema helps to eliminate that. And if you've only interfaced with a star schema warehouse, Data Vault looks very intimidating and complicated because there are lots of tables. But if you can't find the data you want in your fact table and your dimension tables in a traditional star schema warehouse, you have to go looking elsewhere, and it may involve rebuilding, you know, new fact tables or rebuilding the fact table that you currently have. And those kind of changes are really disruptive to the warehouse. You know, modifying and migrating a fact table is a big undertaking. And if you've got it broken up and a little bit more denormalized, it's much easier to make these changes. So, you've got these tables where you've got these 3 table types, very easy to learn, very easy to get your head wrapped around. And they're very flexible, doesn't really matter what kind of data you have. So, that's one set of layers or crosscuts. The next one that I think is even more valuable is that you have the Data Vault is split into these layers. Now, whether they are schemas, whether they're schemas or whether they are individual, I mean, frankly, they could be different databases. They could be different database engines, whatever is appropriate. Your raw and staging area could be in a file-based system, whether it's a whether it's a file share or something like HDFS, Data Vault doesn't care. Data Vault just says you need to have a staging area and then you need to have an area called your raw vault where you're storing history of all those loads. Data comes in, gets loaded in the table in this append-only format so that you have a new record, You have a brand new record, it gets entered into the table. That record gets updated, it is a new record. You don't update the current one you have. And that is really powerful, particularly when you're exploring your datasets. Because you don't really know— this gets everybody, you know, everybody who spent any time working with data knows that there are things you just don't know until too late usually. And being able to have a raw satellite table where you're stacking up your loads incrementally and you're just appending the change datasets lets you go back and it lets you unwind the data. So what, what did this data look like last month? Which is really powerful from a reporting standpoint. I can tell you what the dataset looked like and it doesn't look like that anymore. And here are the 3 changes that happened over the last month. For troubleshooting, it's also valuable because you can see what changes have occurred. But that append-only model is also very easy to work with. The type of SQL that you have to write to load that table is really simple. You don't have to go, you don't have to go look up keys, which is something else in Data Vault that's really that they made the difference between Data Vault 1 and Data Vault 2. One of the big changes was They had to support NoSQL databases, which means they went from sequence keys to hash keys. So, figure out what your key is, hash that value. Figure out what your business key is, hash that value. And then that means that you can now load everything in parallel. You can load your hub and your SAT in parallel. And because you're calculating on load what the key is, They can just go in seamlessly, which speeds up your loads significantly, and it makes loading the database much easier. And it makes it easier for people who are not that comfortable doing, you know, the DBA ETL work, makes it much easier for them to get their head wrapped around what they have to do. Oh, I need to load the table, I need to follow these simple patterns, and I can load my own table. Which brings us to the next layer. After you've got that raw data, that raw table, or that raw schema, you've got a business schema, which is a new thing that was introduced in Data Vault 2.0. And that business schema is where you apply the business logic. In the raw schema, your tables, whether they're hubs, links, or SATs, you don't apply any business logic. It's the data you got from the source as is. Unmodified. But that's never really what you want. So make those transformations from raw into the business schema and apply whatever business logic you need. And this is where it gets— this is where the tables kind of balloon, but it's worth it. You can have a hub. Let's say you're a manufacturer and you have serialized and unserialized parts. Every analyst who's querying that data has to, you know, make a common table expression or put a view together or something to filter out all of the parts that are serialized for a particular, you know, let's say they're doing some quality analysis, and they're only interested in the serialized parts because they can tell it's a much more fine-grained view of the data. And it would be much easier if there was just a table called hub_serialized_parts or hub_parts_serialized. 001, first version of that table. hub_serialized_002, that's a different version. There's something changed in the business logic. It's, it's like, think of it like an API where you've got Version 1 of the API, and version 2 of the API, and version 3 of the API. Yep, now you have multiple tables, but they provide different business logic, and it cuts down, it reduces the risk of you making a change that someone didn't anticipate. If you're consuming the table that you're consuming, you know, the first version of the table, and there's a second version, your things don't break. You get to make the choice. And this is something that we, as data architects, we don't do a very good job of. We're really good at planning our migrations, but we don't think about what would the simplest way be. Just duplicate the data. If storage is really cheap, duplicate the data, version 1 of the API, version 2 of the API. Oh, maybe we should treat all of our tables as APIs or as individual data products and start thinking about the lifecycle of individual tables. That gets really interesting. Then those business tables, you got lots of business tables, right? But they all have different business logic applied to them, and it makes it much easier for an analyst to come in and go, "Hmm, well, I want this flavor of the data and this flavor of the data, and I know that the business logic that's applied in those tables is very clear, understandable, and the teams that are managing those tables stand behind them. And I don't have to go make some custom code with some weird query that may be above my skill level. All I have to do is just join the tables together and things get filtered out.

I think, yeah, you're right there in that as technologists, we love complexity, right? We love to make things of beauty, right? We love to take a problem and just beat the snot out of it with the most beautiful solution we've ever seen. Sometimes a simple solution would do it. And I think that's the same with Data Vault. For me, Data Vault is a set of patterns, right? So it's a set of patterns that have value because they are the best or the, you know, they're a set of good patterns that we can use. The Data Vault community has fallen into the same problem that I've seen with the Scrum community, right? Is that, you know, with Scrum, There are some people that say, here's the Scrum Guide, and if you don't follow the Scrum Guide to the letter of the Scrum Guide, then you're not doing Scrum. And I don't agree with that. Well, I agree you're not doing Scrum, but I don't care, right? I'm saying there's a bunch of patterns embedded within the Scrum Guide. They all have massive value. Use them if it fits you. If it doesn't, don't use them. And same with Vault, right? So, you know, there are a bunch of immutable patterns in Vault. They are, you have a hub, you have a set, you have a link. That's about it, right? Everything else really is a choice as far as I'm concerned. And the thing I like about Vault, you know, and you kind of articulated it, there are hub sets and links. There are only 3 types of data. There are really only 6 patterns, create hub, load hub, create set, load set, create link, load link, right? It's a really simple coding pattern. Now there's lots of complexity once you start throwing it at real data. You know, we talk about bridge tables and same-as and our uncle links and, you know, there's lots There's lots of other things that we sometimes do to make it easier when we have complex data, but they're just choices. And so what happens is we get into these religious arguments. So in our product, Agile Data, under the covers, we use Vault-ish, right? We don't use the term hubsets and links in the product because what we found is our audience is an analyst, right? It's a data-literate person that's on the business side, not the engineering or the technical side. And so, we found that those words scared them. So, we call them concepts, which is a hub, right? I have a concept of a customer and their business concepts. They can go out and say, I can see a customer, I can see a product, I can see a store. We have detail about those concepts. It's a set. The customer has a name, the customer has an age, the customer has a gender, the store has a name, the product has a SKU, And then we talk about events rather than links. And we do that because we say, if you looked at your organization, what business events or core business processes can you see? Oh, I can see customer ordering a product. Cool. Well, that's a link, right? You know, these three concepts or three hubs that are embedded. So for me, that's really valuable, right? Those patterns are just so simple and they're so flexible. We deviate a little bit from the DV 2.0 mantra. So we actually use a history layer that isn't vault, right? That's not vaulted. Typically, you call it a persistent staging area. And so we don't vault that data, right? We don't break it out into hubs and sets. And there's a whole lot of reasons we don't apply that pattern, but we're okay with that, right? That's just a choice we made. It has cost and consequences. And the same thing is we spent a large amount of time on that last layer, right? What we call the consume layer, or people typically call presentation. And so we said, okay, so we don't want our analysts to have to understand how to join all these tables together. So what we can do is we can create views or tables that make them consumable, right? We can denormalize a link out to its hubs, yeah, and denormalize that out again to include all its sets and give them one big, wide, ugly table. Now, performance is an issue sometimes, but with the cloud databases we have now, they pretty much eat anything up to a terabyte without even blinking when you denormalize it. It's just a thing of beauty. So, for me, Data Vault is a pattern, right? There are a small number of immutable patterns that if you're not doing that, then you aren't doing Data Vault, the modeling technique. And then there's a whole lot of ways of working patterns that have come out in 2.0. From my point of view, if they have value, adopt them. If they don't, find something else that works for you. So, you know, the thing that people often say is, I prefer star schemas over Data Vault because it's easier for the users to query. And I'm like, great, do a Data Vault and then put a star schema on top of it.

Yeah, and that's what the standard says. You've got your business vault and then you have your info vault. And in the info vault, You'll have big wide tables if that's appropriate. You'll have traditional, you know, dimensional star schema data marts built, whatever's appropriate. And that's really important to know. And I think that that's in the layering in the Data Vault standard. That's one of the things that's really important to break out. You've got, you know, your raw persistent staging area. You have— which question for you on the getting to your, um, history layer, are you doing an append-only model there?

Yeah, yeah, so it's upset only. So we mirror the source system tables, that's the structure of our history layer. We do upsets only, right? So we are effectively inserting new change records as we see them. And we're just not vaulting it, right? We're not breaking that table structure out into a hub and a sat, like, like the doctrine says, right? But we, we, we're not doing that for a reason.

I will agree with you. One, um, I'll tell you, I've done the same thing. And in fact, in the, in the book, in the raw layer, that first table that gets built, staged and built, that's first SAT table, there are no hubs and link tables built because frankly, I don't see the need to. In most cases, what you wanna store is you wanna store the history to go back to. Now, it probably makes sense to break out hubs and links in the business layer because you're trying to apply some business value, and there may be some value in saying, "Oh, hey, I have to alias this entity so that we get the right product names." And it is optional. I see almost no use, particularly in linked tables, in the raw layer. I don't see a use for that.

So, I spent an inordinate amount of my time a while ago trying to understand raw versus business. And I don't see myself as a dummy, right? I'm not highly technical, but I can read other people's code, right? But I think I have an innate skill to see patterns, right? And When I see a pattern, what I focus on is classifying what makes the pattern and what makes an anti-pattern for it, when that pattern should be used, when it shouldn't be used. And raw versus business killed me, right? It was like, well, is raw its own database and business its own database, and now we're duplicating data? And if it is, I don't care, that's cool. If you store it twice and it has value, I don't mind. But that's not true, because actually what the answer is, well, Business is just another set of hubsets and links next to raw, right? It has business logic. So I'm like, okay, so what's business logic? If I'm conforming customer from two different source systems, yeah, that's a business thing, right? That's a high level of complexity in those rules where we're making stuff up, we're inferring things that aren't seen in the data. So that's a business rule. But if I'm flagging a part, you know, to say in stock, out of stock. Is that a raw thing? Is it a business thing? Well, I'm inferring it, so it's a business thing. And I got to the stage where I went, I don't actually care, right? All I care when I'm working with a team is determine what code you're writing or using and what it's doing, and then determine how you identify the pattern of that code, and then how you want to classify it. And if you want to classify that these type of changes are raw changes and these type of changes are business changes, good. And as long as you stick to that pattern, you know, so for example, if you say that conforming customer is a business pattern, you know, it's business code, then I should never see conforming of an entity or an identity or a concept in my raw layer, right? Because now you've broken the pattern. Now you're confusing me, right? If you say, you know, inferred flags, right, they're a business function, Right? They're part of the business rule. So they should only be in the business layer, right? I should never see them in my raw layer, because now you're confusing me again. So again, it's set the pattern, write it down, agree it with everybody, and follow the bloody things. And so that's where I got to. So I've thrown away the raw versus business definition for Vault. For me, it confuses people. It has little value. We should focus on the type of changing we're making, the size of that change, and then how we describe that as a pattern. And then when we see it as a pattern, in terms of the Xs, how do we move that from an explore pattern to an extract pattern? How do we automate that pattern? Because that's what the beauty of Vault is, right? As I said, 6 pieces of code, create or load, right? Times 3, the ability to automate that is amazing. That's where the value in that, in the Data Vault pattern is.

Yeah, and also to know you don't have to apply, like you've been saying, I think one of the things that you've been saying is you don't have to always use all the objects. If you've got the consume layer and that needs a different structure, if a hub, link, and sat layer probably isn't useful at the consume layer, it's too complicated. So you've already made the decision. So I think, you know, Pat, talking about the 4 Xs again, if you're an explorer, Stage the data, start collecting history, and then start building out your info layer, that consume layer. Go straight, don't do anything in the middle, just do whatever it takes to get the data visible to your customer so that you can start making judgments about the quality of the data you have and actually the— what data you really have. Then you may fall back and do some intermediate modeling, which may have hubs, links, and SATs. Then you go refactor the info layer, that consume layer, you know, and you have to have that flexibility. And that's part of getting that agility in your data architecture is, no, I've got these structures, these buoys that I can steer around, these guardrails I have to sit in, and What's appropriate for me to do right now? I don't have to use all the things. I don't have to do them in this order. And once you break free of that, then it becomes— and this is the thing that is really spectacularly productive about Data Vault is, if you break free from the sequencing of the objects, you get a lot of flexibility. And a lot of agility. You can change your mind.

Yeah, its ability to absorb change is amazing. And so that's why I go back to the Data Vault modeling pattern, right? The pattern of how to model using Data Vault and store that data and load and change that data. It's amazing, right? It is a thing of beauty. Everything else that's been put around it with DV2, I think it degrades the value of the model. So I'll give you an example. DV2 says big modeling upfront, right? It says if you don't understand the core business concepts and you don't model the organization, you get a source-specific vault, and that's bad, right? And I call bullshit on that, right?

I must have missed that chapter.

It's in the conversations, right?

No, no, it absolutely is, but in my experience, my first experience building a data vault from scratch, with a team of people who are not database developers, went through this pattern. Get the best data, get the data the company's using, start putting that in the database, and start looking at it. We found out very quickly that it wasn't going to work, that the data was bad. The counts of parts that everybody in the company was using were wrong. Because the system that they were getting the data from, they had implemented some incorrect logic and they were miscounting things. And, you know, that's, that's just the way that it is. So we scrapped everything, then we got a new set of data that we thought was much better, and we found new problems. We scrapped probably half of that. And then we got the third set of data that solved the problems that we had, and we went forward with that. And that kind of iterative development is really conducive to— I mean, forget the pattern, the modeling pattern that you're using. You have to start thinking about it incrementally. You can't get attached to— if you do a whole lot of upfront modeling, a whole front of upfront design, you are going to be sorely disappointed when it doesn't work. And the goal is get to that point as quickly as you can. You know, make it through the crappy chasm of doom where you make that inflection point from explore into expand. That's really painful, and you learn lots of things in that area, and it never is going to get better, but it will get better. You know, you might need to give it a few months. You need to be prepared to just jettison the work that you had done. And no amount of planning will prepare you for that.

And I agree, right? So, you know, you make the point earlier that, you know, as architects, when we design the beautiful architecture diagram, it's our baby, you know, and we don't want that baby to get hurt, right? So, it's like, "Oh, don't you deviate from my diagram." But we've got to get rid of that, right? It's like, it's a guess. You know, as soon as we put it into the first engagement, it's going to change. Same with our models, right? Oh yeah. Sitting for 6 months in a room doing the perfect model is not an agile mindset, right? It's not an agile data mindset. Our customers, we want to get the data in front of our customers early to get value, right? So if we can go and say we have a concept of a customer, right? So populate that hub, you know, from a source system, you know, Salesforce, that's where the first master of customer comes from. Cool, grab that, you know, get a hub for Salesforce with a customer ID in the set, add the customer name, whack that on a report and say to them, here we go, you can now count customers. As at today, you had this many, as at last week, you had that many. That is value, right? We underestimate how much value that really simple piece of information is to have a single answer to the same question of how many customers do we have? That's massive value. And then we can incrementally build out. Now, should we stay with a source hub? No, right? We should then bring in another system that has customer. We should conform those to say, well, we've got two different definitions. How do we get a single answer to the question of how many customers do we have across the entire organization, right? And that's a harder problem, but we build that next, right? We deliver that value first and then expand on the value. So, you know, for me, Vault is cool. right? It's a cool modeling pattern. It's simple. It enables us to change, which is what it's all about. But it's just a set of patterns we can adopt the way we want, and very few of those patterns in Vault are immutable. Hub sets and links are, though. My co-founder, you know, he comes from a more dimensional background, and he's particularly agile from his patterns. And he keeps saying to me, so why can't we just have sets and no hubs? And I'm like, here's the 5 reasons we have to have a hub, right? And And it's like, no, there is a set of immutable patterns, right? We will never have sets and no hubs for a whole raft of reasons. But running out of time, so let's go on to the last one, everything is code.

This is a good transition. Okay, so one of the problems— so, yeah, let's do it this way. So, we've been talking about doing that upfront planning and breaking free of that upfront planning process. I will contend that a lot of that upfront planning that people do is forced upon them by the tools they are using. If you came from a traditional data warehouse, enterprise data warehousing background, you were using, you know, let's say your Oracle was your data ware— Oracle was your data warehouse platform, Informatica was your ETL tool. And what does Informatica have going for it? It's got a nice drag-and-drop interface where you can grab an object, configure some parameters in a nice GUI, and then connect that to some other thing, and then connect that to some other thing, and build out your pipeline visually. What about your data model? Well, how did you— before you do the ETL work, you need to have a data model in place, right? Because Oracle's not going to work. It needs something to connect to, it needs to get some metadata from your database in order to be to be able to work the way that it needs to work. So that means you need to have a data modeler. That data modeler is going to use what? They're going to do the ER diagrams in what tool?

EA SPACs.

Yeah, that works because we're picking the big ugly, um, hard to use, right?

So let's say the EA SPACs.

So your data modeler is going to go in and they're are going to model the data and they're going to work with it. You know, how are they going to know what to do? They're going to work with the business analyst. So, you start to build out this waterfall pattern. And a lot of that is reinforced by the tools at best. You know, you have to model the database before you can use the ETL tool. Well, that puts you in a waterfall pattern that makes it difficult for you to make changes. And if you were to take a different— so we're gonna talk about treating everything as code. So what would that model look like if you were going to treat those steps as code? So we're going to model the database. We need to get some data in the database. We're gonna model that. Well, if we're gonna, you know, in your example, if we're going to copy the source tables over, from the source system into our staging area, and we're going to do an append-only type load, insert-only load. Like you said, the, the SQL code to do that is pretty easy. Your bulk loader for your database may create the table for you automatically. You know, you may just point it at the dataset and say, bulk load this data, and it may create the tables for you. Shoot, it may even create the database for you depending on the tool you have. So that becomes much easier. Now, that process didn't require you to do any modeling because the data reflects the— the table reflects the data that you loaded. Now you have to do that load script. If it's a bulk insert, it's just going to naturally just append the data. And then you may have to do some cleaning up afterwards. Or now you wanna move it to the next table. Well, depending on the type of movement, type of pattern you're gonna have, you know, load your SAT, load your hub, whatever it is, that's another piece of SQL code and it's pretty clear. And so now I need to write the SQL to create a table, I need to write the SQL to load the table. I'm the ETL developer, it's pretty easy code to write, Even if I'm not very experienced and comfortable with it, I can look at the last table that somebody else, that the person who was more skilled than me did, and I can copy that pattern. It's pretty straightforward. I mean, that's a revelation to someone who's only queried the databases, someone who's got good SQL querying skills. It is not a heavy lift. To learn how to create a table and how to insert data into a table. It's really easy to do. And once you know how to do that, then that gives you a lot of power to do things like create your own tables. And it's very clear what's going on, and it doesn't require a third-party tool. It's code. Now, the advantage to having that— now you're code-based. Your ETL is in code, at least the tasks you're executing are in code. And creating the database tables, the DDL statements are all in code. If you adopt the pattern that we don't do table migrations, one of the features that the ETL, that the data modeling tools have going for them is that they're very good at automating the table migration process. They take the data from table 1, they put it in a temp table, they restructure that data into table 2— into 1B, and then they load the data in. There, you've migrated the table to the new structure. That's one reason why you have these ETL tools. But if you were to adopt the pattern that just says, I have version 1 of the table and version 2 of the table, it's pretty straightforward. First thing I've gotta do is move the data over from 1 to 2. That transformation, but that's a one-time job. And then I've got a job to load Table 1, job to load Table 2. Again, it's all code. Now, because it's all code and I didn't need a GUI tool to manage that, I can now put this in source code control. Makes it easy for me to share. You know, try sharing your work— try sharing your work if you're an Informatica user. It's not easy. It's very easy to step over somebody else's project. But now you've got it in source code control, you can start to adopt software development practices like version control and having collaborative development. Now, whether you do trunk-based or, you know, you go through the whole Git flow, you know, PR, request, um, please use trunk for data work, please put tests in place. But those are all code, and you can start to now leverage a lot of the software development tooling that's been in place for decades. And you're not spending money on these visual tools, you're not locked into those tools. It's much easier to do collaborative development And it's much easier to make incremental changes and move a lot faster. And that's a big shift.

Yeah. So, I'm gonna disagree with you strongly and agree with you strongly in the same sentence.

Let's see if I can guess what it's gonna be. Go ahead.

So, everything is code, right? So, I agree that everything we should do should follow a pattern that we follow when something is code. Yep. So you've described some patterns. You've described we should version things, right? We have a set of things, let's think of it as code, and they're running and we want to make a change, right? We should version those changes so we can go back, we can go forward, we can see what the difference is. That's a pattern. We should be able to collaborate, right? We should be able to see what our colleagues and our peers are doing. We should be able to work together on the same thing. We should be able to potentially split off and do parts of it and bring it back together, although that is a pattern of high complexity. Again, I'm not technical, right? But I can read code. I hate Git merges, right? I hate merge conflicts because I look at it and I go, why did you let two of us hack the same thing and now make it my problem to fix it, right? Give me a tool that makes my life easy, not harder. But that collaboration, right, that it's not locked away on a developer's desktop and nobody can see it, right? That's valuable. Testing. We learned to test code for many years, but we hardly ever test our data, right? So again, testing is a pattern. Automated testing, regression testing, validation, assertion-based testing, those kind of things are important, right? And we should do it when we do it with data. We should treat data as if it was code. So I agree with all that. What I don't agree with is that we can't have a low-code interface that provides that, right? What we've got is a bunch of legacy products that haven't actually given us the rigor and all the tools and patterns that we need to do that. Now, so that's where I go disagree, right? Because at the moment, we're stuck in a world where only writing code is the way we get those patterns and those features that we need version control and collaboration, testing, right? The DBT world. I don't think everybody should have to write code, right? But everybody should get those features, right, when they're working with data and when they're working with code. So agree strongly in all the patterns, don't agree with the jump to, actually, it requires us to write code. But that's just the current state, right? But we're gonna change that with agile data. But anyway.

I agree with that. There is a point where you need, particularly if you're working from the front end back, that having the low-code, no-code type of use case is really powerful. I mean, there's a reason that Excel, and we'll just stick with Excel before we go to things like Tableau. There's a reason that Excel is the number one BI tool. Tool in the world because it strikes a good balance between giving you the power to do things in code like functions. And, you know, particularly if you think about the things like the new Power BI features and the— all of the data features that are now embedded into at least the Windows version of Excel. Give you the very nice low-code environment to work with. And that's really appropriate for a lot of use cases.

But if we look at our vis tools, right, we look at Tableaus and Power BIs and the Qliks of the world, right, the ones that were the previous wave of visualization tools, we don't get a lot of the features that we should if we're treating visualization as code. We don't have to write code, but we should treat it as code. So it's hard to version.

Well, you can get the XML. You can get the XML format, right?

Because those tools don't have the features we want, right? We are forced to extract the code to do what we want. So let's look at them. Can we version them? Yeah, a little bit, right? It's got better, right? We have some ways of getting versions of those visitors and rolling them forward and back. Yeah, but it's not great, right? Can we collaborate on them? Yes, typically now, you know, well, apart from Power BI and its desktop thing, we've got tools that are server-based, browser-based with some collaboration, right? We can start working together. Can we test them? Hell no, right? Show me a, show me a Vis tool that allows me to run a test that says the number in that KPI widget is actually right. You know, there's no testing frameworks for those Vis tools.

So, Well, you could write a report that tests the data that underlies that dashboard.

I can write a test to confirm the data that's going into the dashboard is right, but I can't write a test to confirm the number on that dashboard at a point in time is right, that nobody's applied a filter or inadvertently added another calculation to skew it. So, again, if we say that the pattern is everything is code and the rigor that we bring to the way we develop software, you know, the DevOps, all that stuff should be brought to the data world, then I agree with you, right? And at the moment, I think we're in the chasm again, right? We're gonna jump over and a bunch of new tools are gonna turn up that adopt that. But again, just, you know, my view is it doesn't actually have to force us to write code. It just has to give us all the features that we get when we do write code. So, yeah.

If we can get there, that would be fantastic. There is, yeah, like you said, you can envision a world where you get the best of both. And I think that there are a lot of companies, if you look at, you know, some of the modern data stack companies, there are quite a few that are moving that way, particularly in the orchestration piece. I see this a lot. You've got, You've got the ability to build your data pipelines out visually, and there's data behind, there's code behind it, there's metadata behind it to do things. But that is a lot of work, and we've got a long way to go before we get there.

Yeah, and look, just looking at time, let's close it out. So we can loop back to this idea of 4X and architects, because what the data stack's given us is a whole lot of new cool capabilities, right? It's great. But what it's also given us is an infinite amount of new complexity. You know, in the old days, we used to buy one or two tools to rule them all. Now we have to take 15 to 20 different products, moving parts, open source, software as a service, platform as a service, and cobble them together to give us all the patterns that we need. So, you know, it's kind of a great time and a bad time for data data and platform architects and the data and analytics space because our world's become more complex. But hey, what do we love doing? We love solving complex problems.

Ideally— We love bright shiny things.

Yeah, ideally, let's try and solve them with simplicity. And more importantly, think about the 4 Xs, right? We're at the explore stage, that's cool. But how are we going to exit some of this stuff when the next wave of technology comes? And some of the things we're building in 5 to 10 years have to be exited and replaced with things that have more value. Think about it, right? How tightly coupled, how interchangeable is the things we're designing? Because new stuff will turn up that has massive value and we need to be able to exit the thing at the end of its lifecycle and introduce the new thing without rebuilding everything from scratch. So, you know, 4X for me, it's, it's It's a nice set of patterns.

Great, thanks. Yeah, I was— it's really changed my whole view on how I think about problems. And it's the first thing when I talk to someone, that's the first thing I do. You know, like, you have— before we're going to talk anymore, we got to talk about this 3x, 4x thing, and then other things will make sense from there, because that's, that's the perspective that I'm bringing Yeah, and it gives us a shared language, right?

So again, like Data Vault, there's only 3 things you can talk about. So there's only a certain amount of language you can use, right? So that shared language means we're on the same page to begin with, and then we deal with all the complexity that we, you know, has uncertainty. So, you know, using the 4X as a baseline for a shared language, where are we and which one of the Xs are we working on right now, is a great way of thinking. So look, thanks for your time. It's been awesome.

Thank you.

And we'll catch everybody later.

Thank you, I appreciate the opportunity.

That, Data Magicians, was another Agile Data Podcast. If you'd like to learn more on applying an agile way of working to your data and analytics, head over to agiledata.com. Io.