The patterns of Data Vault with Hans Hultgren - AgileData #41

Welcome to the Agile Data Podcast, where we talk about the merging of agile and data ways of working in a simple in a simply magical way. Welcome to the Agile Data Podcast. I'm Shane Gibson.

Hans Holkrund here.

Hey Hans, thanks for coming on the show. Today we're gonna talk about this thing that I've had a love of for quite a while, the concept of the Data Vault. Before we rip into that, why don't you give the audience a bit of a background about who you are, how you fell into this world of Data Vault and what you do?

Awesome, will do. Shane, great to be on with you. We've known each other for quite a long time and I'm excited to be here with you today. So thanks for that. Yeah, I'm Hans Holkrandt. My background is primarily in the IT world, starting probably the better part of 40 years ago. I was teaching at the University of Denver here for quite some time, started a program back in the day, the first Master of Science in Data Warehousing Business Intelligence that we had here in the US, and then from there started with Vault in late '90s, '99, 2000, and we started a company now, Genesee Academy, primarily focused on training and certification in these patterns and methods around Data Vault. I've been doing that now, obviously, for the better part of 20 years. That's of course how I got to know Shane and spend some time down in your beautiful island.

Been lucky enough to visit you in Denver and go up to Red Rocks and the amphitheater up there and Evergreen, and I So I've got to say that part of the world's got a special place in my heart now. A bit like Vault. There's a few patterns in the world that I stumbled across when I was a consultant, I had a consulting company. A few of them, I feel like an ad on TV over here years ago with a razor, I think it was a Gillette, and it was the owner of the company, he said, I love this razor so much, I bought the company. And for me, Vault's one of those things that I stumbled across it, I started to use it, I fell in love. and my consulting company, I used it a lot. We used to do a lot of work with you around the training to help people in New Zealand understand how to use this thing of beauty. And for our startup, Vault's under the covers in the backend of what we do for Agile Data. Vault's a fundamental, the data modeling technique we use at the core. And why is that? Look, I'm not a data modeler. I'm not a technical person. My co-founder's a techie. I can understand patterns, but I've never done data modeling early in my career. I always left it to somebody else that did. And me and Kimble, lots of those ERD horrible diagrams that I love to hate. But when I saw Vault, I saw or a set of patterns I could understand. Something where it was an approach that was based on patterns. I could understand those patterns and I could apply them. I could explain them. And for me, that's why it enabled us to democratize data modeling out to a whole lot of people rather than what we had at the time, which was one specialist that sat in the cupboard, typically had a beard, scratched the beard for 6 months, came out with this beautiful entity model for the whole organization, which we could never implement. I'm sold and have been for years. What I'm seeing in the world is a lot of people don't know what Vault is, don't understand how it works, don't understand why it might be suitable given their context. So let's go back to basics. When we talk about Data Vault as a modeling technique, what's the core of the patterns? What's it about? How do you describe it when you teach somebody?

Yeah, so that's a really good point. I think that I'm probably as surprised as you are that 20+ years later, there isn't more of a presence of these things out there. I think part of it is honestly that It is, as a starting point, a little bit counterintuitive. So what we've been taught for the longest time is really, if we go back to object-oriented, is this idea of encapsulation of when we talk about any kind of a concept, and even 40 years ago, we're talking about person, place, thing, event. When we talk about a person, place, thing, event, no matter what it is, we model that as one cohesive concept, which, which then almost naturally moves to the form of one kind of a table. And therein lies the rub, right? Because we know the reasons why we do that is it becomes an entity, like in a third normal form, it becomes a dimension in dimensional modeling because it's important to maintain the consistency and integrity of that concept, especially in an enterprise environment. It's important to keep that integrity by having it in one place under one key. And of course, anyone who's done any modeling, understands that one of the predominant features of a modeling paradigm is this key dependency. And whether you're technical or it's business-oriented, really what you're saying is it needs to work by us having descriptors and relationships that relate to the instance of something that's identifiable. That's pretty much it. That defines the heart of all modeling. So in a way, what we've done is we've funneled our thinking into this singular concept model, this encapsulated concept model, this object. And what we're saying here is basically in our pattern, if you take the simplest approach to it, the pattern says, yes, keep that consistency, that key dependency, keep it all together. One, you're only describing one instance of one record, keep that together. However, allow yourself to have multiple tables. And this is logical and physical, right? Multiple tables that can then be added on later so you can build incrementally, or they can be changed without impacting all the rest of it. So this re-engineering issue becomes a non-starter now because we can just add things on. And of course, if we do need to modify something, we're only modifying a very small part. And so when you saw it and like you said, you fell in love, I dove into this head over heels. And the same thing, we fell in love. And then when you go around the globe and that there's other people that have arrived at the same logic and that it makes sense to do this and there's discernible pattern rules that are repeatable and predictable, it makes the whole thing work great. I would say, circling back to your point, it hasn't seen probably the adoption we would imagine it would have by now. However, if you're like me, when you do run a class or if you're on a conversation, you do have a few hours with data engineers and you have that conversation, you show them what you're doing. I mean, at that point, I would say the vast majority of those people look at it and can say, "Yeah, of course.

Wow, that's the way things should be." And I think there's one thing that took me a while to realize is that Data Vault is a form of modeling that we can articulate as ensemble modeling. And there are other forms of ensemble modeling in the world, right?

Yes, absolutely. And that, I gotta tell you, one of the benefits was, is when I was working on writing my book, I had enough time to really focus on writing it. But at the time I was living in Stockholm and I could take the time to travel and visit with people that were doing similar things. And so at that time, I didn't have to travel far to meet with Lars Rönnback with Anker Modeling, because of course he's out of Stockholm. Worked in the university there. So I get to meet with him, Danny Shiner with Trivadis and head inversion modeling. And I put things like Focal Point, and there's various iterations of what people have referred to as 6NF, or because again, that can be interpreted in multiple ways. You could look at temporal modeling from an agile temporal modeling perspective. So there's, there were definitely a good number of other people doing this, being part of our consortium. When we got these people together, we really, I think, what would you say, we, we embraced the commonalities of what was similar about these patterns and found that truly they're probably 85, 90% the same, at least from the beginning point of what we're doing. And so in that regard, that is what we refer to as the ensemble modeling, because it's the essence of this kind of break things apart to multiple components retain that consistency through key dependency. We say unified decomposition as a term for it, but that really is what drives pretty much all of the ensemble methods.

Okay, so let's jump into explaining some of those patterns for people who haven't heard them before. One of the reasons I loved it was an implementation approach. It was actually, we could build code libraries that applied the pattern 101 times from that same piece of code. And that was really based around the idea that in the core of Vault, there are 3 objects that we care about from a physical modeling point point of view. Hubs, sets, and links. So hubs, satellites, and links. And the reason that's important is that means actually in a, in a basic Vault model, we only have 6 bits of code. Yeah, create hub, populate hub, create set, populate set, create link, populate link. Now there's lots of variations we can talk about, but we can effectively automate the building of a Vault based on those 6 bits of code. We pass some parameters to the create hub code, it creates a hub. We pass some parameters to the load hub code, it will load that hub and we can do that 100 or 1,000 times off that 6 bits of code. And those bits of code become incredibly hardened. We use them so often that they become bulletproof, which is what we want in the agile DataOps type of CI/CD world. So if we think about that, let's start off with the first two. Let's start off with Hub and ASAP. So if we think about decomposition, if we think about, we have a core business concept of customer, and there's some things we know about the customer. We know their name, we know their date of birth. So we've got some things that we can use to describe that customer. Customer. When we're modeling in Vault, how do we do decomposition for something like that simple core business concept?

So that's a really good point. I think we're taking, like you said, customer to start with, and you would start really in any kind of a concept modeling perspective. You say, first of all, identify what it is. And like you said, in this case, we say customer, important, of course, it's business-driven, we got a good one to work with. And then, like you say, you have a date of birth, you have a phone number, you have an address, something else that that helps describe elements of attributes that help describe that customer. So in typical modeling, any kind of modeling really, again, dimensional entities if you're in 3NF, or just concept modeling in a logical perspective, you would take the attribute sets and tuck them under that key for customer in a table. And what we're doing here is we're saying, yeah, keep that key dependency, but put them in a separate table. And really, you can define as many tables as you want to have those attributes in them. And so the hub, what we refer to as a hub there, would just be the identifiable instance, the key of customer. So first of all, small challenge, especially in enterprise warehousing, is let's establish a unique key that can be used throughout the enterprise, recognizing that it's going to be referenced by and data will come from multiple different divisions, departments, and functions of the organization all towards a central concept. So, we're establishing that enterprise-wide key is the first challenge. And to be clear, that's the exact same challenge that we have whether this is dimensional modeling or 3NF modeling or any form of modeling. You have to start off with square one, which is you have to have a uniquely identifiable key for that concept, and once you have that, you're good. Now, whatever that is, however you got to it, that would become your hub. But there's nothing else in there. It's just that instance. So that, that becomes like this core firm underpinning that it's there forever. There's nothing about it that can change. It exists. If you want to tell me later it doesn't exist, that would be context. That would be because it did exist at one time. If you're saying it no longer exists, now you're talking about history and context, but that hub just lives on its own as only basically one attribute. Of course, as you mentioned with the loading patterns, we have a technical wrapper that goes around it just to say, here's how we identify it, here's where it came from, here's when it came. Then the satellites would take all or any of those parts of those attributes and form separate tables whose key is the hub. So this way it's still the same dependency because the key of any satellite is just the key of the hub. That's it. Now, because it's temporal, because it's a temporal, we track history, time tracking environment, we're also going to couple that with a datetime stamp to make it a two-part key in the same way we would for a Type 2 dimension or historically tracking 3NF. So it's a very simple pattern. It's— but as we all know, the way to interpret that is to say concurrently active records could only be one. That's the whole theory behind Type 2 dimension as well, which is concurrently active, there's only one record, but for a different time slice, there can exist another record for the same key. So, we're just stacking the history in there. That's how satellites work. And typically what you end up with is 3, 5, 7, yeah, maybe 8 or 10 satellites around a hub, and they're logically grouped set of attributes. So, maybe all the things that rate, qualify your customer, might be in one. And stuff about geographic location and such might be in one. Your primary or main one that may have like the date of birth and phone numbers and things may be in another. Or they could, again, they could be all in separate satellites, one attribute per sat like Anchor, or it could be mostly just everything in one sat. Of course, you gain more of the benefit from this pattern if you logically split them, because then of course as you're building incrementally, if a new function of the business starts to deliver additional context about a customer, it can put that additional context in a new satellite without any re-engineering. So you benefit greatly from a logical separation of the attributes.

All right. So there's actually a lot of patterns embedded in there, and I'll break it out into what I see as the conceptual patterns that we use, and then some of the physical ones, because as technologists, even in the Vault community, we love to argue the detail and the effective dating versus flagging is one of the really interesting technical ones we argue about. But we go back to that core basic. So we have a hub and a hub basically just holds a key. That's all we're saying. So we identify what the key for a customer is, as a customer ID, whatever it is. And that is a difficult problem. Let's not underestimate how hard that is in an enterprise system to have a single view of customer and a single key. But let's just say we start off with one system and we're relatively lazy because I am, and I do a source-specific customer key. and we'll talk about that later. So I have this key, which is customer ID comes from Salesforce. Let's not use that. Usually it's HubSpot because it's a nicer system. So customer ID comes from HubSpot. So we create a hub customer and it holds a key. And, and the other pattern you talked about there is that hub is now immutable. When we saw a customer ID turned up, it goes in the hub and we never delete it because we said at some point in time, we've seen that customer with that ID. We know it happened and we want an immutable record, which is one of the benefits of Vault. Baked in is everything's immutable. It's one of the core patterns. And then in the set, we hold the detail about it, about the customer. We might hold their name, their date of birth, some demographics around gender, anything else. And again, a core pattern of Vault is those sets, they're temporal. They are effectively SCD2, so slowly changing dimension type 2. So what does that mean? It means every time we see a customer and let's say they change their name, We go, we found that customer, we have a record for them in the SAT. It's based on their name. We've seen a new name turn up. We insert a new record. It said at this point in time, the name was this. And we keep doing that. We keep rack and stacking these temporal changes into the SAT. And that is a core pattern of Vault. So we don't have this argument we used to have in dimensional of SED type 1, type 2, type 10, blahdy blah. It's type 2 by default. That is one of the core patterns that you don't break. Now we may have some technical implementation choices. We might flag the previous record as being end-dated. We may only have the effectively a date where we saw the business effective of that record turn up, and we may write some code that windows to say at this point in time, what was the active name for that customer? There's a whole lot of technical implementation that we may want to do when we build the vault physically, but that's just based on a set of patterns and a knowing the type of technology that you use, the way you think, there's pros and cons of all all of those. Don't sweat those, just pick one. And the only point you've got to do is be consistent, just same pattern for the way you implement it. So that's it, set and hub, really simple. Now sometimes we break the sets out like you said, so we can go and say, I'm going to have one hub which is hold my customer key, and I'm going to have two satellites. And I might decide that I'm going to physically model those sets based on fast-moving data and slow-moving data, right? So maybe I'm for some reason holding a number of orders in a set for some reason, and that's a fast-moving piece of data. So I may want to store that into a different satellite. So the rate of change of the satellite is slightly different. Maybe I, I want to do it based on numbers and text. I have a whole lot of case notes and I want to store that into a separate satellite for some reason. So they're just choices. What I do is I start off with a core set, and then when I get a new field coming through that I need to add, I create another set. Why do I do that? Because I have less blast radius. Of change, I can just create another set with that new field. I automatically get the ensemble back and the things I'm touching are less. What's the cost and consequence of that? Now what I've got is multiple set tables. And that is one of the things that people don't like about Vault is actually you get a plethora of tables. So you end up with one hub and 5, 10, 20 sets over time for that. So multiple tables that describe that customer. There's ways you can deal with that from a physical implementation point of view if you want to. So they're just again, choices. So. Hub holds a key, SAT holds descriptions about that key. What's a link? Because that's our third core pattern for Vault.

And let's definitely jump into that. I would say also, just to piggyback off your SAT conversation for a moment, you're absolutely correct. I think that there's a lot of flexibility afforded to us in how we do that. One thing I can tell you is from, I would say, more active deployments in the last 15, 18-year window, we really haven't run into cases where people have been stumped by satellite design. So even though I think now listeners can agree, it probably sounds a little soft. We're saying, hey, you can do this, you can do that. But in the end, if you take into account that there's a few variables, the rate of change of data for sure is one variable. What kind of data, what class, what function does it belong to? What does it mean? What's logically grouped based on some kind of a logical grouping mechanism, whether it's a on ontology of some kind, whatever. What ends up happening is that people will design SATs, and then ultimately they will turn out to be pretty good. And of course, if you need to refactor a SAT, you can. It's re-engineering, of course, it's work. Create a new one, eventually cut off the old one, make the changes you got to make. Maybe it's data migration, maybe it's not. But the point is that generally, we end up with a bell curve. Like, I think even in your worst case scenario, you were talking about 20, I would say that we usually end up with somewhere between 3 and 7 satellites on the foundational concepts that we model. And, and certainly it's a bell curve with the high end of that being, say, a couple dozen, maybe it's 24. And we've seen that in a couple of client sites where it is. And upon review, right, we go through and say, yeah, actually these are valid, logical, good 24 because they happen to have a discernible amount of those things around, in this case, customer would be a good example of that. That would be one, client or customer. Another one might be if you're in the medical field and hospitals, whatever, around the idea of a patient or around an appointment. There could be more than you would typically anticipate would be normal. But getting beyond that couple dozen, I've never seen it. And like I said, the certainly within 1, probably 2 standard deviations from the norm, you're the first standard deviation, you're in the 3 to 7 for sure. And as you get to beyond that, you're 1, of course, it could always be just 1, and then maybe it's 9 or 10. It's not going to be crazy. So just to give some comfort in that, in practical reality, it hasn't really been much of a challenge. When it comes to a link is a, is a way we form From a technical view, it's how we form the constraints of relationships between concepts. And of course, if you're listening, you're a 3NF modeler, you're talking about a foreign key embedded into an entity, wherein it forms a constraint of a relationship with another concept. What ends up happening in all ensemble methods, anchor, focal, to all the ones out there, definitely Vault, is that we move those relationships out of the concept, which of course, makes sense. Unified decomposition and the hub's only a key, so there's nothing else in there. So the links are just relationships. So all they are is they're a representation of a combination of keys that form a relationship. Really important thing about this, and this is where it's probably worth, if you're getting into Vault now, is to spend a little time in understanding the dynamics of what it means to be a link, what it means to represent a relationship, because let's put it this way. A combination of keys and a link, from a technical view, should approximate a foreign key constraint. So, if you're basically in a pattern of 3NF, and you're writing CREATE CONSTRAINT, whatever, entity customer class ID with customer class ID, and you're combining those two keys and creating that constraint, that line of code is effectively a link. Now, the thing is that the link is however, designed, not just a every foreign key gets its own link. We design them by the naturally correlated concepts. And of course, this is driven in multiple different theories from how we gain this knowledge from the business. There's of course discussion around CBC and MBR, we call core business concept, natural business relation, but also the core business process that drives how those things get combined. So let's just say for argument's sake that just like you have to trust your instinct in how you design those satellites, which isn't nearly as scary as it sounds, right? When it comes to link relationships, you're going to end up with a table that's just a set of foreign keys. That's all. It's super simple. But the way you decide how to combine what keys is going to be based on what we refer to as the natural business relationship. And so, for example, if you have a sale, and in that sale it would just of course require there to be a customer and an employee and a store. In other words, the customer, employee, and store are requisite components of a sale occurring. That's the definition of what it means for that event to occur. Then you're going to end up having a link, a singular link, that ties the sale with also that customer and the employee and the store all in one link. And it would be the proper design to combine them all in one. And so that's the element of understanding of natural business relationship is what drives your decision as to which foreign keys belong together in a link. Again, just to reiterate, when you step back, the link still doesn't mean anything by itself. In other words, that link isn't an instance of anything by itself. It's the relationships of something. So in, in this example I just mentioned, that link that ties those 4 together is the relationships, header relationships of a sale. That's what it is. But by itself, it doesn't mean anything. It just means, hey, I'm showing you what are the natural correlated header relationships for the sale. So there's always gotta be for the fill in the blank, whatever that hub is, right? In order for it to work.

The way I think about it is I think about it as core business process. And I think it was on one of your courses that you talk about Peter the Fly. It was Peter. And the way you described it was you're in the organization, you're standing there and you're Peter the Fly. You're stuck on the wall and you're watching these things happen. What does he see? And I still use that. So I think about, let's use a simple example. Customer orders product. We see that happen. So we know there's a customer ID because there's a customer involved. We know there's an order ID because they've ordered it and we know there's a product ID, a thing they've ordered. And we'll come back to many in a minute. And then we might see another process. We might see customer pays for order. So we know there's a customer. Yeah. Customer ID. We know there's an order ID and now we know there's a payment, but there's no product involved. We're now paying for an order. We're not paying for a product, or are we? Good question. And the next one is store ships order. Okay, we've got a store now, new thing involved, new hub, new concept. There's a shipment, so we've got another core concept coming through, and we've got that order. Same ID. Oh, actually, are they shipping orders or are they shipping products? What happens if they ship half the order, two products off the order, but not the other two? So maybe it's actually store ships product for an order, right? Now we've got four keys involved. So this is why we do it, because we're having these conversations. And then we go, okay, customer returns something. Well, what are they returning? Is, are they returning the order? Are they returning a product? And so we can see that we have these conversations about core business processes. And by doing it, we're identifying these concepts. Concepts are a hub. The core business process are a bunch of relationships. They're the links. So when I see customer orders product, I'm going to have a link table that's got 3 keys in it, customer ID, an order ID, and a product ID. I'll start off with that and see if it works. When I see customer pays for order, I'm gonna have a link table with a customer ID, a payment ID, and an order ID. But if I see customer pays for product on an order, then I'll have a full-part link. And one of the benefits of Vault is you can very quickly physically, logically, or conceptually model and, you know, what it should look like, then build it physically, cuz it's just a bunch of keys and see whether the data actually fits what the business think they do. And that's one of the values of it is gives us that agility. We can go, we think it's customer customer orders product, we go and we quickly prototype it, and then we go, actually, it's not. There's a whole lot of extra complexity in that data or in that core business process that they didn't tell us about, that we didn't see. So for me, that is part of the value is it allows us to be very quick and agile and learn early, get feedback rather than big design upfront, big build, and then, oh, that's not what we wanted, or you got that wrong.

I would say just to piggyback off of that, Shane, is that one of the great things about this is that we have come kind of full circle to where a lot of the, let's see, engineering, mulling over the case and trying to figure out what's going on, how to model it, a lot of that's been taken away, or to put it another way, we've been freed up to be able to rely on what's actually happening. So when you get back to that Peter the Fly thing, what's beautiful about it is that we can say, for example, that the customer is purchasing these products in a store from this employee, whatever, and model that in the most logical way. Of course, here's two different grains represented, two different levels represented, one at the header and one at the product of this one-on-one product. And whichever one we're interacting with is the one we model to. So it's just very simple. We just follow what Peter sees. When you have the payment or the shipment, We do the same thing again, but from the perspective of that new event. So for shipment, it's, what am I shipping? I'm shipping products. They happen to be part of an order. Okay, great. Does the shipment know about the products it's sending? If it is shipping at line item level, at product level, then it knows, then that's what you map to. If it only knows about an order, then you map to the order. And that's because that's just a mirror a reflection of the exact business that you're seeing. That's it. And so what's nice is we never have to worry about when we build the shipment that it's going to impact or change what the sales model was, because the sales model is already done. And remember, we're doing additive incremental. We're bolting on. We're just going to bolt on what was there before. So we leave that where it is because when the shipment comes, that's its own subject area, its own feature, its own thing. It has its own relationships. They're already done. We just have to identify them. But that's not going to change what we had before, because what we had before was also a valid process. And so you're going to see many links between the same concepts that drive these different relationships. If you have your third normal form hat on, you're going to look at that and say, wow, circular relationships, bad, rip it up, throw it away. It's not good. But that's because you have your 3NF hat on. You got to take that off for a minute and think— put on your agile incremental build hat, the one that says that when something is there, you don't have to re-engineer it when you add something new. And the relationships that are unique to shipment are not the same relationships that are unique to sale. You don't have to change, modify, or impact them in any way. You just bolt on the new one. And I think the last point to consider just to— and you can tell I get excited. This is why the pattern works so great. The last thing in what you said that I think is good to consider is when you start to put the data in, if there's a gap wherein the data you have can't fill the model that you now designed, there's one of two things. Either the data is right and you need to change what you're doing, or the business is right and the application that captures the data is deficient. In which case, now you can have a communication with the business and say, hey, this looks like to be the way your business actually works. For example, your shipments know at the product level which products were delivered on that shipment. Your data doesn't capture it. And then your business might say, oh shit, we should capture that. And then there might be a revision to the source system because it's not meeting the need of what the business is looking for. So When you have a gap between the two, it's actually a very meaningful thing. It's either revamp the model, something went wrong, or it's the source systems need to be adjusted to meet what the business requirement is. And that communication, you can imagine, would be hard-pressed to happen if you didn't have this catalyst to point it out to you.

Yeah, and I agree. So, Vault gives us a hell of a lot of agility. It gives us the ability to adapt to change. That's a perfect example. If we see customer orders product and we see store ships order, as people who have done data work for a while, we know there's a problem. We just intrinsically know, but maybe they don't. Often we talk about seeing is believing. Show them the problem in the data and then they'll go, yes, or they know the business problem is here, they just can't articulate it. So let's use that example where we go, actually, store ships order. There is no way for us to tell if we're Peter the Fly what products were actually shipped. Because it's not in the data. When somebody sees it, they put it in the box, but that's not captured anywhere. So we can infer, we can join those links later on and say when a customer orders a product and a store ships an order, we can infer that those products were shipped. But we're making shit up because Peter the Fly in the data hasn't seen it. We have no data proof that product was actually shipped. And that's typically a business problem because now what happens if they didn't ship it? It's like we inferred you shipped it. The customer said, I didn't get everything. We're like, we sent you the order. Could you tell us which product you didn't get? Now that's a shit business process, isn't it? So why wouldn't we change our business process to capture the products that we are shipping? Now, let's say that we've been running this for a year. So we've got this link, which is a store ships order. And then we finally fix it, right? We do a digital transformation or they actually work in an agile way and they get to the backlog and they actually solve that problem. Them. And now we have data about the product being shipped. What do we do in Vault? We create a new link because we say there's a new business process which has data that supports us. So now we create a 4-part link. We have a store which ships a product related to an order. So now there's just 4 keys. And we have a whole lot of technical choices about whether we have the data to backfill that for history or not. But from day one, after we implement that process, we have a new link. It starts capturing that information and that data is now captured going forward. That other link, store ships order, we may retain it. It may still have value. We may retain it for a while and then decommission it. We have a whole lot of technical implementation choices, which is great. But the key thing is we've just created a new link in 10 seconds and that data is now available. And that's the other core pattern of a link is it is a many-to-many relationship. So we say, okay, so customer orders product, 3-part link, and there is no way that the customer can place an order with no product. Never happens, never. And now we know that in the data world, never happens is not true. So let's just say agile release of the software and somehow they freak it out and actually customers can now place orders that have no product lines. What's going to happen? In that link, we're going to see a customer ID and an order ID turn up with a null product. There's going to be no product ID. Now that table is just going to absorb that, right? Now we should have observability that says, whenever we see a link that has missing keys, tell us about it, because we care. There's something that we thought should never happen that has happened, but it just absorbs it, right? It says, the data says, this is what you did.

I think that gets back down to the philosophy too. So most of us are on this discussion around agile data for the purposes of enterprise integration, warehousing, something in that realm. And when you get into that, in that realm, we are going to be copying in data that's already been captured. We're not the system of record, but we're taking in data that's already been captured, and it should have been subject to data capture rules that prevent that kind of an error from occurring. But now we have the, the benefit and also the duty to, to become the feedback loop for the business to say, hey, look, we'd like to point out to you that there is a potential issue here. And it looks like the potential issue is that we have 1.8% of your sales here don't have any product on it, or your orders or shipments don't have a product on it. And they say, oh, that's not right. And then you can go and investigate that and find out what's causing it. And hopefully it's a simple data error. And that the actual sources are accurately actually capturing it, but we're just not receiving it, or who knows. But it could also be that it's right at the point of the actual source system's data capture system of record that it's not adequate and there's an error and it needs to be fixed, and then they can prioritize that. And so we're helping them out in a way, almost auditing and feeding back to them what could be a problem with their business processes. As they're deployed in their sources.

And the key thing is it adapts to change. The pattern of how we load that link doesn't change. It just adapts to it and says, the data told me this is what happened. Now, whether that's the business process you expect to happen or not, that's a different conversation and a valuable one. So if I think about in the early, when I had a team doing star schemas with this modeling technique that really, I felt dumb because I couldn't quite understand the intricacies of fact tables and grain and a whole lot of stuff. I kind of, in my head, I did a mapping and I go, if you've got a dimension, an EC2 dimension, you effectively take the key of the dimension and that becomes your hub and you create everything else out of the dim and that becomes your set or sets. And in my head, that's all we do is just decompose a dim down to those two things. And then you've got the first part of your Data Vault ensemble. And then if I think about a fact, we take the fact and the link almost becomes the fact that's holding the relationship. So rather than foreign keys between the fact table and the dims, we're effectively just getting a longer, wider wide table of keys, and that effectively allows us to reconstitute the fact. So again, we don't hold the value of the fact in the link, we hold it in a hub-set combo. But I think about it that way, is that if you've got a fact table, that's the equivalent of Vault as a link in my head.

I would say, just to clarify, you can say from the kind of backward engineering perspective, yeah, that a dimension would be basically a hub, its key would be a hub, and also all its attributes could be split to all the different satellites. If it had any rolled-in outliers where there was a break from the true key dependency, which you do a lot with DIMMs, you would then probably have alternate outlying hubs and link relationships to those, like store to a region or something that would be further out, one level out. But really super important is that a fact table, first of all, doesn't map really purely very well to the Vault at all. But if you were going to follow that pattern, keep in mind that whether it's a dim or a fact, the first thing is it's gotta have a hub, right? So the first thing that's gonna come out of that fact table is a hub representing whatever the event is that drew those things together. And then its link would be one starting point for the link relationships between them. But of course, the thing with the fact table is it's got a lot of mixed grains and nulls and redundancy. It's probably gonna end up being multiple links in effect when it becomes a business model. But I think you're right that the event-based hubs and the links that glue together the concepts are more or less in the fact table area. And of course, kind of foundational concept, not event-based, but person, place, thing hubs and their satellites of context will be in the dimensions. Like I said, with the exception of some of those links to outlier hubs. But what's really interesting is you're looking at it from the fact DIMM to the vault model. What I think is really interesting here is if we model the vault model correctly, if you just take that thinking and flip it, if we model those hubs as the foundational core business concept, which would be the element like your beam core, basically dimension, then the satellites in combination would basically be a fully flattened DIMM. And so pulling from a hub, that's say customer or product, folding in those attributes and then finding it to a dimension should be a trivial process. And of course, if you're in memory, you don't do the whole thing. You just select the attributes that are pertinent to your query at the time, what question you're asking. And of course, interestingly enough, even if you have 100+ attributes for a customer, for example, The average number of attributes that you ever use for downstream reporting in a dimension is somewhere in the 3 to 7 range. And for example, if I'm doing regional product analysis, like which products are popular in what regions, I need to know the product sales associated with the instances of a sale record by a customer. And from customer, all I really need to know is their postal code in order to establish the region. So for that particular query, all I need is one attribute. So folding that into a dimension is super easy. And of course you might bring some demographic attributes as well, and hence you get to 7. But the number of cases where you ever need to pull into a virtual mart delivery a dimension with all 120 or whatever attributes is got to be less than half of 1% of the time. So that's what makes this whole thing Agile and why I think we're talking on agile data.

If we think about common patterns, we often see a three-tiered data architecture. Source data come in and being historized and immutable and raw. Some form of middle area where we model heavily to bring in the business concepts or to combine data or to change data or to clean data. And then some way we consume it, some presentation or consume layer. And I crack up that we go through these waves of what's cool. Databricks just came out with their medallion architecture of bronze, silver, and gold, which is pretty much raw, stuff you frig with, and stuff you use. And I reckon, and again, I don't have the numbers, but I reckon if we went and actually did a survey of all the vault implementations in the world, we would see the use of dimensional models or star schemas in that consumption layer, in that third layer on top of the vault. A lot of the times, they may be physical, they may be virtualized, because we can create them as views. But we see turning it back into a dimensional model for consumption by tools is very common. And that's because a lot of the tools are written to use star schemas. Or there is, people tell me all the time that analysts, everybody knows how to use a star schema, so it's the best tool. And I call bollocks on that because you've gotta learn it. Once you've learnt it, it's easy. I actually say one big table. Everybody knows how to use one big table unless of course you want to count customer because then you gotta do count distinct. So there's nothing wrong with going into a Vault. And then putting a star schema on top of a vault for the way you consume it. And when we talk about patterns, now what we have is we can codify that as a code pattern for physical implementation. We can go, we know the relationship between a hub and its SATs. It's an immutable law. We can't break that. So we can write a bit of code that rehydrates that hub and those SATs back to a DIMM. It's just a piece of code we use 100 times. We know that a link holds a bunch of keys to the hub. Right, and we know the hubs hold a bunch of keys to the sets that describe those things. So now we can write a piece of code that picks up those keys from the links and goes back to the hub and then goes back to the sets and then build a view, dimensional, one big table, Snowflake activity schema. They're just patterns of how you consume it, but they can all be mapped programmatically back to the Vault model, which means we can automate it, we can make it easy. One of the things we talk about is the problems of Vault is complexity of tables. There's just, there's lots of them. And in the past, we cared about that for two reasons. One was storage, 'cause we used to have a really expensive on-prem database. It's gone now. We've got the cloud analytics databases. Storage is relatively cheap. They can hold thousands of tables and we really don't care anymore. And then the second one is how hard it is to use as a consumer. And again, that's why we have to bring this consume layer in. We have to make it easy for them and we have to automate that. But Vault allows us to do that. So would I ever suggest that somebody builds a Data Vault and then has the analyst and hit the hub sets and links tables whenever they want to do anything? Hell no. Be nice to them, give them a lens, give them a view, give them a nice pane of glass to make their life easier, which is what we should be about.

I agree. I think though, one of the interesting things that in my 20 years with this has come full circle is that we've gone from, I guess, taking it for granted that, yeah, it's more complex. And what we found now is literally it's easier and less complex. And this is what's really bizarre about the whole paradigm, because you actually end up in ThermoForm and other forms of modeling, creating table structures that have confusing or get cumbersome in their usage, or you don't understand why they're there. Whereas in our forms of Ensemble Modeling right now, there isn't a hub out there that isn't a core business concept, or let's say line item, some kind of a key for a major relationship that you have. And so when a business reviews that without looking at all the 3 to 7 satellites, but just hubs and links, and they're named properly, that actually kind of mimics in a way the way our brain works. It's very simple and easy for people in the business world to see their model represented in these hubs and links, CBCs and MBRs. So it actually, from that perspective, makes it easier. And what's also interesting, the satellites, is if I'm in one business area and I'm looking for a certain set of context, and we've designed those satellites logically based on business functions or whatever type of data rate of change, they're going to spot their satellites to see what they're looking at, and it's going to make 100% sense to them. Versus looking at that 120 attributes where, you know, 85 of them mean nothing to them and it just confuses them. This is where they can focus in on what they're looking at. And of course now being able to select only the pieces you want makes it easier. So I would say that communications about the model with business users and with everyone is easier this way. I still agree with you, however, that I would put this lens over the top of it for any specific set of business users to view it in their, what is typically dimensional view, because that is how they're used to, and it is for them cleaner and simpler to visualize in that manner when they're looking at the data itself, to see it in that way is cleaner. One other last interesting thing is the amount of space it takes to store every slice of history and all the data. As it turns out, on Ensemble Patterns, it will take less space on if we had disk, it was DASD, certainly on cloud. I recognize it's irrelevant now because it's all pretty much free, but it's a little bit less than otherwise. But the table count does mean more joins if as a general platform. Now, even if I only select one satellite out of 7, it's still one extra join to navigate from the hub to the sat. What ends up happening there is we've had platforms as of the last 5 to 10 year window that are space-free but process-expensive. And a big part of the process-expensive side is you're petting the cat the wrong way with a lot of joins with these methods. So it's expensive. So I think from our perspective is looking to the places where we can see how to make the repeatable pattern and the selection of a subset of links and SATs for a particular deployment really helps mitigate any of that pain. And coincidentally, the current models of things we're using today have actually gotten much more efficient in mimicking, I say mimicking joins, but in processing and working with joins than they were before. So I think we're getting technology bailing us out anyway.

Under the covers, we use BigQuery. BigQuery doesn't like joins. It's a columnar database. It does them. It doesn't like them, but we've never had a problem with it not running them. We just try and remove them for the user as much as possible. And the other one is, that cost one's a really interesting conversation, cost of compute. So, you know, if you look at BigQuery, Snowflake, you pay for what you are computing. They charge you different ways, but it's always based on you query more, you pay more. That's the model. And so with the Vault stuff, you can think about it in clever ways. If we have users that always go in and write a query, which is SELECT then we pay for scanning all that data, right? Or our credits for the thing to run for the period of time to process all that data. But if we're monitoring, as we should in DataOps, we should be monitoring who's running what query, what the common queries are, what the common columns that are getting hit regularly, then we can restructure our— so let's put the commonly hit fields in a set. Let's give them a view or a physical table or a dimensional model that is customer and most commonly used customer fields. And they can select star all they want and your cost goes down. And to be clear, we give them a second table or view or star schema, which is everything. And we say to them, if you need everything, then go over there. We bring this idea of friction in for a reason. Go hit this one first. It's cheap. It's easy. If you need everything, it's there. There's just a little bit more friction for you to go and use it because there's a cost and consequence of you doing that. There's many ways we can structure those satellites and the way we provide that data to be consumed based on our context. It's always about our context. What are we focused on? How does our organization work? What's important to us. So if we go back, so hubsets and links, they're the core immutable things within Vault. There are some other patterns. There's SALS and HALs and a few others, which we use for specific cases. So I don't wanna go through all of them because we will run outta time. Obviously they can come on your course and find out all the rest, but let's just cover HALs and SALS because there are variations on some things we use on a regular basis. So can you give me an explanation of a SAL and then a HAL or a HAL and then a SAL?

Up to you.

Sure.

If you're, I'll explain it in two ways. I think that if you're accustomed to third normal form, it's effectively the ensemble embodiment of a recursive relationship. If you're not as familiar with third normal form, what we're saying is that if there is some kind of a relationship wherein, from the same concept, you want to relate it to itself. For example, one would be for deduping. You would say, I have 3 records in here, that came from 3 different functions of the business, even though I did a good job on my enterprise-wide key, I'm not 100% confident that these 3 records are all the same, but I believe they probably could be. So what you do is you say, let me form a relationship between, let's say customer again, customer A, which is Hans Holmgren with an HH102 key, and customer B, which is Hans Holmgren with a 191724 key, some other key from a different place. Now, I have reason to believe they're the same, but there's also a chance that they might be just two people named Hans. It could be. So what I do is I say, let me form a relationship between those two records where I tell you that I believe these are the same. So that would be a same as link, or a SAL, same as link. So you just have a link that says I have— I'm a two-way link and I'm pointing both of my directions to the same hub. So I can grab two different records, and I'm telling you that I believe they're the same. What's nice about that is that we can then further add a descriptor hub to that instance. If you look at it from an entity with a recursive relation, you would embed the foreign key in the entity pointing to itself. That's all recursions in 3NF. But what's beautiful about this is that you can have 5 SALs outside of that Customer Hub, because there's no weight on the customer concept at all. ASAL is floating on the outside, right? It's floating around that instance of the hub key table for customers, so it's not embedded anywhere. So let's say you throw one on there and say, oh, I'm going to match first name, last name. Oh look, I got 50 matches, they're all the same. Of course, if you're in Sweden, I can tell you right now well, that's not the same person because that's not enough information to make it match. So then you have another cell that says, look, both these systems may not have all these attributes. And even if you did, they may not be entered the same way. But I tell you what, if you did have all these attributes and if they were entered the same way, I'm 99.9% sure it's the same person. So now you can say my algorithm is first name, middle name, last name, date of birth, city of origin, mother's maiden name, highest level of education. Okay, once you've gotten to that, you're like, shit, that's the same person. So now if you think about it for data science purposes, for reporting purposes, how are you going to use the data downstream? It's imperative to know what's the strength of that deduping logic, and it shouldn't have to be either or. So what we see now is as many as you want. And of course, if one of them is not useful, you toss it. There's zero impact on the model. Because it's of course just the same as a link. Now the hierarchy link, or the how, it actually works technically the same. It's a single link with a key pointing to the hub twice. So two different keys pointing to the same hub. But here it's saying there's a parent-child relation. And what's really nice about that one is you name the first one in the link that's called how, the how link, You name the first one parent and the second one child, and you load it with a parent-child relationship. What's beautiful about this one again is you didn't do recursive hierarchy for this like you would in 3NF, where you had to embed a foreign key. The benefit of that is that now, guess what? When you go enterprise-wide, there are multiple customer hierarchies, there are multiple product hierarchies. So you end up having a product hierarchy that's based on its marketing or sales or campaigns. You have another one that's based on the kit that it was built from. You have another one that's built on what regions it's in. The hierarchies can be from different perspectives. You can have 3 Hows, 5 Hows on, on a product that are each having their own specific meaning. Plus, if you have a descriptor keyed instance on those, you can now accommodate, if you're familiar with dimensional modeling and all the issues of the jagged and sparse hierarchies and different issues of jumping levels. Well, because we can describe each one individually, you can be succinctly outlining any form of hierarchy at any level, jagged, whatever, and it embraces and captures it instantly. So, it's a very flexible manner. It's actually one of the exciting things I think about Vault is that besides just the unified decomposition bringing us this kind of unprecedented agility, the whole realm of deduping and warehousing, which is important, and the whole realm of how do we capture varying hierarchies has been like 10x simplified by using these elements. What's your opinion, Shane?

Yeah, I agree. I like it because it takes a repeatable problem, or a few of them, and it gives us a repeatable pattern. So that one you mentioned about customer matching, that's the perfect one. So in Vault, we say customers should only have one unique customer ID and we should have a single hub called customer. Peter the Fly sees you walk into a store, you get your ID, sees me walk into a store, I get my ID. That's nirvana. That's the pattern that we really want. But I rarely see that in an organization that has more than one system. We end up with customer ID in Salesforce and customer ID in HubSpot and customer ID in the e-commerce store. And 9 times out of 10 or 11 times out of 10, a customer ID in this Google Sheet or an Excel spreadsheet that somehow became a system of capture. And so cells allow us to do that. We say, like you said, we think these customers are the same. So let's store that information and use it. And that relationship of this might be the same as that. And same with products. We think here's a product catalog, here's another product catalog. We think those are the same products, but the SKUs are different because they're coming from two different things. We think they're the same. And then hierarchical links or house, the classic one for me is Emp, Emp and Manager. So you've got a table of employees and one of those employees is a manager and they have a team that report to them. So now I've I've got a link where I go, this employee relates to this employee. And again, that how hierarchical link allows us to do that with simplicity. And it keeps track of changes over time. If I change that relationship, then that link's gonna get a new record to say, hey, Shane no longer reports to Hans. He's, he's been promoted and he reports to Remco. And we're just gonna see that new relationship turn up. So I love them. I think they're really great. I think one of the problems with Vault is you go and learn the three cores, hubsets, links. and then you get some ones added on, sales and house, and they make sense. And then we start going into some more complexity, bridge tables, a whole lot of other things that have value for certain contexts. And then we get confused. So again, start off small. Looking at the time, we could always talk about lots of use cases, header-detail modeling, is address a concept or a descriptor, but we're out of time for that. So we might have to do an advanced Data Vault patterns podcast. But I just wanna come back to the core. One of the things you mentioned was the ability just to look at the data model and understand something.

Something.

And a lot of that comes back to naming standards. As technologists, we love to argue. Is it hub_cust? Is it H_cust? Is it H-customer? Is it cust_H? Who cares? Pick one and don't change it. So if it's hub_concept name, then make sure everything looks like that. And same with sets. Is it set_cust demographics or is something else. And the reason that's important is, again, one of the values of Vault is automation. So once you have standard namings, you can codify it so that it's always created that way and it always looks that way. One of the anti-patterns I see is if you're using Vault and you have a bunch of data engineers that are writing code to manually create and populate the hubsets and links, then I suggest you have a look at different patterns. 'Cause remember, there's only 6 bits of code for those core objects. So automate it, have some config or metadata. It's a data modeling technique that's designed at its core for automation. And that's why we use it in Agile Data because we can write code once and use it many times. So let's just go through the ones that I picked up talking to you. Given it's based on some core patterns that are immutable, we can automate the creation of that data and those structures. Our code becomes hardened over time. So it's the DataOps, write once, use many approach. Approach. The data becomes immutable. When we saw a piece, something happened in the data in the organization, we store that forever. And we can make changes to the organization or put layers on top to represent that data in the way it should have happened. But that is immutable. Everything's historized. SATs are ECD2 by default, right? So we always have all history over all time. We can adapt to change. So links are many-to-many. If something needs to change, we create a new link. Or if the data changes, it absorbs that change for us. We can re-engineer at will. We can add a new set with very small change on the rest of the model. We can create new links. We can, if we wanted to, if we ended up with 100 sets, we could decide to rebuild a whole new ensemble with instead of 100 sets, 5 sets and repopulate history from those previous sets into the new ones and use that as our new consumable layer, run forward with that. Because it makes us feel better. And then we've still got the old set sitting here archiving with history because that's what we saw at that point in time. They're all good choices. We can modify small parts. We can take an incremental approach. We can go and build customer orders product, deliver that value to stakeholders early, and then say, right, now we're going to go and deal with shipment. Oh, shipment's hard. It's going to take us some while to understand what the hell those systems are doing. And then they might say to us, could you do payment? We just go have a look. Yeah, no, that's easy. It's just customer pays for order. There's no complexity in there. Let's pump that one out while we're starting to think about the complexity of shipment. So again, it gives us the ability to deliver incrementally, deliver small bits of value fast, which is what Agile is about. It deals with master data. So this whole data mesh thing that is really annoying me because yes, I agree that democratizing and decentralizing back is valuable. We get a whole lot of benefit out of that, but nobody's talking about how the hell do we combine it. It again because customers everywhere, how do we get a single view of customer? That's what a data platform is about. And Vault takes care of that. It doesn't make matching any easier. It doesn't make actually identifying it's the same customer any easier. But once we know how to do that, storing it and using it, that's the core of what it does. What else have I missed? Any other kind of core whys on Vault?

I think you really went through a great list there. Like you said, ability to automate repeatable patterns, hard hardened over time, immutable concepts, everything is historized, it's by default. That's the way SATs work. Super easy to adapt to changes, re-engineer at will SATs, links. And like you said, you can always refactor, which does require some re-engineering, but it's minimal comparatively. And of course, the pattern that we recommend, you could do anything you want. The pattern we would recommend is you build the new SAT, you keep the old, you do a data migration if you need it. Same thing with links, if you're refactoring them to make keyed instance or something, keep the old one as well. Put the new one in and move forward from that day. Constantly adapt. You can modify small parts without hitting the rest of it without the impact. Incremental build any order, which was, I think, what you highlighted, which is, hey, let's do shipments. Wait, complex. Let's do something else, purchasing first, whatever. And what's really nice about that is, we always try to point these out. You treat them all as if they are their own thing. It's an easy bite-sized nugget. And when they go into the main model, they don't cause you to have to re-engineer. And it's— if someone out there listening is a dimensional wizard, it's just as if you're adding new fact tables, new star schemas into an existing Federated Mart environment. You're using the same dimensions, but you're just borrowing those keys to pull them into a new fact. But it doesn't impact any of the other existing facts or what they were before. It's all additive, right? And that's a huge benefit to the incremental build, which is really important. And then, like I said, also master data. What's nice about that is it's a continual working effort. The more we learn about which ones are the same, or actually find out they're not, or whatever, we're never screwing up the core. We still have the data we can work with and represent it in whichever way. And even if you have a master data initiative in the operational world, you can match it against your sales that have been doing their work to integrate from the warehouse side and then learn from each other. Is this right? Did we get this correct? Both from our side reflecting what the business is saying and the business side seeing what we're observing. We can communicate with each other on the two. There's one other quick thing I've mentioned is that because of this incremental build, people sometimes look at Vault and say, "So many tables, the total cost of ownership will be higher." Maybe the long-term total cost won't be higher, but the initial will be. And as it turns out, what we found is the initial cost of ownership is lower, as well as the total cost, because we're now able to realistically start with one small component. That once you finish it does not have to be re-engineered again. So actually being able to start with a small increment, and then when you're done with it, it's done. Like, it won't need to get re-engineered when the other stuff is added on. And then of course everything else you add on is easy to add on. So you end up being— from the portion of your team, the function of your team that's dedicated to expanding the model, that effort effort will be less. There'll be less of an effort moving forward using this technique. So that's super cool.

Yeah. And again, automate. Don't treat it as a manual Turk job where you've got people writing the code for a single hub in isolation, because that's just an anti-pattern.

One quick thing. If in the future, like you mentioned, if we need to get into some of these other topics around header details and link-to-link, multivalued link satellites, if people are interested and we should do another session, we could probably spend an hour on those features of the pattern. But then people listening would and should be already familiar pretty much with the foundations of OLTP modeling.

I think for those ones, maybe we'll do an actual video, 'cause I think examples, we can see the data, 'cause you start to get into some of the complex use cases. So before we close out, here's a question for you. If I look globally, I would say that USA is star schema or Kimball-centric. The majority of the work done in the US is around dimensional modeling. If I look at Europe, it's heavily Ensemble or Vault-centric. Is that true from what you've seen of all the customers and all the people you work with? Do you believe Europe seems to be more Vault-centric than the USA? And if so, any ideas why, given you've been there pretty much from the early days?

Yeah, I gotta say, surprised that is still true. And I think that you're right to observe that. I think obviously you're in the center of this. You have been for a long time. My involvement with New Zealand has been largely through you and colleagues in that group. I see a good amount of uptick because my lens to that environment is basically through your network and through people that we've met over the last 10 years down there. Same thing for Australia. So I think there's a better percentage of people adopting these patterns, or they're more open to it there than they are even in the US. Europe and Nordics pretty much lead, I think, that charge as far as adopting these techniques. Definitely Netherlands, Sweden, probably top those charts, and then all surrounding areas in Western Europe and Nordics seem to be doing quite a bit of it. The question mark as to why, I wish we could solve it. If you think of the pattern in technology advancements. A lot of them come from the US, and it's not been an obscure reality. It has been true for real that these things may come from the US, but they get adopted in Europe and Nordics and Australia, New Zealand before they do in the US. I think it's a super strange thing. It's not to say there aren't a lot of people in the US doing it. There are. But if you look at comparatively, it's definitely much more of a pattern in these other places. The only thing I can say is that it seems to be culturally that if you're in Western Europe, if you're in Nordics, if you're in New Zealand, you're in Australia, if you're in charge of a warehouse program and you're the enterprise architect, it seems to be a little more of the authority and the weight and the responsibility of that rests with you. Whereas I think in the US there seems to be a little more of that authority actually resides in a person 3 levels above them. I think there's a little bit more of that culturally here in how the large organizations run, where they just say, okay, CEO, nobody got fired for hiring IBM kind of thing. And then it's not really happening as much as it should be. And it's too bad. I think we could all benefit more from it in the future. Interestingly enough, in the last 2 years, we've had a much larger uptick in our certification training for people from Accenture, Ernst Young, Deloitte, the bigger companies, and a lot of US, comparatively to prior years. Still in all, the vast majority of things are happening with you guys and with Europe and Nordics.

And even Australia is funny because if I look at that, Brisbane was a hotspot for Vault whereas Sydney and Melbourne wasn't. It was like a regional hub and I could probably tie it back to a couple people I know there who led the charge and they were doing great work and therefore we're successful doing it. My view is, does Vault have problems? Yeah, there's a whole lot of patterns in Vault that make things hard, like any modeling pattern. Do I hope we get a new generation of modeling that retains the benefits of Vault and solves the problems with some other things? Hell yeah. Haven't seen it yet, but I mean, I go and I look at Unified Star Schema And I look at Activity Schema and I look at those and I go, are they the next thing that solves the problem for me? And they solve one problem and then they bring in a whole lot of other problems. Again, it's Vault with star on top. You probably end up with a data architecture that absorbs multiple data modelings at certain points in time, given certain context. In our product, we do one big table of Vault, often because there's some value. We would pretty much do one big table. As a consume view over and above a star schema if we can, because there's some benefit to us in the way we do it. There's a consequence. Like I said, you gotta count distinct customer, or you've gotta hit the customer ensemble. You gotta go to the view that's just about customer. Don't count customers off the link or the event that is related to a purchase. Go and hit the table that's called customer, funny enough, and how many customers we got. But anyway, there's cost and consequences of anything. There's patterns and anti-patterns. And yeah, if you think you've got got a problem around immutable data, around historization by default, re-engineering, small changes, start early and deliver value fast, then have a look at Vault and see if it fits for you. So, if they wanted to do that, how do they get hold of you? Because I've been through your training, you're not paying me to say this, it wasn't bad, it was okay. I use it, which means I learned something. If people wanted to get hold of you, what's the best way to get in touch with you?

You can email me at any time you like. It's hans@geneseeacademy.com, and it's G-E-N-E-S-E-E, so a whole bunch of E's, geneseeacademy.com. And then of course you can just go to geneseeacademy.com and check out the website for classes that come up. We did pretty much all of ours in person. With COVID we did 100% online. Now we do probably 50/50. We were doing probably more than half of our classes were public classes, were open enrollment. Now we probably do at least 85% of our classes are private in-company classes because the organizations that are using this technique want to get groups of people trained. And so we do more private classes, but there's still a good number of public classes available to take. So look on the website and definitely sign up. We run certification on it. We're now issuing cool looking badges so you can get your credentials on LinkedIn and get paid more or whatever happens in that realm. But if you come to the class, all I can say is bring an open mind and roll up your sleeves cuz we'll be modeling in these classes. This is a modeling-centric class. It's not a, hey, here, let's read this text and repeat it back to me. It's an interactive modeling class and you learn a lot about the pattern getting in it. It. And I think it's worth it. And I think you can agree with that as well, is when you take the class, getting into the actual touching, feeling the pattern through modeling it is really the way to get a hold of it and to learn it.

I learn by doing. Like anything in data, it looks simple. The pattern's there. You got a hub. It's a bunch of keys. That's customer. And then you get whacked with a source table that's a party entity and holds customers and employees and suppliers and contractors. Okay, how do you model that out? It's simple, but again, going through that, learning by example is valuable because then those kind of complex patterns, they're going to hit you, turn up, and you know how to deal with them. Still takes a brain, but you've got the toolkit. I'll put all the links in the show notes for anybody that wants to get ahold of you. Obviously you've got a book as well, so I'll put that in there. Have a read of that if they want to do that. And glad that you've got badges. Just the next thing you need to do is t-shirts because there's no Data Vault t-shirt coming out from you guys. I think that's probably the next thing to add to your—

I think so. And can I consult with you later about design? Because I was telling somebody the other day, I still have a batch of the most awesome t-shirts that I got from you, the Optimal BI ones. And there's some great ones.

Yeah, with our new startup, there's a new website called adt.style, and there's a bunch of t-shirts, and I'm wearing one today. It's the Bring Back Data Modeling. So if anybody wants a Return of the Data Model t-shirt, just go buy one. You can walk around agreeing with us that data modeling is an art, it's been lost, and it should come back. It has value. So regardless of what your flavor of model is, just make sure you do it, right?

Actually, that's a really good way to wind this thing up. I agree with you. I think that right now, especially with a lot of the lake and toss it all in there and it just automagically works, I think we all know as data professionals that's never going to be the case. And so probably the most important thing of all from this pattern-based podcast that you have is just do model. We need to get into modeling again. And there's a lot of things that, that are effective and can work. And what we're focused on here in these discussions is what might work better for certain purposes. And that's it. But either way, we should all be doing it. And I'm also like you, Shane, is I have two sides to my Vault story here. One is that I feel like now it's so much easier and so much stronger than I thought it was 10 years ago. I think it's awesome, and I think it's easy. I think it's really the best way to go for a lot of things. But at the same time, I am ready— as soon as the next best thing comes out, I will jump on it in a heartbeat. And I think we all need to be able to do that. We all need to be able to say, hey, I've done it this way for 15 years, but here's something, let me look at it objectively. Yep, that is better for this purpose. And then please, then jump on it. I will commit to doing the same We should all be doing that. I hate to be the guy saying, oh, I've always done it this way. You know, that's— may as well just hang it up then, because that's not going to get us anywhere.

That's right. And we learn new things as we touch new data. Just bring it back into the pattern and get the pattern to absorb that new complexity when we can. Excellent. All right. Hey, thanks for coming on the show. And I hope everybody has a Simply Magical day.

Thanks for having me.

Data Magicians was another Agile Data podcast. If you'd like to learn more on applying an agile way of working to your data and analytics, head over to agiledata.io.