AgileData #37 - Conceptually Modeling Concepts, Details and Events in AgileData

Welcome to the Agile Data Podcast, where we talk about the merging of agile and data ways of working in a simply magical way. Welcome to the Agile Data Podcast.

I'm Shane Gibson, and I'm Nigel Vining.

And today Nigel and I thought we would do a little bit of a deep dive into what we call concepts, which are one of the three types of objects that we create and store data as within agiledata.io. So the three objects that we store are concepts, details, and events. And the 30-second thing that I use to describe them is concepts are a thing we care about, something we need to manage. So an example of a concept will be a customer, a product, or an order. Detail is something that describes those concepts, that provides some context. So it might be the customer's name, the product's type. Or the order value. And the third object is an event. And an event is where a group of concepts have a relationship at a point in time. And they are normally articulated in the way of a core business process. So we would say there is an event of a customer ordering a product, and we store data in a certain way when we see those events. So Nigel, we, under the covers, you know, everything we do in the event layer is stored as either a concept, a detail, or an event once we've defined it. In the config, you know, is that a lot of work? When we implemented that thing, was it months and months of programming to get those patterns for those objects?

No, Shane, actually the Data Vault modeling tends to be very nicely aligned to the actual code under the covers to make these things work. I guess I was actually just going to start by preface this with my background's Kimball. So, when I historically modeled data, I always thought in the terms of dimensions and facts. But actually it was a very small learning curve and adjustment to me because effectively a concept is the key of a dimension, a detail is effectively all the attributes of a dimension, and the event is effectively all those keys of a fact table. So for me to go from the facts and dims world to a concept, detail, events was actually really easy. But what I love most about the Data Vault C's, D's, and E's is actually the wickedly easy to automate under the covers because the pattern for a C is really straightforward. It's basically a table with a whole lot of keys in it. D's is not much more complex. It's a table with a whole lot of attributes in it. And events again is actually a table with multiple key columns in it. So under the covers, we can automate the snot out of creating and maintaining those. And we can even extend them on the fly really easy. We don't end up with that situation known to developers all over that once you've built your dimension table and deployed it, a month later someone comes along and says, oh, we've missed an attribute out of it. We need to put in. What you end up with is dimension tables in mature warehouses which have your date columns out on the right, and then there's another 2, 3, 4, 5, 6 columns after those because they've been added on after the fact. The Data Vault paradigm lets us effectively extend out our detail tables at any point with no real hit.

Yeah, and I think it's important to explain that under the covers we use the majority of the Data Vault modeling patterns The reason we use different terminology, so anybody that's used to Data Vault will be familiar with hubs, SATs, and links, or hubs, satellites, and links. And so for us, a concept effectively is a hub, detail is effectively a satellite, and a link is effectively an event. The reason we call them concepts, details, and events though is our audience is analysts and business people. And so when we talk about hubs, satellites, and links, those are quite technical terms. That really don't make sense to them. But when we say, you know, we have a concept, which is something you manage, it's, you have a concept of a customer, you have details that describe that customer, and that customer is part of a business process. There's an event that happens at some stage involving them. The business people understand those terms. It was funny, I was on one of the Data Vault meetup things that we do globally every year, and it was meant to be in Rome this year, but unfortunately it was remote. And we were having a bit of a chat about, you know, the software market at the moment and how Data Vault's been around for a long, long time and lots of people were using it in their data platforms as a modeling technique, but the tools themselves are pretty immature or have a low sense of adoption. And my standing joke was, well, why? Because if you're going to create a Data Vault, you only need 6 bits of code: create concept, create detail, create event where you populate the table, create the table, load concept, load detail, load event, right? So there's only 6 bits of code to make your whole data platform work with Vault.

Yeah, that's exactly right. And even if you come to a level of standardization under the covers, you can actually get that down to two effectively, which is create your table and update your table, because the patterns, they actually share some commonalities and you can just smudge the lines a bit to even simplify it. It's funny you say that, actually. I'd completely overlooked the hubsets and links because I've been calling them Concepts, Details, and Events for so long, I've actually forgotten that's where they actually came from.

Yeah, well, maybe we'll come out with Data Vault 3 and we'll rename them. Probably not. Yeah, in the Data Vault world, there's lots of arguments on the detail. As they say, a group of architects is an argument of architects. A group of Data Vault modelers, an endless argument about particular use of a single field. So for us, we have adopted some things that work for us, for the way we work with our customers and the way the Google platform that we leverage under the covers, make the best value of that. So let's talk about these concepts, Nigel. So, you know, in the dimensional world, you know, we have these things called dimensions. You know, yes, a concept is almost like a key on a dimension, but sometimes it's not. So I'll give you an example. Dates. We don't have the concept of a date and we don't hold that as a concept. A date is not a thing we manage. A date is a piece of detail about something else. It's the date that an order was placed. It was the date that the customer paid the invoice. So that's bound to the date of the invoice. So we don't hold dates as concepts like we did in dimensional modeling. So there are some slight tweaks in terms of the way we use the Data Vault modeling compared to the way we do the dimensional modeling. But you're right, it is fairly similar. And that also means, doesn't it, that once we've done this modeling in the event layer, we can quite easily present all the data in a consumable way as dimensional star schemas again if we're using a tool that consumes star schemas by choice, right?

Yeah, exactly right, because it's not very much— well, sorry, it's no effort whatsoever for us to put a C and a D back together and call it a day. Same with the event table, we can easily join an event table back to its C's and D's and call it a fact. So by effectively utilizing a pattern, an easy to implement pattern, we can turn a C, D, and E back into a collection of DIMMs and facts, and the tool that's sitting on top of it, the BI tool of preference, will read those quite happily and not realize that under the covers it's actually something else.

I think the other thing is when I've worked on some of those dimensional-only data warehouse projects that didn't go so well, we were commonly doing something called a factless fact. So what was that?

A factless fact. So a factless fact, we were typically trying to join something together. I'm just trying to think of an example of one. You put me on the spot. A factless fact. There's nothing to measure, but we needed to join two things together. So I don't know, you threw that one in there. You got an example?

Actually, I'm trying to think. I mean, I normally treat them as hierarchies, but that's probably not a good example. So, they're an exceptional, like, they're an exception to the fact modeling technique, to the dimensional modeling technique, right? The factless facts. It was something that was used where there was no value for that fact happening. So, you know, it was an anti-pattern effectively. And the point I was trying to get to is within the way we model, you can have an event that's not driven by an order value. You can have a set of concepts that have a relationship at a point in time. And that event concept, that event table we have, where we say these concepts have this relationship as at this point in time, is valid, right? It doesn't have to be a numerical number driving the fact that that relationship happened. There just actually has to be a relationship in the data to say that it happened.

Absolutely. And that's quite a nice real-world thing. Something happened. These three things intersected at this point in time, a person and you know, something they did, or— I can't think of an example for that either.

I've just Mr. Googled it. Student attendance at a class.

Oh, right. Yeah, okay. So I guess you could say you could count that and say it's one attendance, but that's basically as far as the metric that would, you know, be attached to that fact. You could say a student attended a class, count it once. But otherwise, you're right, it's just three keys: a student, the class, well, I guess the attendance is actually the count they attended.

Yeah, so I think with Factless Facts, it was where, you know, there was a count of 1, and the 1 didn't exist in the data, so we had to make it up. Whereas for our events, student attends class, that's a classic event, right? We just have the student concept, and the attendance concept, and the class concept, and where we see a relationship with those 3 things happened in the source system somehow, we store that as an event record. To say that at this point in time, these three things had a relationship, right? We saw them happen. So, there's probably some complexities when we start talking about concepts because our source systems are never clean, they're never beautiful, they don't make our life easy. So, in your experience, what are some of the things that you've seen that are a little bit more complex patterns or techniques or recipes we have to use when we're creating these concepts?

Okay, I guess the one that springs to mind, what if we have multiple source systems feeding into our data warehouse? Are we combining those keys together into a single concept? Are we storing them separately? What do we do with keys coming from different places?

So the answer is yes, we are all of those. So they're just choices. So what we can do is we can define source-specific concepts. So we could have, say, a Salesforce CRM that has a customer record in it. And maybe we've got Xero, a financial system that's got a customer record in it. And Salesforce is holding the events around communications and service calls, and Xero's holding the records around invoices and payments for those customers. So we can create two concepts. We could create a CRM or Salesforce concept. And populate that with the Salesforce data. And we could create a Xero or financial customer concept and populate that data with Xero. We would be very— we would recommend that they have different names, right? So either financial customer and CRM customer or Salesforce customer and Xero customer. So we know when we're looking at the data which concept we're looking at. Now, we may then want to actually have a single list of customers. So what we can do then is create another concept, right, called customer, or single view list of customer, or master customer, or golden customer. Yep. And then we can take the two concepts and we can slam them together. Now, if the keys between the key that identifies a unique customer in Salesforce and a unique customer in Xero, well managed and synchronized. So if you're a customer 99 in Salesforce, you are customer 99 and 0, then just basically slamming those keys into that one concept is magic, right? It'll do it all for you. 9 times out of 10, that's not true. 9 times out of 10, the way we describe those customers or identify them— sorry, not describe them— is different. So then what we have to do is write a rule to say how we're going to deal with that. If the way we create those identifiers is unique for each system and they don't overlap, there are no collisions, so customers in Salesforce always start with letters, A, B, C, D, E, F, G, and customers in Xero are 1, 2, 3, 4, 5. We can slam them into the same concept, but we won't get to see the deduping. So where, you know, McDonald's in Salesforce and McDonald's in Xero, they won't give us back the same, all the records from both systems, all the events. So what we do there is we have to write a rule, right? And that's a matching rule that says, how do I know that McDonald's in Salesforce is the same as McDonald's in Xero? And then what do I do about that? And we end up creating one record, one concept record for McDonald's from 1 to 6. We may find out that funny enough, there's actually 6 McDonald's customers in Salesforce, but they're all the same customer because we just don't use the system properly. So we'll write a rule there to again, create a single ID for McDonald's in that, in that single list for that concept. So just done through rules.

Great.

Okay. So that's just one pet technique and recipe that we use for that. The other one then is really important is this idea of details. So we have details coming out of Salesforce for McDonald's and we have details coming out of Xero for McDonald's. So we may have an address. The addresses may not align. So we will normally create a detail table with the Salesforce detail and a detail table with the Xero detail. So we can always see what the data looks like based on what the source system looks like or the Data Factory. And then we, again, we have to make some choices. What do we pick as the single address that is the master? And again, there's just rules. Is it the last one that was updated? If we've got 6 McDonald's, do we say the address that's seen the most often is the winner? Do we say, actually, salesforce is normally right, zero is normally wrong? So if you see one in Salesforce, use it. If you don't, grab the one from Xero. So again, they're just rules that we use to identify what's important.

So you're effectively using rules to remove the complexity and ambiguity in your source data. So by applying the right rules, you're effectively getting to your, what you call, master records, and it's just a rule.

Yes, because everything we do is just a rule. And if you are a consumer of the data, you don't want to have to typically choose, because you don't understand the rule you need to apply normally to figure out what's right. That work should be done by the data magician in the event layer before it becomes consumable.

You just want to see the unique list of customers and their current address details.

Yeah. And I always thought, yeah, this isn't going to happen that often, but I read a research paper a little while ago, and what they were saying is organizations between 0 and 50 employees of these has on average 7.8 source systems, 7.8 data factories, which blew me away, and over 50 employees. I can't remember the number, but it was a lot more. So, if you think about 7.8 systems, how many of those are storing information about a customer? Unusual not to have to combine this data at some stage.

Absolutely. So, this is somewhere that the concept of concepts and rules would really be beneficial for these organizations to clean up their data, present it back in a, I guess what you'd call a single view of customer.

Yeah, and because we take an agile approach, we can do it incrementally. So, if reporting on customer from Salesforce is the most, the highest value piece of information for the organization right now, just create the concept of a Salesforce customer, start reporting on it, don't worry about Xero. But if actually combining the customer records across 2 systems or 3 systems is your most highest problem, your biggest pain in the bum right now and the highest value, or de-duping the customers from one system is your problem. Again, you just write rules that create concepts that deal with that problem, and we have the flexibility to change it later on. I mean, one of the other recipes or patterns we can use is you can, from the beginning, just create one single customer concept and don't do source-specific ones. And when the keys come in, always write a rule, make sure that the identifiers for the customers are unique and are combined and we don't hold them separately. That's okay as well. We tend not to do that ourselves because we want to incrementally show value to our users. So creating the concept of a Salesforce customer and allowing them to count it and query it has value for them. And we can do that quickly. And then doing Xero on its own has value for them. And we can do that quickly. And they can use that data while we then figure out the rules to create that single list. So that's one of the things we're really keen on from an agile way of working is you can chunk it down to small bits and, you know, when you get a rule that takes you some time, there's some value there already for the consumer.

Great.

Yeah. So that's the concept of a concept. I've always struggled with that one a little bit because a concept has a terminology around concept of a concept. So yeah, I was always wondering if we were going to come up with a different term, but actually the idea that we, or the notion that we have a concept that we manage We want to count and it has value. It's pretty useful. And the cool thing is I've worked with teams before where we have our analysts modeling our data for us without falling back to that age-old pattern of the grumpy enterprise data modeler who sits in his little window office for 9 months to come out with that perfect model that nobody can implement and nobody can use. So for us, it's about model fast, model well, and then change it when you need to.

Absolutely. Oh, fantastic. So that's probably concepts. So next time we'll do details. We'll dig into details, which is the second part of this equation.

Then we'll get to detail on the details.

Huh?

Detail of details.

Excellent.

All right. Well, thank you. And we'll catch up soon.

Thank you.

Let's make magic happen. And that, data magicians, was another Agile Data podcast. If you'd like to learn more on applying an agile way of working to your data and analytics, head over to agiledata.io.