AgileData #27 - 3 agile architecture things - Brian McMillian

Welcome to the Agile Data Podcast where we talk about the merging of agile and data ways of working in a simply magical way.

Welcome to the Agile Data Podcast. I'm Shane Gibson.

And I'm Brian McMillan.

Hey Brian, good to see you today and thank you for coming on the show. I think in this one we're going to talk about three, three things. We're kind of going to go on a really interesting journey. So we're going to talk about the thing called Forex Data Vault and why it's the best data modeling process in the world for now.

And everything is code. But before we rip into it, why don't you give us a bit of a background for people who don't know who you are or your journey into this agile and data world.

Thanks, Shane. My name is Brian McMillan and I am a career enterprise architect. Been practicing enterprise architecture for about 15 years or so. My area of expertise is really in the data and business intelligence space and I've worked primarily in large companies, everything from manufacturing to IT providers to the defense industry.

Excellent. And you know, enterprise architects in agile and data is often seen as an oxymoron, right? We're seen as anti pattern because.

A lot of enterprise architects tend to treat themselves as the cop at the end of the road. Right. And all the work's done and they come in and go, oh, bad work, bad work. Go back to the beginning. So it's really great to actually talk to somebody that's in the enterprise architecture space from large organizations, but has more of an agile mindset. So what's your experience with that? Do you find that a lot of the people with the, the role or the title enterprise architect is actually bound on a more waterfall type process?

Well, I think so. You know, in our enterprise architecture has been around for a long time and you know.

It has been around when, you know, as long as Agile's been around, enterprise architecture has been around. But most of the enterprise architecture in particular is geared towards large enterprise projects and products that are pretty well established or so large in scope that we think we've got so many moving parts we need to coordinate them.

That lends itself well to trying to plan out as much as you can, even though we know that that seldom works.

My experience with agile and architecture comes from way back, oh geez, probably late 2000s, 2008 or so, where I was on a team that was building a data warehouse to store server performance metrics and ticketing metrics and call center metrics for electronic data systems. And we had just started to do ticketing work. We had a very large enterprise customer that we were onboarding and their requirements were changing so fast. We basically just had to say, wait, whoa, stop.

We can't work the way you're expecting us to work and we're never going to be able to meet your needs. And that's where I got introduced to the extreme programming XP Agile. And that just lit a fire. Like, you know what, we can do this in an agile way. We can treat the architecture incrementally. And I think that's the big that, you know, if your architects that you're dealing with are not treating their work incrementally, then you have a problem. You can't start from scratch and start planning everything. You have to do a little bit of planning, do a little bit of implementation, a little bit of planning, a little bit of implementation, wind things back that don't work. So that is really important. Now, a little bit about my background. I didn't come at the architecture job from the normal route, which is, you know, I'm a computer scientist, I'm a programmer, I come from the application space. You know, I'm a systems administrator, I'm a database person, you know, I'm a dba. That's where most architects come from. I came from the business side. I've got a degree in economics on that warehouse team. I was the business guy. Did, you know, kind of everything from sales support to working with the customer to, you know, dealing with the requirements intake to doing demos for people. So I came at it from that side. This idea of product life cycle is critically important to figuring out how you should be behaving because, you know, so I'll make a controversial statement here. There's absolutely nothing wrong with Waterfall. There's absolutely nothing wrong with Agile. There's absolutely nothing wrong with somewhere in between. It's wrong if you're applying it at the wrong stage. And that's where this idea of Forex comes. And Forex is a little bit of a misnomer. Kent Beck, who was one of the original signers of the manifesto for Agile Software development.

Wrote the original test unit. Test framework has been very active in the IT community for forever. And he worked at Facebook a while ago and one of his jobs was to evangelize the way that Facebook does their work. To hear him talk, and we can put something in the notes here, some links to this, but to hear him talk about it, there was this idea, he came into Facebook and they didn't operate anywhere like he expected them to. Be operating. And he set out some time to try to learn how they were operating and what they were doing was they were operating under this idea of a product life cycle, which has been around forever. You know, product life cycle is nothing new, but he coined this idea, this idea called 3X. So product life cycle falls in three stages and it's a sigmoid curve and S curve down at the bottom, low value. Basically everything is unknown. And this is what he calls the explore phase. You're exploring, you're trying to figure out what's going to work. You're trying to figure out who your customer is, you're trying to figure out what they need. That's that explore phase and that's where all these projects start. And then once you find a customer and you start to get some traction, you start to go into expand and that's where the curve starts to ramp up exponentially. If you're lucky. Well, if you're lucky or unlucky, depends on, depends on how you feel that day.

In that expand phase, you're getting more customers, you're providing more value and things are breaking all the time. So it's constant firefighting mode. And then eventually that will tape, you'll get a handle on your product and that will taper off and it will start to level off. And that's what he calls the extract phase, where you're extracting the value out of the system. You have, you know, you've reached a comfortable level of customers, comfortable level of, you know, value that you're delivering to your customers. And then the 4x piece is one that I think is critically important, which is you need to exit from there. You need to think about how you shut things down. And that isn't part of the original 3X model. It's an addition. But it's critically important because everything, every product you build is going to have to be turned off at some point. I was thinking about what's a good example that I think the best one is a car manufacturer. They have two to three year cycles where they completely retool the car. If you bought a Honda Accord two years ago, or let's say four years ago and you were looking to buy a new one, now it's a very different car, has a lot of new features, it is not the same car. And lots of product company, lots of hard product companies, big capital intensive product companies are really good at this IT world, particularly in the enterprise world, we're not very good at this for some reason.

We have things that run forever they get ramped up, they get turned on, they work, they're stable and we are terrified to turn them off in the IT world. We need to start thinking more clearly about that.

Yeah, I like that. So the exit was the one that tweaked my interest because we've all worked for organizations that have replaced or refreshed their Data platform every five to 10 years because technology change. They're on old, on prem servers or big expensive hardware and they're moving to the cloud or before that they did the client search of a wave. But often you'll see an organization that only ever moves 80% of their workload off the old platform and the last 20%, which is incredibly hard, still lives there. Right. And so now they're maintaining two platforms and getting, you know, benefits from the new one, but they just don't want to spend the money to, to kill the old one or, you know, the number of times we don't monitor the usage of the information products we produce. So the dashboards, the reports, the APIs, you know, we just, we build it and we leave it and we let it go stagnant. We never monitor if it's been used and we never kill it. Right. We never exit those products. And yes, they're probably low effort, low cost to maintain, but there's still a cost, there's still an effort, right. The mess that they create, the dependencies. So we need to do that tidy up, right, we need to think about that exit. So the other reason, apart from this idea of, well, how do we plan to exit the things we're building at the right time? The other thing that I find really interesting is if I think about applying the four X's with teams that are kind of adopting a new agile way of working data and analytics teams, it's almost.

Kind of similar to the forming, norming, storming kind of idea, right?

Absolutely.

We're exploring ways of working, we're seeing what works for us and what doesn't. We then get good at it and we want to scale, either scale the team or scale the processes or scale what we're delivering. Then we get to the stage we want to automate it, you know, the whole DevOps data ops just to, to, you know, extract the value and, and you know, get more value than the effort we're putting in and then hopefully we don't exit. I was thinking it was like as we get rid of the team, you know, the magical software takes over and I wasn't quite sure how to apply the, the last external Team politely.

Well, I think, you know, my recommendation is so when you start, so when you start getting into expand and you start ramping things up somewhere in the middle. So let's wind it back. So think about processes in ways the team works. Let's, let's, the, the people all work differently in each of these different phases. You have people who are, let's say, more comfortable in one phase versus another.

So let's just think about like project management, you know, trying to figure out what should we be working on. So initially in Explore, you're in what I call pre agile. You're you know, business model, canvas, lean canvas, type things. You're just trying to find a customer who has something that you're interested in and then work with them. Then at some point in Later, Later Explorer, you can start to apply more agile ways of working. You know, you can start to do a teeny bit of, you know, a teeny bit of planning. Whether you know it's a sprint or however you, you do that, or if you're, or if you're lucky and you are just moving straight into just straight flow, that's great. And that will carry you up into Expand. Somewhere in Expand you're going to start to get a handle on things and you'll be able to prioritize what you need to work on. You're looking for the constraint, you're fixing constraint in the middle of that and you can start to switch over to things like critical chain project management, where you are planning out the constraint, working around the constraint and planning the work you have to do around that constraint. Everything else is much looser because you know what's expected and you know how important it is that you deliver this feature at this particular time. And whether you're on track or not, it's not so much did you hit the date, it's are you on track at that middle point? In the middle of Expand is when you should start thinking about, okay, we've got a handle on this, what would the next version of this look like? And you need to peel off some of the team and start them back in Explore and they start thinking about features and functionality that would be in the next platform and they start doing their work. So you've got one S curve that comes up and then you have another S curve that's starting in the middle and will eventually replace. And ideally what would happen is.

When the top of the extract hits the expand on the next big thing, you can start to transition your customers to the next big thing and Then that exit now becomes a matter of winding those customers down, winding that infrastructure down. And it's the same process as in explore, I'm sorry, in Expand, but in reverse. You're figuring out where your constraints are, and you're working the things that are not the constraints and turning those off, so you wind things down that way. But it is the same team. The work is just starting to change, and you need to start planning for that. And that's how you grow one version after another version after another version so that it's not a problem.

So the other way, I kind of, just as you're talking, thought about it and taking it back to that enterprise architecture. Right. And this idea of Agile architecture. And so one of the podcasts I follow that I love is Catalog and Cocktails from Data World. And one of the guests or one of the listeners on there kind of came up with a term a while ago that they use quite often I really love. And they talk about brakes on a car aren't there to stop you moving. Right. Or slow you down. Brakes are on the car actually there to make you safe when you go faster.

And so from a Agile architecture point of view, it's the idea that the architecture skill set is not there to put barriers in place and to stop and put gates. They're there to put guardrails. They actually have to be in front of the work and go, okay, if we're going to go into that area and that's new, we're exploring, or we've done some work and we're expanding and then going into extracting what are the policies and processes we need to be in place to be safe. And so from that point of view, if we think about it, as an architect, our goal should be to make ourselves redundant, not needed by the team for that piece of work that we've worked with them, we've gone. The guardrails are in, you understand, the things you need to do, the breaks you need to build to make yourself safe, to go faster and go for it. Right. And then as you said, it'll be another iteration where there's a whole lot of new unknowns. And then the architect comes in to help the team or the skills and the team are there that they can figure out what the new guardrails are. Right. So for me, that's how we should be thinking as architects when we're working with teams that are data and analytics focused and moving into that agile way of working.

Yeah. And each of those phases, again, here's where these Phases come into play. Each of those phases has different sets of guard rails with different strengths. Like, you know, I think another analogy, if you don't like the guardrail, one that I, that I happen to like, is channels and buoys. So, you know, you're, you're on a boat. You know, a big, huge boat needs to stay in the channel. If you're in a kayak, you can go wherever you want. You know, what, what's going to be able to carry the most people? Well, clearly the bigger boat. And as that value increases, one of the ways you manage the risks, you start to bring those guardrails in, those buoys in, stay within the buoys and dig the channel a little bit deeper so that you have the capabilities you need. And I think that one of the things that's really important for architects to keep in mind is you need to be loose and explore. You need to be very strategic in expanding. You know, there will be, you know, you may have some set of guardrails that you want to, you know, say a set of technologies you want to work within. You know, a set of technologies are going to work. You need to be very critical about where the constraints are that are popping up and be ready to shift if needed. You know, I, I, one of the, one of my sayings I'm rather fond of is all software sucks. Your job is to figure out where it's going to fail first and work around it. You know, you make a decision, there is, you know, it doesn't matter what product you pick or what strategy you pick. You're going or what pattern you're following. At some point, that pattern is going to break down for you, and you need to be ready to recognize it and decide what to do about it. But as you move up that curve, though, it's, it's a little bit. You need to be much more strict. You know, data governance is a huge deal. When you've got a product that's in the extract phase, everybody's counting on you. It better, you know, the data better be there. It better have a good, solid interface. You should be paying attention to it. The observability is critically important in the middle, you may have to loosen that up a little bit because things are changing so quickly.

There's a term out there that people kind of adopt around professional laziness, but for me, I tend to like positive words. So I call it the happy path. And it's this idea that if there are things a team can leverage that are fit for purpose, do the job they need it to do and are readily available, they're going to adopt it. Right. It's a happy path. It's like, well, that works. I don't need to worry about it. I can go solve some of the problems that haven't been solved. So again, from an agile architecture point of view, what we want to do is help them build these things that, as you said, they're buoys that keep them in their lane, their guardrails, and the teams will adopt it because it's easier for them. It makes sense. They're not going to go and reinvent it. We'll rephrase it. Most people won't go and reinvent it. I have worked with the developers whose first job is they typically say it doesn't matter what the code looks like, I'm going to rewrite 70% of it because that's the mentality. But we just vote them off the island. This idea that if things are in place and they're reusable, people will reuse them because it just makes sense and everybody is a good person and they will do what's seen as right for the organization and their teams. Kind of moving on from that one because I was tempted to deep dive into data governance. A whole podcast on its own, which I'm happy to have because for me, data governance is definitely not. It's before the explore stage. Right. We're still at decisions by committees and, and, and I hate that. But, but before we deep. Let's not deep dive in data governance, let's move on to Data Vault, something I love. Right. So great for me.

Yeah, there's been a wave of modeling techniques that have come out over the years. For me, Data Vault is currently the best technique I can find data for a whole raft of reasons. It has some problems and, you know, hopefully somebody will come out with some new modeling techniques that give us the flexibility of Data Vault and solve some of the problems that it presents. But from your point of view, you know, data Vault, why, why not? What's your view?

Data vault modeling technique, I just love, I think it solves a lot of really difficult problems. And the first one is that it gives you a framework to think about the data problem. You have like the life cycle of your data from acquiring the data to actually making it fully functional and usable to whoever the end user is. So it provides a bunch of natural layers.

To cut the problem, to cut the cognitive load for a developer down to bite sized pieces. It does this a couple of ways. The first one is and the thing, if you're familiar with data vault, probably the thing that you think of first are the hubs, links and satellites. The three basic table types.

The hubs represent your entities. Those are your customers, products.

Sites, whatever. The entities that are important to your business, they get expressed in a hub. And it's every customer gets a record in the hub. And you can look at the hub table and you can count star and see how many customers you have.

Which is really useful metadata about the system. You know, if you come into a data vault, first thing you do is you look at all the hub tables. Okay, now I know the domains that this, that comprise this warehouse.

Then you have the satellite going on the other end, you have the satellite tables which are the details and that varies. You know each domain is going to be different and it's not very prescriptive at all. Hub table structure is very prescriptive. Satellite table is not, except for how you link and a couple of other.

Neat little fields that they require you have. And then the link table, it just a place for you to express the relationships.

So one, that happens all the time, those customers, I've got two customer records, but they're the same customer. Well, I can express that same as relationship in a link table so that when I go to query the information, I can say, oh, I want the standardized customer name from the link table to join up with the data, the detailed attributes that are in my satellite table. And that's the basic structure. And one of the great things for people who are developing in the data vault model or consuming the data vault model is they only have to learn those three. If you can learn what the capabilities are and the purposes are of those three tables, you're good to go. It's very easy for you to maneuver around the warehouse. Then you may have some extended joins to do, which is a problem with the data vault. It's not like a fully third normal form system where you may have lots of data.

Lots of tables that you have to join across. You basically have three basic tables, but you get the advantage of combining those tables up like you had a more denormalized system. So you can bridge different satellite tables across the link table or the hub tables, but there are lots of joins. Which is a valid complaint about the data vault is that I've got lots of joins, but if you're doing anything complicated, you're probably going to have lots of joins anyways. And the traditional star schema helps to eliminate that. And if you've only interfaced with a Star Schema Warehouse.

Data vault looks very intimidating and complicated because there are lots of tables. But.

If you can't find the data you want in your fact table and your dimension tables in a traditional star schema warehouse, you have to go looking elsewhere. And it may involve rebuilding, you know, new fact tables or rebuilding the fact table that you currently have. And those kind of changes are really disruptive to the warehouse. You know, modifying and migrating a fact table is a big undertaking. And if you've got it broken up and a little bit more denormalized, it's much easier to make these changes. So you've got these tables where you've got these three table types. Very easy to learn, very easy to get your head wrapped around.

And they're very flexible. Doesn't really matter what kind of data you have. So that's one set of layers or cross cuts. The next one that I think is even more valuable is that you have the data vault is split into these layers. Now whether they are schemas, whether you know, whether they're schemas or whether they are.

Individual, I mean frankly, they could be different databases, they could be different database engines, whatever is appropriate. You know, your raw and staging area could be in a file based system. You know, whether it's a, you know, whether it's a file share or something like hdfs. Data vault doesn't care. Data vault just says you need to have a staging area and then you need to have an area called your RAW vault where you're storing history of all those loads. Data comes in, gets loaded in the table and this append only format so that you have a new record. You know, the, you have a brand new record, it gets entered into the table, that record gets updated, it is a new record. You don't update the current one you have. And that is really powerful, particularly when you're exploring your data sets because you don't really know. This gets everybody, you know, everybody who spent any time working with data knows that there are things you just don't know until too late usually. And being able to have a raw satellite table where you're stacking up your loads incrementally and you're just appending the change. Data sets lets you go back and it lets you unwind the data. So what did this data look like last month? Which is really powerful from a reporting standpoint, I can tell you what the data set looked like and it doesn't look like that anymore. And here are the three changes that happened over the last month.

For troubleshooting it's also valuable because you can see what changes have occurred. But that append only model is also very easy to work with. The type of SQL that you have to write to load that table is really simple. You don't have to go look up keys, which is something else. In Data Vault that's really that they made the difference between Data Vault 1 and Data Vault 2. One of the big changes was they had to support NoSQL databases, which means they went from sequence keys to hash keys. So figure out what your key is, hash that value, figure out what your business key is, hash that value. And then that means that you can now load everything in parallel. You can load your hub and your SAT in parallel, and because you're calculating on load what the key is, they can just go in seamlessly, which speeds up your loads significantly. And it makes loading the database much easier. And it makes it easier for people who are not that comfortable doing.

The DBA ETL work, makes it much easier for them to get their head wrapped around what they have to do. Oh, I need to load the table, I need to follow these simple patterns and I can load my own table. Which brings us to the next layer. After you've got that RAW data, that RAW table, or that RAW schema, you've got a business schema, which is a new, new thing that was introduced in data vault 2.0. And that business schema is where you apply the business logic in the raw. In the RAW schema, your tables, whether they're Hubs, links or SATs, you don't apply any business logic. It's the data you got from the source as is.

Unmodified, but that's never really what you want. So make those transformations from RAW into the business schema and apply whatever business logic you need. And this is where the tables kind of balloon, but it's worth it.

You can have a hub, let's say you're a manufacturer and you have serialized and unserialized parts. Every analyst who's querying that data has to make a common table expression or put a view together or something to filter out all of the parts that are serialized for a particular. Let's say they're doing some quality analysis and they're only interested in the serialized parts because they can tell.

It'S a much more fine grained view of the data.

And it would be much easier if there was just a table called hub serialized parts or Hub parts serialized.001 first version of that table, Hub serialized.002 that's a different version, there's something changed in the business logic. It's like, think of it like an API where you've got version one of the API and version two of the API and version three of the API.

Yep.

Now, you have multiple tables, but they provide different business logic. And it cuts down. It reduces the risk of you making a change that someone didn't anticipate. If you're consuming the table that you're consuming, you know the first version of the table and there's a second version, your things don't break, you get to make the choice. And this is something that we, as data architects, we don't do a very good job of.

We're really good at planning our migrations, but we don't think about what would the simplest way be? Just duplicate the data. If storage is really cheap, duplicate the data. Version one of the API, version two of the API. Oh, maybe we should treat all of our tables as APIs or as individual data products and just start thinking about the lifecycle of individual tables. That gets really interesting. Then those business tables, you got lots of business tables. Right. But they all have.

Different business logic applied to them, and it makes it much easier for an analyst to come in and go, well, I want.

This flavor of the data and this flavor of the data. And I know that the business logic that's applied in those tables.

Is very clear, understandable, and the teams that are managing those tables stand behind them.

And I don't have to go make some custom code with some weird query that may be above my skill level. All I have to do is just join the tables together and things get filtered out.

I think you're right there in that as technologists, we love complexity. We love to make things of beauty. We love to take a problem and just beat the snot out of it with the most beautiful solution we've ever seen. Sometimes a simple solution would do it. And I think that's the same with Data Vault. For me, data Vault is a set of patterns. So it's a set of patterns that have value because they are the best or they are a set of good patterns that we can use.

The data vault community has fallen into the same problem that I've seen with the Scrum community is that with Scrum, there are some people that say, here's the Scrum Guide, and if you don't follow the Scrum Guide to the letter of the Scrum Guide, then you're not doing Scrum. And I don't agree with that. Well, I agree you're not doing scrum, but I don't care, right? I'm saying there's a bunch of patterns embedded within the scrum guide. They all have massive value. Use them if it fits you. If it doesn't, don't use them. And same with vault, right? So you know there are a bunch of immutable patterns involved. They are, you have a hub, you have a set, you have a link. That's about it, right. Everything else really is a choice as far as I'm concerned. And the thing I like about, thing I like about vault, and you kind of articulated it, there are hub sets and links. There are only three types of data. There are really only three, six patterns. Create hub load, hub. Create set, load, set. Create link, load link. Right. It's a really simple coding pattern. Now there's lots of complexity once you start throwing it at real data. We talk about bridge tables and same as and her uncle links. And there's lots of other things that we sometimes do to, to, to make it easier when we have complex data, but they're just choices. And so what happens is we get into these religious arguments. So, so in our product Agile data, under the covers, we use Vol ish. Right. We don't use the term hub sets and links in the product because what we found is our, our audience is an analyst. Right. It's a, it's a data literate person that's on the business side, not the, not the engineering or the technical side. And so we found that those words scared them. So we call them concepts. Yeah. Which is a hub. Right. I have a concept of a customer and their business concepts. They can go out and say, I can see a customer, I can see a product, I can see a store. We have detail about those concepts. It's a set. The customer has a name, the customer has an age, the customer has a gender, the store has a name, the product has a sku. And then we talk about events rather than links. And we do that because we say, if you looked at your organization, what business events or core business process can you see? Oh, I can see customer ordering a product. Cool. Well, that's a link, right? You know, these three concepts or three hubs that are embedded. So for me that's really valuable. Right. Those patterns, they're just so simple and they're so flexible.

We deviate a little bit from the DV2.0 mantra. So we actually use a history layer that isn't vault, that's not vaulted. Typically you call it a persistent staging area. And so we don't vault that data, right? We don't break it out into hubs and sets. And there's a whole lot of reasons we don't apply that pattern. But we're okay with that, right? That's just a choice we made. It has cost and consequences.

And the same thing is we spent a large amount of time on that last layer, what we call the consume layer, or people typically call presentation. We said, okay, so we don't want our analysts to have to understand how to join all these tables together. What we can do is we can create views or tables that make them consumable. We can denormalize a link out to its hubs and denormalize that out again to include all its sets and give them one big, wide, ugly table. Now, performance is an issue sometimes, but with the cloud databases we have now, they pretty much eat anything up to a terabyte without even blinking when you denormalize it. It's just a thing of beauty. So for me, data vault is a pattern. There are a small number of immutable patterns that if you're not doing that, then you aren't doing data vault, the modeling technique. And then there's a whole lot of ways of working patterns that have come out in 2.0. And from my point of view, if they have value, adopt them. If they don't, find something else that works for you. So.

The thing that people often say is I prefer star schemas over data vault because it's easier for the users to query. And I'm like, great, do a data vault and then put a star schema on top of it.

Yeah.

And that's what the standard says you've got, you know, so you've got your business vault and then you have your infovault. And in the info vault you'll have big wide tables, if that's appropriate. You'll have traditional, you know, dimensional star schema data marts built. Whatever's appropriate. And that's really important to know. And I, I think that that's in the layering in the data vault standard. That's one of the things that's really important to break out. You've got, you know, your raw persistent staging area. You have which question for you on the getting to your history layer. Are you doing it? You're, you're doing an append only model there?

Yeah, yeah. So it's upset only. Okay, yeah, so we, yeah, so we, we mirror the source system tables. That's the structure of our history layer. We do Upsets only. Right? So we are effectively inserting new change records as we see them.

And we're just not vaulting it. Right? We're not breaking that table structure out into a hub and a set like, like the doctrine says.

Right.

But we, we, we're not doing that for a reason.

I will agree with you. What I'll tell you, I've done the same thing. And in fact, in the, in the.

Book.

In the raw layer, that first table that gets built, staged and built, that first sat table, there are no hubs and link tables built because frankly, I don't see the need to. In most cases, what you want to store is you want to store the history. To go back to now, it probably makes sense to break out hubs and links in the business layer because you're trying to apply some business value and there may be some value in saying, oh, hey, I have to alias, I have to alias this entity so that we get the right product names and that it's, it is optional. And I don't, I see almost no use, particularly in link tables. In the RAW layer, I don't see a use for that.

So I spent an inordinate amount of my time a while ago trying to understand RAW versus Business. And I don't see myself as a dummy. I'm not highly technical, but I can read other people's code, right? But I think I have an innate skill to see patterns. And in that, when I see a pattern, what I focus on is classifying what makes the pattern and what makes an anti pattern for it, when that pattern should be used, when it shouldn't be used. And RAW versus business killed me. It was like, well, is raw its own database and business its own database? And now we're duplicating data. And if it is, I don't care. That's cool. If we store it twice and it has value, I don't mind. But that's not true because actually what the answer is, well, business is just another set of hub sets and links next to raw, right? That has business logic. So I'm like, okay, so what's business logic? You know, am I if I'm conforming customer from two different source systems? Yeah, that's, that's a business thing, right? That's a, that's a high level of complexity in those rules where, you know, we're making stuff up, we're inferring things that aren't seen in the data. So that's a business rule. But if I'm flagging, you know, apart from, you know, to say, in stock, out of stock. Is that a raw thing? Is it a business thing? Well, I'm inferring it. So it's a business thing. And I got to the stage where I went, I don't actually care. Right. All I care when I'm working with a team is determine what code you're writing or using and what it's doing and then determine how you identify the pattern of that code and then how you want to classify it. And if you want to classify that these type of changes are raw changes and these type of changes are business changes. Good. And as long as you stick to that pattern. So for example, if you say that conforming customer is a business pattern, it's business code, then I should never see conforming of an entity or an identity or a concept in my RAW layer. Right. Because now you've broken the pattern. Now you're confusing me. If you say.

Inferred flags, there are business functions, right. They're part of the business rule. So they should only be in the business layer. Right. I should never see them in my RAW layer because now you're confusing me again. So again it's set the pattern, write it down, agree it with everybody and follow the bloody things. And so that's where I got to. So I've thrown away the raw versus business definition for Vault. For me, it confuses people. It has little value. We should focus on the type of changing we're making, the size of that change and then how we describe that as a pattern and then when we see it as a pattern in terms of X's, how do we move that from an explore pattern to an extract pattern? How do we automate that pattern? Because that's what the beauty of Vault is. As I said, six pieces of code create or load times three. The ability to automate that is amazing. That's where the value and the data vault pattern is.

Yeah. And also to know you don't have to apply like you've been saying. I think one of the things that you've been saying is you don't have to always use all the objects if you've got the consume layer. And that needs a different structure. If a hub link and SAT layer, it probably isn't useful at the consume layer, it's too complicated. So you've already made the decision. So I think, you know, Pat, talking about the four X's again, you know, the.

If you're an explorer, stage the data, start collecting history and then start building out your info layer that consume Layer, go straight. Don't do anything in the middle. Just do whatever it takes to get the data visible to your customer so that you can start making judgments about the quality of the data you have and actually what data you really have. Then you may fall back and do some intermediate modeling, which may have hubs, links and sats. Then you go refactor the info layer, that consume layer. And you have to have that flexibility.

And that's part of getting that agility in your data architecture is no, I've got these structures, these buoys that I can steer around, these guardrails I have to sit in. And what's appropriate for me to do right now, I don't have to use all the things, I don't have to do them in this order. And once you break free of that, then it becomes, and this is the thing that is really spectacularly productive about Data Vault is if you break free from the sequencing of the objects, you get a lot of flexibility and a lot of agility.

You can change your mind.

Yeah. Its ability to absorb change is amazing. And so that's why I go back to the data vault modeling pattern, right? The pattern of how to model using Data Vault and store that data and load and change that data. It's amazing, right? It is a thing of beauty. Everything else that's been put around it with DV2, I think it degrades the value of the model. So I'll give you an example. You know, DV2 says big modeling up front, right? It says if you don't understand the core business concepts and you don't model the organization, you get a source specific vault. And that's bad, right? And I call bullshit on that, right?

Because I must have, I must have missed that chapter.

It's in the conversations, right?

No, no, it absolutely, it absolutely is. But in my experience, my, my first experience building a data vault, you know, from scratch with a team of people who are not data, not database developers, went through this pattern, get the best data, you know, get the data the company's using, start putting that in, you know, start putting that in the database and start looking at it. We found out very quickly that it wasn't going to work, that the data was bad. The counts of parts that everybody in the company was using were wrong because the system that they were getting the data from, they had implemented some.

Incorrect logic and they were miscounting things.

And you know, that's, that's just the way that it is. So we scrapped everything. Then we got a new set of data that we thought was much better and we found new problems. We scrapped probably half of that and then we got the third set of data that solved the problems that we had and we went forward with that. And.

That kind of iterative development is really conducive to.

Forget the pattern, the modeling pattern that you're using.

You have to start thinking about it incrementally and you can't get attached to. If you do a whole lot of upfront modeling, whole front of upfront design, you are going to be sorely disappointed when it doesn't work. And the, the goal is get to that point as quickly as you can. You know, make it through the crappy chasm of doom where that you make that inflection point from, from explore into expand. That's really painful. And you learn lots of things in that area and it never is going to get better, but it will get better. You know, you might need to give it a few months. You need to be prepared to just jettison the work that you had done and no amount of planning will prepare you for that.

And I agree. Right. So you make the point earlier that as architects, when we design the beautiful architecture diagram, it's our baby and we don't want that baby to get hurt. Right. So it's like, oh, don't you deviate from my diagram. But we've got to get rid of that. Right? It's a guess. As soon as we put it into the first engagement, it's going to change. Same with our models. Right. Sitting for six months in a room doing the perfect model is not an agile mindset. It's not an agile data mindset. Our customers, we want to get the data in front of our customers early to get value. If we can go and say we have a concept of a customer. So populate that hub from a source system. Salesforce, that's where the first master of customer comes from. Cool. Grab that. You know, get a hub for Salesforce with a customer ID in the set, add the customer name, whack that on a report and say to them, here we go. You can now count customers. As at today you had this many. As at last week, you had that many. That is value. Right. We underestimate how much value that really simple piece of information is. To have a single answer to the same question of how many customers do we have? That's massive value. And then we can incrementally build out. Now, should we stay with a source hub? No. Right. We should then bring in another system that has customer we should conform those to say, well, we've got two different definitions. How do we get a single answer to the question of how many customers do we have across the entire organization? And that's a harder problem. But we build that next, right? We deliver that value first and then expand on the value. So for me, Vault is cool, right? It's a cool modeling pattern. It's simple, it enables us to change, which is what it's all about. But it's just a set of patterns we can adopt the way we want. And very few of those patterns involved are immutable. Hub sets and links are they. My co founder, you know, he, he comes from a more dimensional background and, and he's, he's particularly agile from his patterns. And he keeps saying to me, so why can't we just have sets and no hubs? And I'm like, here's the five reasons we have a hub, right? And it's like, no, there is a set of immutable patterns, right? We will never have sets and no hubs for a whole raft of reasons. But running our time. So let's go on to the last one. Everything is code.

This is a good transition. Okay, so one of the prop. So, yeah, let's do, let's do it this way. So we've been talking about doing that upfront planning and breaking free of that upfront planning process. I will contend that a lot of that upfront planning that people do is forced upon them by the tools they are using.

If you came from a traditional enterprise data warehousing background you were using, you know, let's say your Oracle was your data warehouse. Oracle was your data warehouse platform. Informatica was your ETL tool. And what does Informatica have going for it? It's got a nice drag and drop interface where you can grab an object, configure some parameters in a nice GUI, and then connect that to some other thing and then connect that to some other thing and build out your pipeline visually.

What about your data model? Well, how did you. Before you do the ETL work, you need to have a data model in place, right? Because Oracle's not going to work. It needs something to connect to. It needs to get some metadata from your database in order to be able to work the way that it needs to work. So that means you need to have a data modeler. That data modeler is going to use what they're going to do the, er, diagrams in what tool.

AI Sparks.

Yeah, that works because we're picking the.

Big, ugly, hard to Use, Right. So let's say the a sparks and.

Your data modeler is going to go in and they're going to model the data and.

They'Re going to work with it. You know, how are they going to know what to do? They're going to work with the business analyst. So you start to build out this waterfall pattern and a lot of that is.

Reinforced by the tools. At best, you know, you have to model the database before you can use the ETL tool. Well, that puts you in a waterfall pattern that makes it difficult for you to make changes.

And if you were to take a different. So we're going to talk about treating everything as code. So what would that model look like if you were going to treat those steps as code? So we're going to model the database. We need to get some data in the database. We're going to model that. Well, if we're going to, you know, in your example, if we're going to copy the source tables over from the source system into our staging area and we're going to do an append, only type, load, insert, only load. Like you said, the SQL code to do that is pretty easy. Your bulk loader for your database may create the table for you automatically. You may just point it at the data set and say, bulk, load this data and it may create the tables for you. You shoot. It may even create the database for you depending on the tool you have.

So that becomes much easier. Now, that process didn't require you to do any modeling because the data reflects the table reflects the data that you loaded. Now you have to do that load script. If it's a bulk insert, it's just going to naturally just append the data. And then you may have to do some cleaning up afterwards. Or now you want to move it to the next table. Well, depending on the type of movement, type of pattern, you're going to have, you know, load your sat, load your hub, whatever it is. That's another piece of SQL code. And it's pretty clear. And so now I need to write the SQL to create a table. I need to write the SQL to load the table. I'm the ETL developer. It's pretty easy code to write, even if I'm not very experienced and comfortable with it. I can look at the last table that somebody else, you know, that the person who was more skilled than me did, and I can copy that pattern. It's pretty straightforward. I mean, that's a revelation to someone who's only queried the databases, you know, someone who's got good SQL querying skills. It is not a heavy lift to learn how to create a table and how to insert data into a table. It's really easy to do and once you know how to do that, then that gives you a lot of power to do things like create your own tables. And it's very clear what's going on. And it doesn't require a third party tool. It's code. Now the advantage to having that now you're code based, your ETL is in code. At least the tasks you're executing are in code. And creating the database tables, the DDL statements are all in code.

If you adopt the pattern that we don't do table migrations. One of the features that the data modeling tools have going for them is that they're very good at automating the table migration process. They take the data from table one, they put it in a temp table, they restructure that data into table 2 into 1B and then they load the data in there. You've migrated the table to the new structure. That's one reason why you have these ETL tools. But if you were to adopt the pattern that just says I have version one of the table and version two of the table, it's pretty straightforward. First thing I've got to do is move the data over from one to two, that transformation. But that's a one time job. And then I've got a job to load table one, job to load table two. Again, it's all code now because it's all code. And I didn't need a GUI tool to manage that. I can now put this in source code control.

Makes it easy for me to share. You know, try sharing your work. Try sharing your work if you're an Informatica user. It's not easy. It's very easy to step over somebody else's project.

But now you've got it in source code control, you can start to adopt software development practices like version control and having collaborative development. Now, whether you do trunk based or, you know, you go through the whole git flow, you know, print request, please use trunk for data work.

Please put tests in place, but those are all code. And you can start to now leverage a lot of the software development tooling that's been in place for decades. And you're not spending money on these visual tools, you're not locked into those tools. It's much easier to do collaborative development.

And it's much easier to make incremental changes and move a lot faster. And that's a big shift.

Yeah. So I'm going to disagree with you strongly and agree with you strongly in the same sentence.

Let's see if I can guess what it's going to be. Go ahead.

So everything is code, right? So I agree that everything we should do should follow a pattern that we follow when something is code. Yep. So you've described some patterns, you've described. We should version things, right? We have a set of things. Let's think of it as code and they're running and we want to make a change, right? We should version those changes so we can go back, we can go forward, we can see what the difference is. That's a pattern. We should be able to collaborate, right? We should be able to see what our colleagues and our peers are doing. We should be able to work together on the same thing. We should be able to potentially split off and do parts of it and bring it back together. Although that is a high patent with high complexity. Again, I'm not technical, right? But I can recode. I hate git merges, right. I hate merge conflicts because I look at it and I go, why did you let two of us hack the same thing and now make it my problem to fix it? Right? Give me a tool that makes my life easy, not harder. But that collaboration, right? That it's not locked away on a developer's desktop and nobody can see it, right? That that's valuable testing. We learned to test code for many years, but we hardly ever test our data, right? So again, testing is a pattern. Automated testing, regression testing, validation, assertion based testing, those kind of things are important, right? And we should do it when we do it with data, we should treat data as if it was code. So I agree with all that. What I don't agree with is that we can't have a low code interface that provides that. What we've got is a bunch of legacy products that haven't actually given us the rigor and all the tools and patterns that we need to do that. That's where I go disagree. Because at the moment we're stuck in a world where only writing code is the way we get those patterns and those features that we need. Version controlling, collaboration, testing, right? You know, the DBT world, I don't think everybody should have to write code, right? But everybody should get those features right when they're working with data and when they're working with code. So agree strongly in all the patterns, don't agree with the jump to actually it requires us to write code. But that's Just the current state. Right. But we're going to change that with Agile Data. But anyway, yeah, that.

But I agree with that. There is a point where you need, particularly if you're working from the front end back, that having the low code, no code type of.

Use case is really powerful. I mean, there's a reason that Excel, and we'll just stick with Excel before we go to things like Tableau, there's a reason that Excel is the number one BI tool tool in the world because it strikes a good balance between giving you the power to do things in code like functions and you know, particularly if you think about the things like the new Power Power BI features and the all of the data features that are now embedded into at least the Windows version of Excel give you the very nice low code environment.

To work with that's really appropriate for a lot of use cases.

But if we look at our vistools, we look at tableaus and Power Bis and the clicks of the world, the ones that were the previous wave of visualization tools, we don't get a lot of the features that we should. If we're treating visualization as code, we don't have to write code, but we should treat it as code. So it's hard to version.

Well, you can get the xml, you can get the XML format.

Because those tools don't have the features we want. Right. We are forced to extract the code to do what we want. So let's look at them. Can we version them? Yeah, a little bit. Right. It's got better. Right. We have some ways of getting versions of those visitors and rolling forward and back. Yeah, but it's not great. Right. Can we collaborate on them? Yes, typically now, well, apart from Power BI and it's desktop thing, we've got tools that are server based, browser based. With some collaboration. Right. We can start working together. Can we test them? Hell no. Show me a VIS tool that allows me to write a test that says the number in that KPI widget is actually right. There's no testing frameworks for those VIS tools so.

Well, you could write a report that tests the data that underlies that dashboard.

I can write a test, the data goes that going into the dashboards. Right. But I can't write a test to confirm the number on that dashboard at a point in time is right. That nobody's applied a filter or inadvertently added another.

Calculation to skew it again. If we say that the pattern is everything is code and the rigor that we bring to the way we develop software, you know the DevOps, all that stuff should be brought to the data world. Then I agree with you. Right. And at the moment I think we're in the, in the chasm again. Right. We're going to jump over and a bunch of new tools are going to turn up that adopt that. But again, just, you know, my view is it doesn't actually have to force us to write code, it just has to give us all the features that we get when we do write code. So.

So yeah, if we can get there, that would be fantastic. There is, there is, yeah. Like I said, you can envision a world where you get both, the best of both. And I think that there are a lot of companies, if you look at, you know, some of the modern data stack companies, there are quite a few that are moving that way. Particularly in the orchestration piece. I see this a lot, you know that you've got the, you've got the ability to build your data pipelines out visually and there's data behind, there's, there's code behind it, there's metadata behind it to do things. But that is a lot of work and we've got a long way to go before we get.

Yeah, and, and look, just looking at time, you know, let's close it out so we can loop back to this idea of Forex and, and architects. Because what the data stack's given us is a whole lot of new cool capabilities.

Right.

It's great. But what it's also given us is an infinite amount of new complexity. You know, in the old days we used to buy one or two tools to rule them all. Now we have to take 15 to 20 different products, moving parts, open source software as a service platform as service, and cobble them together to give us all the patterns that we need. So, you know, it's kind of a great time and a bad time for data data and platform architects and the data analytics space because our world's become more complex. But hey, what do we love doing? We love solving complex problems.

Yeah. Ideally, let's try and solve them with simplicity. And more importantly, think about the four X's, right? We're at the explore stage. That's cool. But how are we going to exit some of this stuff when the next wave of technology comes? And some of the things we're building in five to 10 years have to be exited and replaced with things that have more value. Think about it, right? How tightly coupled, how interchangeable is the things we're designing, because new stuff will turn up that has massive value and we need to be able to exit the thing at the end of its life cycle and introduce the new thing without rebuilding everything from scratch. So, you know, Forex, for me, it's. It's a nice set of patents.

Great, thanks. Yeah, I was. It. It's really changed my whole view on how I think about problems. And it's the first thing, when I talk to someone, that's the first thing I introduce. Like you have. Before we're going to talk anymore, we got to talk about this 3x4x thing. And then other things will make sense from there, because that's, that's the perspective that I'm bringing.

Yeah. And it gives us a shared language again, like data vault. There's only three things you can talk about, so there's only a certain amount of language you can use. Right. So that shared language means we're on the same page to begin with, and then we deal with all the complexity that has uncertainty. So using the 4X as a baseline for a shared language, where are we and which one of the X's are we working on right now is a great way of thinking. So, look, thanks for your time. It's been awesome.

Thank you.

And we'll catch everybody later.

Thank you. I appreciate the opportunity.

That Data Magicians was another agile data podcast. If you'd like to learn more on applying an agile way of working to your data and analytics, head over to agiledata IO.