Shane Gibson - Making Data Modeling Accessible

Shane, what's up, man?

Hey, thank you for having me.

It's anytime, anytime. Yeah. Where are you dialing in from?

I'm from a little place called Pikokariki, which is in New Zealand. So for the audience out there that can find a map that has New Zealand on it, we're at the bottom ass end of the world. There's two islands, top island called north island, funny enough, and a bottom island called South Island. I'm at the bottom of the north island on the left on the coast there.

That is a very New Zealand sounding town as well. Yeah, yeah, yeah.

We have some cool names.

Yeah, yeah, yeah. I can't pronounce any of them. They just sound like.

But they sound cool. So.

Yeah, it's the. The one that's really interesting is the. The ones that start with WH because they're all, you know, we get people coming to visit and so we have a town called Whakatane because the WH is Oeconif. Yeah. So it's kind of cool.

Oh, that's hilarious, huh? Guess I'll have to make it there just so I can go on a tour of strangely named towns, so. That's awesome, man. Yeah. And yeah, you are kind of, I guess, as they say, at the ass end of the world too, which I think gives you a unique perspective. Right. You're not, you're not in the thick of things like, you know, as we are in America. So I think it gives you maybe a different, different vantage point to view the world. Right?

Yeah, I think it's. We have some benefits and we have some downsides. So one of the benefits is, is we typically we get to pick and choose the waves or the, the, you know, the buzzwashing that happens that turns up in New Zealand because, you know, we adopt patterns from the US as much as we do as Europe.

The second thing I think that's to our benefit is we're a really small country and because of that we have to be quite innovative. So we'll tend to pick and choose and test things probably a lot more than we see overseas. Downside being is just our critical mass.

For me to go to a conference and actually talk to people, it's typically somewhere between 24 hours and 36 hours of traveling. So that's a little bit onerous. Right. So we don't tend to meet and greet people as much as the rest of the world. However, you know, Covid was a blessing for us in terms of everybody going remote and remote conferences has been a big Change to me and way of interacting with people like yourself.

Yeah, it's super cool. I don't think we would have met if it weren't for Slack and social media and Covid.

But thank goodness for that. But it's interesting too because I see you on the data quality camp, Slack often.

I would say chat. We chime in on a lot of the same threads, I think have a lot of similar viewpoints, which is pretty cool and also kind of echo chambery, but it is what it is. So.

That'S what happens when you're super cool, I guess. But yeah, one thing I think that we wanted to talk about too is data modeling. Right. I mean that's kind of the topic du jour.

But I know you have a lot of strong thoughts on this.

Yeah. So I've been in the data domain for just over 30 years now. Right. And I can kind of break my career up into a bunch of decades. And I'm not technical. Right. The way I kind of talk about it is, you know, I can't code one, code, don't code. And data modeling's always been one of those things that's annoyed me. And it annoyed me because it became a dark art. Right. Whenever we were doing a project back in my consulting days.

Sometimes it just worked the modeling process and it was awesome. And sometimes it was an epic failure, sometimes it was done quickly, sometimes it was done slowly. And it seemed like there was this priesthood of data modelers that would go and sit on their own in a dark room for six months with a whiteboard and come out with this enterprise canonical model and even that word canonical, right? I mean, how unfriendly is that for the whole world? You know, we have a canonical model. And so I spent quite a bit of my career focusing on actually how do we make, you know, what patterns are out there to make data modeling accessible to people who aren't data modelers. And so that's always been one of my focuses is that idea of how do you take something that's complex and then how do you help people do it, but do it with rigor, do it with patterns. I tend to get opinionated every now and again. So this whole idea of an analytics engineer.

We'Re moving to decentralization and self service, which is great. That's what we want, is more people to be able to do the work. But we've gone to the wild west again, which is basically just go write some code and we know what happens. We know in the first six months it's Fine, you just write some code, you're fast, you're agile, two years later you've got 3,000.

Blobs of code and chaos rains. As part of that. I've become really fixated with language. So when DBT talks about a model which is effectively just a blob of code, that actually makes me quite angry because we're degrading the, the practice that is modeling. And modeling is important. So it's about balance for me, it's about not spending six months to a year of one person building this enterprise canonical model that nobody will ever implement. But it's also not about, you know, five people hacking code with no thought for the design of that data up front. And so balance is hard, but balance is important.

It's super hard. Yeah. And writing a lot about that too. Sort of the spectrum of formal and strict versus relaxed and.

Fast, maybe. But I could also make an argument that if you're formal and strict and you ingrain those patterns, you actually move faster at a certain point too.

If you can get over the initial hump of like you say, the canonical model and all the ceremony that goes in behind that kind of stuff. I think that there's definitely a lot of trade offs, for sure.

Yeah. And I'd be interested in your view because often we see a data model described as thing is a thing of a thing of a thing, which is the most flexible data model in the world. Right. You can actually put any data into those four things. But to understand it as a business user, it's a nightmare. Even as a developer or an engineer, to understand the relationship between this typing of things and then subtyping of thing of thing is incredibly complex. Right. So then we're saying, well, as a human, you have to have this massive amount of cognition to be able to look at that, that pattern and then bring it back into a business pattern, describe it. So yeah, I mean, I'll be really interested when you publish to see how you describe formal versus informal. Right. Because that's a really interesting categorization for modeling.

Yeah, I mean, I'm kind of looking at data modeling right now as a way of just organizing and standardizing data to facilitate useful and believable information for both humans and machines. Right. Because it's not just humans that are using the outputs of these models anymore, it's also machines. And so, you know, but the way I'm kind of describing it too though, it's supposed to be cross sectional, so it's not just for analytics, it's also for software. Engineers and machine learning engineers and so forth. So you know, when you try and think of a model from that perspective too, you know, we can talk about this in a bit, but you definitely want to separate that kind of the higher level concerns for conceptual and logical modeling versus the almost infinitude of physical modeling now, because it used to be that modeling at a physical layer was just for databases. But data scientists, they might be using a notebook, for example, and they might be hacking on files, they might be hacking on this, that or the other thing. Or a machine learning model may be made from text right now. So if you're using generative AI and transformers, it's like, well, that's not really a database, so like what is it? But the data has to still be useful and clean. So. And the output of that is a model too. So it's like.

It'S a lot more complicated, I think, than it used to be too.

Yeah, I think the use cases are far more complicated. And before we used to ignore them, right. We used to think about, you know, the modeling we care about is a dimensional model star schema in a warehouse. Because we want to hook up Micro strategy, right? And in that case we'd do a snow snowflake, because micro strategy prefers snowflake schemas over star schemas. So for me it's really interesting around that physical modeling because in the past it was always based around constraints. We would model based on the thing that's going to consume it. Like I said, MicroStrategy would do snowflakes. Power BI. There's this whole argument around DirectQuery versus indirect query. Do we cache it in Power BI or To? Not so we. And that Power BI only likes dimensional models. I haven't tested it. I think it's bullshit, but I actually haven't tested it. So I can't say whether I'm right or wrong. But we tended to model around those constraints we would have on premise databases that were really expensive, the oracles of the world. So we would model in a way that reduced the amount of data we stored because otherwise we'd blow all the cpu. A lot of those constraints have gone. So what we now need to be modeling around is around usage. And your example of a ML engineer is really interesting because if we think about what do they typically want? They typically want a big wide denormalized table with a bunch of columns that are flags, ones or zeros. And in the past it used to be, well, here's our star schema. You go and then create yet Another model on top of that, which gave you that format that you wanted. I think we're at the stage now where actually from a physical modeling point of view, we can start to be lazy. Right. Because we have the benefit of scale at a cost that we haven't had before. So we should be able to deploy a star schema, a data vault model, an activity schema and a feature store as physical representations of the same data. With one click, that's where we should be. Then the question comes back to not the physical structure of the data, but the language. What language do you speak if you're a machine learning engineer? I speak a language of wide tables with lots of rows of raw data that's got feature flags.

For me, data modeling is actually going to become more and more about shared language than the technical implementation that we focused on for the last couple of generations.

It's a really interesting point about the shared language and thought about that actually. I like that. Yeah. Because you're absolutely right. There's just too many. If you kind of zoom out and this is a thought experiment I don't think a lot of people have really done. I think maybe you have, but thinking about data sort of cross sectionally across the data life cycle really and the different ways that it's modeled. But the shared language is really the only glue that holds all that together at the end of the day. Right. So if you have a notion of a customer at the, you know, at an application layer and a report and you know, at a machine learning model, I mean, hopefully these are the same notions of a, say a customer, for example, that you're trying to do something with whether it's report on it or predict on it or you know, you know, they're using the app, for example. Right. I don't know. But yeah, yeah.

So in 33 years I've never seen a customer and data. Right. I've always seen the word in front of it. You know, marketing customer, risk customer.

Yeah.

Final customer, active customer, inactive customer, active marketing customer.

Yeah.

There's always a word in front of it, Right. Because customer is too broad a definition. And then what I always encourage people to do is then pick one of those bigger terms, right. You know, active marketing customer, and then say the alias for that is customer. We all agree that that definition is actually the definition we'll use whenever we use the word customer on its own. And for me again, it comes back to that language. Right. If you can't, if your language is not clear, if it's just this big Bucket of customer.

Right.

If you have a supplier that buys off you, are they a supplier and a customer? Are they a customer?

Right.

Are they some quasi customer supplier hybrid? So that shared language is really, really important. And without that, how do you model. Right. How do you put a conceptual or a visual representation or even a physical representation of that data when you're using different language at the beginning?

Well, it's really hard, especially when you take into account that data is now starting to be shared not just within an organization, but across different organizations. And when you have the name, I don't know, customer, what do you mean by that exactly? Especially between organizations, it's hard enough to use the word customer very intentionally for the reasons you describe, or there's just a million different ways of describing it, you know, and then you start looking at it across the lens of several different organizations. And what do we mean by that or what do we mean by sales? I don't know. Right. So it becomes a very interesting exercise. But modeling is just one of these things where it sort of happens if you, if you're not intentional about it, and it happens if you're intentional about it. So what kind of model do you want? Right. But that shared vocabulary is definitely a very, very, very strong notion for sure. So.

Yeah. And, and you know, in the data world, we love three letter acronyms. Right? And so, you know, scd, type two. Wow. Now, now there's a language that is designed to make it a, a priesthood. Right.

You know, we, if we work in data, we typically know what it is. It, it would be interesting, right? It'd be really interesting to do a survey to see whether the new wave of analytics engineers actually understand what SCD2 is and why it was there. And I still question, do we need it anymore? It was a technique for star schemas, right. It was about breaking change records out from the fact. Because joining all those tables together in one go was so big and expensive that the databases at the time couldn't handle it. Right. So we, we compartmentalized. We said, well, you know, if we break things out into dimensions, then actually we can subquery, right? We can say these three things. Customer, order and product. You know, at a point in time, had a relationship. That's the fact. And therefore we don't need to worry about location or store or chann at that moment for that query. And therefore the query was smaller, ran faster on our databases. That's not true anymore. You know, we've got a customer where we, you know, have a, a table that's just hitting 300 million rows. Right. It's completely denormalized, it's got 300 columns, it's got a bunch of text in there. But because it's based on a column of database, whenever you are hitting it, you're only hitting five to 10 columns. And the database takes care of that complexity. For me, I don't have to worry about that physical modeling anymore as much as I used to. Right. But still have to care, especially when you get the bill. So. Yeah, I think, you know, a lot of our modeling techniques have been designed and based around constraints that we had 20, 30 years ago. And I'm not sure we've iterated them, which is interesting.

Yeah. I mean, back to your, your bed of whether, you know, an analytics engineer these days would understand, you know, the different types of STDs. A. I would wager that they are not familiar with them. And this is based upon.

I think, talking to analytics engineers and seeing how they're learning about data these days.

As well as anecdotal stuff too. Well, it's all anecdotal. Haven't done a survey yet, but some friends of mine had given a talk at Coalesce, the DBT conference on data modeling, and.

Other friends of mine were standing in the back of that conference actually, and watching people type in what is dimensional modeling into their Google bar.

Think about that for a second. This is dbt, wonderful product, great company and so forth. And I think they've done a remarkable job getting the mind share with what they're doing. But this is also data transformation for analytics. Right. So historically that would be, you know, Kimball or some, some approach like that. And now it's like that's kind of a foreign concept perhaps. Could be wrong. But.

Yeah, I think it's also regional as well. So. Yeah, in Europe we see a lot more databall, you know. So I mean, I was around and I was around in the Kimball versus Imman wars. Right. And. And people go, yeah. And it was interesting. Yeah. People say that Admin and Kimball didn't actually have a war amongst themselves. And that was probably true. Yeah. Didn't know them at the time and not sure. But actually for, for us at least in New Zealand and the rest, and I think the rest of the world, we were warring, right? Yeah. There was Team Kimball when there was Team Inman and we were scrapping it out on every project.

Right.

You know, it was like, what's our, what's our architectural design for data? And then we've seen that To a degree, between, you know, dimensional modeling and Data Vault. Right. And what's really interesting is it becomes a religion, becomes a passion. Yeah. And on Team Data Vault and even within Data Vault, right. There's actually two teams within Data Vault. Team Dan, Team Hans. Right. And I'm Team Hans. And so we get this passion around these things that we've got good at doing and then rather than try encouraging people, we try and force them, we become derogatory. Right. Like you idiot, you know, what are you doing when you're not doing SCD2 type 2 on your dims? What you mean you don't even know what that three letter acronym. And so for me, I think the failure of adoption of data modeling or what I call data design in the market right now is not the fault of people like DBT who haven't encouraged it. It's actually the fault of us as data modelers not being good at teaching other people why the art and craft that we do is valuable. And so if you watch what the vitriol that tends to happen on social media, we tend to go quickly into you're an idiot, you don't know what you're doing. And I did a presentation of the virtual conference last year and my presentation was how do we help analysts data model. Right. That was the focus of it, about three presentations in somebody else basically did another presentation which, you know, I would class. How would I describe it? Being polite. Here's 10 examples of 101 mistakes other idiots have had when they're data modeling. So they shouldn't. And the one that got me was the, the presenter brought up a slide and it had a person in a bar playing a guitar and his, and he said everybody can play guitar but they can't play it well. So if you're not, you know, a trained classical guitarist, you shouldn't play guitar. And, and for me that was like just the complete opposite of what I said in my one, which is people can model and there are a whole lot of things need to learn and it is risky and there's times where you want an expert to come in and deal with a level of complexity that somebody who's not really, really experienced can't. But actually we should teach everybody to model in 90% of the cases. And so for me, I think that's part of the problem, right, is that there's a whole lot of complexity in there. And maybe we'll come back to this idea of the data value stream, I call it. But you know, the areas in the way we move data and the different modeling techniques and why it is, why there are different ones, which introduces the complexity. But I think as a data modeling community we need to look at ourselves about the way we engage and make people feel and then ask that question, you know, why would somebody want to talk to us when we basically don't help them understand what we do?

I completely agree with that. I mean, that's one of the reasons I'm working on the book. I am. It's more kind of taking the approach of, I don't know, Bruce Lee with the Jeet Kundu or something like that, where it's very.

Multidisciplinary. Right. As you say value stream. I think it's a really good example of that. Just kind of, there's different techniques. I'm not here to tell you what to do. I think you're capable of making an evaluation of your trade offs based on your situation. But just at least be aware these are techniques that you could use if you wanted. That's the approach I take. I do agree with you and your assessment that there are certain factions in the data modeling world that are very dogmatic and I would say borderline toxic in terms of how they, how they treat people. I see a lot of these same discussions you're talking about online and I mean frankly it feels.

It'S kind of like. And then you wonder why nobody wants to talk to you about data modeling. Right? Because.

Why, why would anyone want to even have like a simple discussion with you about anything if all you're going to do is just criticize them and, and you know, put them down and call them a loser. Right. That's. Nobody wants to deal with that. Yeah.

And you know, some of those people I'd call friends and you know, the way I liken it is, is they're not doing it to be assholes, most of them. Right. It's not, it's not done on purpose.

I remember back in the days when we had on premise databases, we always used to have a DBA database administrator. There was always somebody that tuned The Oracle or SQL Server.

Or InfamyX database and they were typically. And sorry to all the DBAs out there who are lovely, but the ones I dealt with were defined as the word toxic because they were really experts and they were always busy and they were always grumpy and you go to them and you go, you know, one of the team. Because again, can't code, don't code, won't code. You know, one of the team I was working with their queries are running slow, they go to the dba and I'd go, my query is running slow. Right. So basically open ended question, I have a problem. And typically the answer they get back is well, compared to what?

And it was like, well, how does that help? And then it was like, well, I ran it three times last week. Because you learned, right? And you know, we got a one minute response time those three times. And, and now I'm getting five minutes. So there's obviously something wrong with the database. So now we set up this friction, right? It's your database. And then the answer tend to come back was well, no, it's not my database, it's your code. And then rather than go into the database and tune it, the DBA, because most DBAs came from a development background, right? That was their path, that was great developers. And then they became really great DBAs. They didn't go into the code and rewrite the developer's code, so it ran faster and that. And again, it's not because they were on purpose, it was just the way they worked. Right. They were detailed people, they wanted to solve the problem. And for me that's what I see in the data modeling community. People are trying to be helpful. It's just the way, you know, that they engage is kind of out of alignment. I think of the way that people who want to engage on the other side want to be engaged with, if that makes sense.

Yeah, makes a lot of sense.

Yeah.

And I mean, I'm looking at one of my books right now. Was it 50 years of relational Databases? Another Database Writings by Chris Date? I mean, smart dude, good book. Every page is also full of vitriol about like, you know, if you're not following the relational model then, you know, I mean, just, you know, you're just not a great person. It seems like every page I read in the book I feel that way. And it's a very interesting take. But you know, I personally have never seen. What's that?

Oh, you go, no.

I mean I, I haven't seen any business, for example, fail because they didn't, you know, adhere to very strict relational modeling practices. But if you read through the book, then it's kind of like, well, yeah, if you're not doing this and you're building your entire infrastructure on a shaky foundations, I'm like, well, I don't know, I don't think I've ever seen a business fail because of that. I think I've seen it fail because of really poor business decisions. But you know, you'll, you'll read what you want into the world, right? So the lens. Yeah, you have is the lens you have.

So yeah, I think for me, I think there's a bunch of core foundational patterns and you should apply them when you can because they just make sense. And when you decide not to, it needs to be a conscious decision and you need to be, you need to be cognizant of the fact you're cheating. So you know, we've had this breadth, this breather of NoSQL databases turn up, right? And it's now the cool thing to go and build an application with. And there's massive benefits from an agility and speed to market of being able to do that. But you don't get primary keys.

So that's fine, right? You get speed and agility to build that application. That's great. But when you try and use that data, you're just moving the cost somewhere else, right? So now somebody's got to look at the data and go, hey, where's my primary key? Because it used to be a great hint to say this thing's a thing, you know, these two things are related. So you move that work a little bit. And so I agree with you that I can't think of an example where apart from data breaches, where I can say that organization failed because of their data modeling. But I think I could point to lots of organizations where their costs and their speed to market.

Were way higher than they should have been because of bad data modeling practices or trade.

Let's put it this way.

Yeah, trade offs made without a conscious decision. You're making the trade off. Yeah, that's probably a better way of putting it.

Yeah.

So yeah, but you know, if we go back to that layered problem, you know, we have a habit in the data world right now and we have for 30 years and we haven't changed it, that as we move data and add value to it right into that value stream, we tend to want to adopt a different data model technique at each layer. And maybe we have to, Right. I still haven't thought about it enough to go, do we have to? Because there is not one data model approach to rule them all. But we don't tend to have reuse as much as we move across the layers for some reason. And if we think about application development, it took me ages to figure out what the hell this thing my UX designer kept talking about, which was a design system, right? What the hell is it? And really the way I boil it down to now it's reuse.

Right.

When we do a UX piece of design in our app, we then, you know, that design is reused time and time again where we can, where it makes sense. Right. Because it reduces our cost, it reduces the cognition of the user to understand what's on the screen. In the data world, we don't tend to do that. Right. We don't tend to design something and then reuse it in many different places. It tends to be a one and done for some reason.

Yeah. Whereas like front end is kind of all about reusability. Right. Because it's just different components. I mean, if you ever written a react app, for example, it's like that's like literally the notion of it is reusable components. And I think it's just a matter of like having empathy or at least understanding sort of what people on the left and the right are doing. Say you being in data, for example. Right. So upstream with application developers, like how are they using data? They have a very different use case for it than say the analytics person or the ML person. My take on this is that there's just depending where you are in the lifecycle or the value chain, it's just you're probably going to have to.

Think about at least a physical implementation for that use case there. That's about it really.

There's not a one size. It's not like you're going to tell them, oh yeah, you need to put everything into a dimensional model for your, your front end. Like that would just be. They would stare at you very strangely and then walk away. Or at least, at least I would. It's like you don't know what you're doing. I'll see you, but same thing, right?

Yeah. Isn't that the definition of data mesh?

Take the complexity of data modeling, the consumption of the data and push it back on that poor software engineer who's kind of busy anyway. Right.

Yeah, I mean we can get to that too.

I actually had to give a talk a couple months ago at a Starburst conference about. It was a debate on whether data mesh would do away with data engineering. And I had to take the side that it actually would.

Which it was partly trolly for sure. But, you know, and I actually, Jim, Act's a really good friend of mine and so I asked her, you know, why do you think this. Right. Like, what do you, you know, because it's in her book, you know, I think she, she comes from a software background, so she has a different perspective than she's Not a data person, at least like you and I are. And so I think it was interesting getting her perspective where she feels that the, you know, at the end of the day, maybe it is all software all the way down, you know, and.

Then, you know, so I actually, I agree with that. Right, Yeah, I agreed with that when she first wrote, when she first started writing, I was in, yeah, 100. I didn't think we could achieve it, but I thought it was a great vision. What I've seen is a massive softening of that stance. So for me, you know, for me it's not about taking a data engineer or a data modeler and moving them as a separate role back into that team. It's about taking those skills that we have as data modelers.

Yeah.

And enabling the software engineer to do the work. And because we know they're so busy, we can't teach them the skill and expect them to do it because they just don't have time and they don't have the interest. What we need is, I still think there's this idea of as we do in software development, a set of libraries that take care of that complexity for us. And I think that's when data and mesh decentralization back to the software team will become real. In theory, it's what we should do, right? I mean, effectively they're creating data for the organization. Why don't they just finish the job? You know, why do we have to have another whole team that does the second half of the job? But the reason is because actually the business prioritizes getting that software out the door fast. And the argument I always use is, let's go through the scenario. Software development team have got a new feature, right? It's the new cash cow feature that's going to up the revenue of their company. And the company's struggling, right? So they need this new feature to survive. So the software engineering team, they're rocking it, right? They've done this development, it's taken a couple of weeks and they're ready to push that feature into production, right? They go to the product owner, hey, I've got the feature done. We're all good, ready to go to production. Just need another two weeks to do the dimensional model on top of it so it's usable for the rest of the organization. Now the business, the product owner is going to go, screw that, push it, right? Because I want that business value and I'll wear that technical debt, I'll wear that cost later. Because they don't actually wear it themselves, right? That's the problem. Unless we automate it, and I don't know how we do that, then we're just putting more load cognition and effort on the software developers and the business will make the trade off decision they always make, which is, you know, I have to get value, I'll wear some risk and cost in doing it.

That's really interesting.

I.

Have you thought of any ways to incentivize the business to take data more into account at the software level?

I, I've spent many years trying many different techniques. I've used carrots, I've used taxes. I did. So I spend a lot of my time now, apart from our startup, coaching data analytics teams on agile and product practices is how we blend them together. You know, so we've done things like technical debt registers, we've done a whole lot of stuff. We did one where we kind of started but we didn't finish it, which I was really interested in was if we treat data as an asset and what I mean by that is we actually put it on the balance sheet and we actually depreciate it and that depreciation goes back into the data teams to actually fund them to do the work, would that change the way the organization behaves? And Doug Laney's done a lot of work around that, that idea of monetizing data internally for the benefit of the organization. But you know, in my experience it doesn't matter what you do. For some reason business owners make a trade off decision and data for consumption internally comes last. However, I've never worked in a fan company, you know, the big Facebooks, alphabets and Apples of the world and you know, they seem to have data as a core of their organization. So I do a podcast with a guy out of Australia and we have, you know, kind of product experts and agile experts. Come on. And you know what's really interesting is, is I've started over the last 78 episodes. There's a bunch of themes that have come through, right? Because I kind of sit back and I'm looking for patterns and one of the ones that's really interesting that's been described a lot is that organization started after 20002010 are inherently agile by default, right? They don't do scrum, they just work in an agile way, right? Their growth mindset and once before that they're hierarchical, right? They have organizational structures and Conway's law and all that kind of stuff.

And so for me I can almost infer. And it's an inference, right? I'm guessing here that the post 20002010 companies are also data driven. It's just what they do. Right. Every decision they make is typically made based on data. Everybody before that is made on hierarchy, opinion, the hippo in the room. And so I think maybe that's where the market's lying. Right. Is we're trying to change these behemoth organizations, trying to change the culture and we won't. Right. I was talking to somebody the other day and they said big corporates are trying to invent data mesh. And it's like, well, they're never going to. Right. You need a green fields environment to at least get completely decentralized on day one to work.

So yeah, we'll see. Right. Can we turn those big behemoth ships around? I haven't managed it yet. I've done small changes. Right. Every now and again the rudder knocks them one step left and we do some good practices. But it's hard.

It is hard. Yeah. And I know some people that are saying they've done data mesh at larger companies, but the buy in, it seemed like was a really difficult part. Right. And then.

The way they accomplished data mesh, it seems like a lot of it was data sharing type stuff. So it's not exactly like, I would say the purest form of mash, at least as it's described in the book. But, you know.

It'S baby steps to get there, I guess. I don't know, I don't know if, I don't know if we'll ever hit the ideal of true data meshes. Shamac describes it. I mean, I hope her company is really successful in making it happen. Like I said, she's a good friend of mine and I'm rooting for her. So tbd, I guess it's, you know, it's interesting because you'd almost think that it's, it would be the, the same revolution that the microservices revolution and how that happened. But I think what you're hitting on is that data is just that much harder. And that's, that's the crux.

So, yeah, and, and why that's the, the universal question I've yet to answer.

I, the best I've got so far is.

The data team that are working with the data are not in control of the data. And I liken it to. Because I'm really upset on Data Kitchen and Chris, because I love the whole analogy of recipes and food with data. Right. And then he kind of went and gone down the whole Data Kitchen route. So it's like, ah, you got there first, you bugger.

So I keep going back to it because it just resonates with me. And so if you think about a data team, what's happening, right? We've got a bunch of really talented chefs in a kitchen, right? And they're really experienced at making awesome meals. And we've got a bunch of customers out the front of the shop that want those meals, right? They want their data. They're data starved, right? And what happens is the ingredients that we need get shipped to us and put in our storeroom right there on the loading dock. And we go out there and half of it's rotten.

Oh, I got, you know, I'm gonna go make this beautiful tomato chorizo pasta, and I go out there and my tomatoes are rotten. What am I gonna do about it, right? How do I make a meal out of that? And that's the data problem is, you know, when you're a software engineer, you're in control of the way that data is captured. And I think one of the changes that the whole DevOps thing and why it was so successful is we enabled the software engineer to be in charge of their own kitchen equipment, you know, and so therefore the thing that was out of the control at the time, which is, you know, the servers and, and that kind of stuff, we containerized it, we microserviced it, we gave them the ability to do it themselves. And that removed both the reliance on other people, so their time to market was much faster. But it also allowed them to bake in good quality. From a data point of view, we're still stuck with somebody shipping us data even if we go collected ourselves, right? But they're shipping us data. That's crap. And yet our customers want a beautiful meal. And that's why we have data teams, and that's why it's so hard. And then just the complexity of the rottenness of that data, right? It's hidden, you know, oh, look, the data looks all good, but actually when you get the context, you know, I now have pikokariki in the south island of New Zealand, which is just wrong, you know. So rather than a rotten tomato, what I've got is a little bug that's sitting right in the core. And it's not until I've chop that puppy open, I was going, ah, you buggers.

That's an interesting, that's an interesting take. I haven't actually thought about that, the.

Lack of control over the, the outcome. Right, but it's definitely true. I mean, you've worked as A data person I have and I can't, I can't count how, you know, too many times I've felt like I was in control of the data. Right. You're on the receiving end of it. Not even most of the time. And, and, and the thing is you're working in a kitchen too or you're working at a restaurant where the customers, their tastes change. They might want pizza and they might want chicken wings or something and well, I don't know if you serve both. Maybe you do.

Yeah, well that's, I don't know, domain driven.

They change their request every second. It's like well actually I want Chinese food right now actually. So you have that.

But we're getting there. So the idea of data domains, right, the idea of having teams that are bound by domain is effectively solving that. You know, I remember I was in Dallas pre covered and I went to barbecue over there and oh my God, I loved it. Right. But we were in Vegas and we had one of the most amazing Chinese meals I've ever had in, in Vegas, which I typically, you know, always say is the worst food in the world. So this idea of domains, right? I know when I go to a Chinese restaurant I want Chinese. I know when I go to a barbecue I'm going to have barbecue. There's a very clear indication of what you're going to get. That's what data domains give us. I think agile ways of working give us the ability to change. We no longer, well we should no longer do six months requirements up front because we know we're investing in a recipe that somebody might not eat, right? Well they might go too much paprika. Well bloody hell, I've spent six months doing that recipe. So we get some techniques from there.

I'm a great fan of Lawrence cause work around beam, I use it all the time for the last 10 years. So we have ways of doing small requirements work up front in a language that business users understand. You know we can describe the difference between chicken and beef and you know, get clarity or vegan and not vegan, right. You know it comes from an animal, it doesn't. When we kind of manufacture animal protein from DNA, which we're doing now is going to be a really interesting conversation however before we get to that complexity and then so if we look at the front end there's actually lots of patterns out there now we can adopt.

And you know, and then we, we look at the friction. So we can take Lawrence's beam methodology or pattern and I can convert that to A dimensional model or data vault model with no effort, right? Because it's just the language that's used matches the language of facts and dims or hubs, links and sets. And then we look at the, the quality problem and we look at the stuff that Chad's doing around data contracts. You know, so what does a restaurant do to stop the problem of rotten tomatoes, right? They have an agreement that they're going to get good quality produce from the supplier and then they have somebody sitting there going, those tomatoes suck. Take them back. So the idea of a data contract, right, and not just contract of schema, but contract of frequency and freshness and the context of the context of the data, if that makes sense. Again, we're starting to get those techniques, right, we're starting to get those patterns, but, but nothing in the data modeling space really. There's no innovation or iteration happening. I mean, activity schema, I think was an attempt.

But it's an interesting one too because I feel like a lot of the onus is going to be on software engineers. I mean, the book I'm writing is actually geared more towards software engineers. It's not for data people per se, because if you want to address the root cause of a lot of these issues, you're going to have to start at the source. And I think it's at least making engineers aware.

Of these. Again, no sense in reinventing the wheel. You know, relational modeling works fine if you use it. You know, understanding how to properly use a document, database and model data. And that works fine if you know how to use it. Again, it's just what I see with software engineers and interact with many is that they.

You know, they do what they can to hit their sprint, right? So, and in a lot of cases, you know, they're coding in maybe environments where there's an orm, an object relational mapper too. So I've, I'm not sure if you ever used one, but it is astoundingly easy to introduce lots of fields into your database. Lots of duplication or more at the drop of a hat. You know, I need to add a field in so this form can work. Oh, I'll just do that. And the downstream effects of this is.

Well, it's kind of what you see right now. So my take is that, you know, how you attack the data model, anything upstream is simply by working with software engineers to at least reintroduce them to a lot of these practices and make them aware of the trade offs and make them understand too that the feedback loop between what they're doing and data, and data coming back into the app. I mean that's becoming more of a real thing too. And.

You can choose to code in a silo, but there will be impacts to you. Maybe not directly, but.

At least immediately, but there will be. So as every enterprise becomes more data driven.

Yeah, yeah. I mean, yeah, it'd be interesting whether we actually get an automated semantic layer on their stuff so that both sides.

That's where I always felt the semantic layer should have started was it was at the application end dealing with the. I mean I'm a process junkie too so I mean with a value stream, map out where all the waste occurs.

Where it starts. Like you want to address the root cause. I completely agree. The semantic layer is I think in the most ass backwards place possible. To be frank, it's job for our reports. But I mean, yeah, and if we.

Take that comment you just said. Right. Key takeaway is when I go and work with a new data and analytics team, one of the first things I get them to do is actually describe their value stream. Who does what, where's the data flow, what tasks are done, who does that work? And we visualize it, right. We do nodes and links on the screen, it's a graph. And then we figure out where the wastage is, where the constraints are, where the problems are. And you'd be amazed how many teams don't understand their flow of work. They don't think of it as a factory. They don't understand, you know, if you say to them you're a factory, every time a piece of data moves and a person touches it, it's a station in a factory, right? Map it out. You know, go read the Phoenix project, map it out. Where's the wastage? Yeah, where's. Where's the problems? Optimize it. You make three changes and you'll make yourself far more efficient. And in a week, because you know where it is, you just haven't visualized and agreed you're going to change it.

So yeah.

Some of the simple techniques are there, you know that, that one, it's about that. You can come back to your point. The software engineers are incented to make changes as fast as possible to get business value. And their tools are now really, really effective at that. But they don't understand the downstream impact that the other team has. An example is a customer of ours building up their own app. They're a startup, no SQL database, all good.

They come back to us and they give us.

A combination of snapshot Files, data. So they push the data to us. So they give us some snapshots and they give us some change data. And one day they came back and they go, the numbers are wrong. The thing as a data person, you always love to hear because you go straight into that DBA mode compared to what it's like. Describe the problem. We have a bunch of customers in the dashboards that aren't in our application. Excellent.

But you're not sending us change deletes, you're sending us snapshots. So we don't know that customers disappeared and they're like, ah, ah. So if we just send you the deleted records when we delete them, then it'll be right. And it's like, yeah, that's all we need. But they didn't think about it that way because they're not data people.

Why would they?

Yeah, yeah, why would they? Yeah, it's just like what's gone from the app, like, problem solved, right? Another one, the one that really, really is, gets you is so, because there's no SQL database, there's no keys. So we had a key field, it was keyed off a guid. And then they needed to make some changes, so they changed that, effectively, that structure. So it changed the grain of the data, so they retyped the GUID and changed the grain of it. So now it became parent child key. Right. And for them it was fast. It was like, yeah, we need to make this change for the app. This is the best way of doing it. The whole NoSQL database re optimized itself and now we're starting to get this feed with a key that is no longer single grain, but as a multigrain key. Again, it does bad things to you. It's like going back to something going, look, it's fine, you need to do that. You need to be agile and flexible and able to do that. But here's the downstream impact. We need to work on those ones together and we can't automate that yet. A lot of it's that conversation, that language, and helping people understand what we do and why what they do affects it. And then what can we do to reduce the blast radius of that change? Because that change needs to happen. We can't say you can't add a column to your database. Ridiculous. You think you own your database like we used to do.

Well, that's, I think, where value stream mapping is incredibly helpful. And I wish more companies would use this and more teams, because if you focus on sort of the end product of what A customer wants and the external customer, the end customer, not some internal facing one. But then you start seeing kind of cross sectionally, if you can follow how data moves, information moves, which is one of the things in value stream mapping you do.

You get all the stakeholders, software engineers, data people, business, et cetera. And I think that accomplishes a lot of that shared vocabulary you're talking about as well as has empathy for the end product. At the end of the day it's not like again I don't think anyone's doing this stuff maliciously, it's just you're incentivized to focus on what you're doing, you know, paid to focus on. And if I'm, if I'm not paid to, you know, care about the data team's needs, then why would I, right? Like I have enough things to do right now and that's perfectly understandable, right?

So yeah, they got product, product owners up their bums, right? Going build more, faster and they're really good at iterating their way of work to reduce some of that manual stuff. So here's an interesting question for you then because I just think about it. So, so there's lots of patterns in the software engineering world and there's lots of patterns in the agile world and lots of patterns in the product world that we can adopt, right? So for example, Personas, right? We should be using Personas for our users for data, right? And understand the different types of users we have. Oh, there's machine learning engineer, right? They want the data in a certain, they want raw data. Again you look at the medallion architecture from databricks which is just a copy of a three tier architecture we had 20 years ago, but good that it's getting regenerated, right? That raw combined or cleaned and consumable, right? Standard architecture. And we used to have this argument 20 years ago which is nobody's allowed in raw who Nobody can. You can't trust them in there. And then you get a back there as a data miner and they're like but I need the raw data to do behavioral modeling because I need some flags to say this person bought five products versus this person bought 10 and what they bought. And so we see those kind of data problems turn up. And so given we've lost the art of data modeling because we're not teaching it to the new generation that are self serving themselves and given the software engineers are now using NoSQL databases that don't need a relational data model, what are you seeing? Are you seeing that when somebody's designing an application. As a software engineer, are they still using a mirror board, whiteboard, whatever, just to draw out a structure of the data conceptual model, or are they just ripping in and writing code and refactoring it as they go?

Now, I think even if they do the whiteboarding up front, which is, I would say, pretty, it's maybe 50, 50 when they're starting out.

It's so easy to add new fields in after the fact that you're just like, okay, do I need to have a formal committee meeting on this or do I just add a new field? Right. And so that, I mean, if you look at a lot of code, I mean, I was at a, you know, a dev shop the other week and somebody had go on their, you know, they're just writing a go app and I mean, you just kind of tell like what was happening. I mean, it's just too fueled in the database. Not a big deal. Added in, done, right? So I just think it's because it's easy. It's easy. So, you know, and again, you just need to make the app work and that's it. So even if there is like formal modeling done up front.

NoSQL makes it really difficult, I would say.

There's not an ERD, for example, that works with a NoSQL database. Lord knows there's a million different NoSQL databases too. How do you approach it? But same with even nested data. Maybe you'll draw it like.

Your parent, then children and so forth, but.

It.

Just ends up being a bunch of squiggly lines with a bunch of key value pairs. And that's about it. So the notion of modeling, if I'm looking at my table over there, I'm looking at cod's book, right? That thing is all about just set theory. And algebra is very formal, right. And then you might translate that into an ERD. I know COD hated ERDs, but maybe that's a tool that you want to use to design your database. I just don't see a lot of people doing that these days, or at least as much. So.

Yeah, so. So let's look at how the software engineers have refactored their practice, their way of working. Right.

They've got some new technology that makes them faster and then they've, they've altered their practice to say, there were some things we used to do that don't have as much value now, so we're not going to do them. Right. There's some consequences of that behavior, but we're not going to Model heavily upfront because our rate of change, our ability to change and, and the speed we can make that change compared to the old days where we had a spec, right, which took six months and we have a design, it was signed off by the architect and somebody would go and test that the application we built the data model in the application actually managed main, you know, match the spec, not the user outcome and the product was changed that it said customer is the king. Right. It doesn't matter whether the spec matches what was built, it matters whether the value was delivered to the customer. And sometimes we get it wrong and we just iterate it. And the data world, what have we seen? I mean, I'm amazed that we now have a whole category of graphical ERD modeling tools turn up again. Yeah, it's like, why, Like I'm king. We should be data modeling, but we shouldn't be doing ERD diagrams as our technique for data modeling anymore. I mean, that's 20 years ago, you know, and you see prettier and easier to use than EA Sparks and whatever the hell those crap tools we used to use, right, for data modeling. But we haven't refactored that practice yet. Our brethrens and other domains have. And I'm intrigued, right, because we've seen massive change in the data space. But have we really? Have we seen a change in the core patterns or not?

That's a good question. I don't know that we have. I mean, you know, we're still asking the same questions. I rant about this a lot on LinkedIn. It feels like we're just kind of in Groundhog Day with a lot of the same questions we ask. Right. You know, one of them is like, oh, what value do we add? It's kind of like, well, you have to ask that question. I highly doubt you're adding anything. So there's that sort of inverse correlation between those. But you know, as I tell data people too, if you kind of want to know where the field's going, I would say just look at software and at least, you know, there's at least borrowing of a lot of ideas. You know, I'm not saying it's a direct one to one. Mapping data is different. But data ops, data reliability, engineering and data observability, I mean these are literally just software practices applied to data. And so as far as modeling goes though, I feel like maybe the equivalent to, you know, with analytics engineers is, is really that maybe the DBT model and how they're doing it, which is one big table typically, right so it's just, if it's easier to do that, then that's what people are defaulting to and.

Oh yeah, so I'm going to disagree with you there. So, so if we think about data value stream, so with the data value stream, I think one big table at the end, right? Consumable, one big tables is what we should be going for. The technology can handle it now. But what we end up with if we do that is we in the middle layer, right? And the designing of our data is we end up with thousands of jobs and thousands of tables that have the same data and there's no way of optimizing that. As a user, I don't care about the technology. I'm as a user, which column holds custom and that's a data design problem. So what I'm going to say, I think we're going to see, you know, because most of our changes happen via technology, right? And I think we're going to see technology help us solve this problem and change our practices in one of two ways.

If, especially with ChatGPT now. So if we think about taking those 3,000 blocks of code, you know, you're writing DBT and having the machine create the model for us and then rewrite the code to conform that model, then we now get design data. But we're designing as a chaos theory, right? Everybody's kind of just writing stuff, but the machine is actually coming up with a design as a recommendation and then rootro's fitting it. Or we get a semantic layer, which in the old days semantic layers were at the BI tool layer because that's where the value was. We're seeing them start to be pushed back. But again, actually the semantic layer is the gluing of those 1,000 big tables. To say when you talk about active marketing customer, here's the thing, you hit and we find where that field is for you. You don't need to care, right? So that's where I think technology will change our practices.

It could, it could. I mean, I'll push back and say, like, you know, as a longtime looker user, for example, you know, when I look at lookml code, there's some code I think that has discipline to it in the sense where there's no duplications. And in some cases I've seen LookML where it's just, it's bananas. I don't quite understand what just happened there. Where the whole point of LookML, which, you know, is a semantic layer to, you know, just attach to a BI tool like others you describe, but it did get unwieldy. I was like, I don't know how you got into this master. There's now, you know, your LookML file is almost as big as all your DBT code and that sort of defeats the whole purpose of it. Right? So, yes, but that's due to a.

Lack of practice, right? That's because.

Oh, it's lack of practice. I'm not disagreeing with you on most of this. I mean, we're actually in violent agreement, I would say in most of this stuff it's like, because the same, the same practices that happen with NoSQL where it's just, you know, just let it rip. The same thing's happening in analytics engineering. It's just like one big table isn't at fault. But it's like, you know, 1000 DBT models is more of a symptom of a lack of practice because again, it's so easy just to make a model. It's like, screw it, I'll just do that. Like, why, why make it coherent?

So, yeah, and so if we come back to data modeling, what's the problem? It's not easy to make a data model. So easy to create a piece of block of code that creates data that's valuable. Right. It's so easy to create one wide table that performs now and people can use. It's still bloody hard to design your data. And so that's the problem.

Right? Yeah.

If you make it easy, right? If we, if we make it where it's a no brainer, it just happens and it's low effort. We do it. It's because it requires high cognition, a separate set of tools, you know, a massive amount of experience and training depending on the data you're looking at, you know, header detail. How do you model that? You know, party entity, how do you model that? Manager, employee, how do you model that?

You know, was it Shopify? You look at Shopify and you go 15 tables. It's really easy. Yeah, but no, we've got an order. Order has a transaction, transaction has a return. Yeah, a refund. Refund only turns up in the data when a refund exists. So you start getting the schema of the data you get given change, right? And. Oh yeah, yeah. One of our customers was, they had a bunch of products on their order.

And then the payment, they use gift cards, the payment basically was allocated against the order. But what they needed was they actually needed to understand based on the different payment types, how much of each product line was credit card versus gift. Card. And so that was complex because Shopify doesn't do it for you, right? So you have to basically allocate the costs based on a simple algorithm. But again, there's complexity in what you think is a, you know, it's just Shopify, right? You know, customer orders product, customer pays for order. And that's always my, my first question. I always ask somebody when we go through, you know, some training is, is it customer pays for order or customer pays for product? Is it store, you know, customer returns order, customer returns product, you know, that design, that, that business question infers your design more than anything else, more than the technology, you know, star schema, data vault, whatever. So, you know, for me it's about, it's still complex, it's still a high level of cognition and until we find a way to remove that, we won't, most people won't model.

Yeah.

And I guess the consequences of that are, you know, as we were talking about earlier, it's trade offs, right? So I mean there is, there is debt you incur by, by not modeling, you know, and what that means is you just have a lower, I think a lower understanding of your business for one. I mean, at the highest level, data modeling is not necessarily meant to come up with a physical implementation of data in your database. It's meant to come up with an agreement on concepts and business rules and how data relates to that. So if you, if you, if you don't want to put in that work. And again, this is what I'm writing about. Like I don't really care what you do with your time, but do understand that if you, if you don't take this, this exercise seriously, the consequence is that you, you simply don't understand your own business and the data that you're reporting. And that's kind of mine.

So yeah, so I haven't thought, we're.

Working with, so I haven't thought about.

This before, but you, you, yeah, you've made me think about it. So, so I put, actually if I was going to answer it, I answer it this way. And, and this is not well formed because I'm basically just pulling it out my bum right now.

But based on experience, right? If you don't design your data, you've got a $5 million problem. And the reason is.

There'S two reasons an organization rebuilds greenfields their data platform, right? And if you look at organizations like, if you, if you think about our domain, the data work we do, you know, we, let's look at it as our restaurant, right?

We build a restaurant, it costs us $5 million. We run the restaurant for three to five years and then we go, yeah, that didn't work. We burn it to the ground and we go and build another restaurant in exactly the same place. Different ovens, different people, different patterns, maybe different recipes. You know, we go from barbecue to Chinese. Yeah, so there's a. Yeah, so there's two triggers, right? First trigger is technology. Yeah, we see a technology wave come, happens. The technology we're already using is works. It's got some problems, it has value. But we like shiny, right? So somebody comes in, they want to make a name for themselves. It's like, yeah, cloud data warehouses and they are much better than what we had. But we're going to rebuild everything from scratch because of that first trigger point. The second trigger point is mad person's knitting. Somebody comes in and goes, it's been five years. There's 3,000 blobs of code. Whether it's Informatica code or DB code or whatever, the next shiny thing is these 3,000 things and it's chaos and we can't understand it and we can't find our data anymore. We need to rebuild it. And that second one, that five million dollar bill, is because we didn't design the data up front. Lightly design the data up front. Right. So there you go. That's, that's my takeaway. If you're not willing to change your technology every five years, design your data and save your $5 million.

Or to send A. Shane, one of.

The two, oh, look, I only need 10%.

But you can.

I live in Utah. We do that a lot.

You come from a consulting background, right?

Yeah, yeah, I've done it. I've done a bit of that.

Yeah. And so have I. And you know, you kind of. What I say to organizations I work with now is when you engage with an external party with a vendor, first question you should ask them is what's their business model? How do they make money?

Yeah.

And you know, in the past as consultants, we made money by doing data design because we got paid to spend six months in a room doing it. And it was a hard task and there were very few of us that could do it. And therefore it was expensive and we got paid lots.

Now we don't do it. We spend, you know, we have teams of people that write lots of code. And we know in five years time because we've seen it before, we know in five years time we'll be back or somebody else will be and we'll Pick up the one that they did. So for me, yeah, the incentive for consulting companies to optimize the way a customer works is not there. Right. Their business model is around bums on seats in early rate. But yeah, I don't know about you, maybe that's just end of the world New Zealand thing. I don't think it is.

I don't know. I mean my, my business model of Torari was always different where it was.

Flat rate, fixed price, fixed scope. So I had every reason in the world to kind of like finish quick and get out.

And the other incentive I had too, I mean when Matt and I had started the company, it was like it was just the two of us and now it's the two of us again. You know.

We never really liked doing services, you know, kind of a hands on keyboard, button chair kind of work. It just, I don't know, I felt like I had a job and if I wanted that I would choose not to work for myself and go get a job. So it's a lot easier that way. But the approach that we would take actually is coaching and I think showing data teams how to do what we do and really sitting alongside them, pair programming and just I think helping them level up their skills. And people love this because they feel like they're empowered, they feel like they're in the driver's seat and there's always a constant tension. If you're a services company to come into a client most of the time they're probably going to hate your guts on some level because you're a higher paid version of what they're doing. They feel threatened by you. And I felt like the jiu jitsu move really was, well, what if we just helped you become a better version of what you're doing and we have no incentive to stick around longer than we have to. So.

Like I said, people tend to like that approach. That's how we made money. Was just, I would say sort of the almost doing an anti consultant type of thing in a way. But it worked for us. I can't say that it's a good idea for a lot of people. It's probably a really terrible idea, but.

I think it's a good idea for people and it's a good idea for customers, you know, effectively. It's a coaching model, the same one that I do when I need to do a side hustle. The problem is it's not a good model for a consulting organization that has 50 people that can code need to be off the bench. Right. So Go back to the business model. Right. If you ask your consulting friends, you know, partners, how do they make money? And they go, well, we coach your team to be as good as us and then we exit until you need us to coach them something new. That's the partner you want. Right. If they go, you know, we've got the experts in the market. It's like, and how do you charge for them? You know, hourly rate, daily rate, even fixed price. Right. Because fixed price is still time and effort. It's still effort based. Yeah. Then run and find somebody, you know, that's going to coach you. Out of interest, did you, did you coach them on data model?

We would, yeah, I would, you know, when necessary. I feel like it was. It's one of those things where it's like, I don't know, showing people how to floss their teeth or something. Some people want to learn this and others are like, I have no interest in learning this stuff at all. Just show me how to use Snowflake. Right. So. But I think implicitly what I've learned is, you know, you want to sell things for the action and the outcome. Not really.

So if we're going to talk about data modeling, then it's like, well, let's think about your business. Let's think about the data we're going to put into Snowflake, for example. Right. So what are you trying to accomplish? What are the reports you want to generate really? And work from there, Work on the outputs. If you try and approach it from data modeling, people think you're just giving them a lot of extra work to do. I think.

Yeah, that's how you pitch it.

So that's an interesting one. So again, I've got a slide I do when I coach teams and I talk about these three ways you can design your data model. Your data, right? So source specific. We model off the source. So we go spend some time looking at it. Output specific, so we look at what the user wants on their report, their dashboard or the thing, and business process specific. And over the years I've become very opinionated. We model based on business process and then we map it to our source systems and what the user's after.

That makes a lot of sense.

Yeah. So who does what? So I know we're almost out of time, so I'm going to flip the thing. One question for you. So given you're doing all this research on data modeling for your book, what is the best one data modeling technique you should use and why is it data model?

I think it's not a data modeling technique at all. I'm more convinced than ever. I think that actually taking a step back and doing value stream mapping and process mapping is probably the best modeling technique you can learn. It just helps you understand the flow of.

Data and money through your organization and I think it helps you unlock where all the bottlenecks are. If you can do that, you have a superpower and I would say the world's your oyster in terms of whatever data modeling technique you want to apply. If you don't understand how things flow, as Charlie Munger says, you're sort of a one legged man in an ass kicking contest.

You had a pretty serious handicap. So I don't know, what about you? What would be the, you know, one technique to rule them all?

I'm a data vault bigot only because it's the best. It's the best. And data vault 1.0, right? So I talk about data vault modeling, Not Data Vault 2 and all the other crap that goes supposedly with it. So for me, Data vault at the moment because it's the best modeling, most flexible modeling technique I can find and it's got a bunch of problems and I hope somebody comes up with something better.

But I go back to your point there about Value Stream. So if you've got a team, get them to map out their process and how they're working. When the team touches data, highly valuable. And then the next thing you want to do is you want to map out your core business processes as an organization and use the term who does what. And you're going to see if we take an econ one, customer orders product or customer places order, customer pays for order, you know, store ships product or store ships order. You're going to see this hierarchy of business processes and that's your data model. That's your shared language. Whether you physicalize it in dimensional or vault or activity, who cares? And as soon as you start asking who does what and you map it out as, you know, as a graph, as links and nodes on a screen, on a whiteboard mirror or whatever, you're going to find the complexity, right? You're going to go to that problem of does a customer return the order or do they return a product? Because they're two different design decisions and yeah, value stream, who does what? There's your conceptual model, 80% of it, right? And your big areas of concern, you know, they stand out like kind of what you need.

That's all you need, man. Really. If you're to ask me like my favorite data Modeling technique. You know, I would say, like, another answer is, you know, if you were to do. If you're looking at ufc, like, well, what's your favorite? You know, what's the best mix? What's the best martial art or what's the best move that you could use in a fight, Right. I think it completely depends on where you are and what's happening to you.

There's no shortage of modeling techniques out there. I think that's.

And some, I think, that are tbd, like in streaming data and machine learning, that's still, I think, kind of up in the air if there's any canonical way of doing stuff. But.

You need to learn what works, I mean, and be able to apply it. So I love that.

I love. I've never thought about it. Mixed martial arts. That's a great way. That should be the theme of your book, right? The mixed martial arts.

Oh, it is. It's actually a big theme of it. Yeah. I mean, I, you know, totally nerd out. I didn't used to do it, but it was like. The thing you quickly realize is, yeah, if you're like the kung fu master saying, oh, yeah, kung fu is like the best martial art, like, that guy's gonna get his ass whooped pretty hard very quickly.

In spectacular fashion. And that's how I feel like a lot of people approach data modeling these days, where it's like the one true technique, and I'm like, I don't think there is. And if you try and do that, especially in a cross sectional way, you know, across the data life cycle, it's. It's the same thing. Like, again, going into fight, only being a boxer, like, you're gonna get taken down and manhandled. So that's what happened. So.

Yeah, yeah, I'm with you. Right. Yeah. There's a bunch of patterns, and you get good at them, and you always go towards the one that you know the most.

Right.

The one you're the most comfortable with, because that's, you know, that's when you're in fight or flight, that's what you refer back to. Right. So wrestlers always take you to the map boxes, always hit you in the face. Right. That's how it is when they're in trouble, but you still need that toolkit or you're going to get your ass whomped. So, yeah, there we go. Yeah, that's what we need. Mixed martial arts of data modeling. Yeah.

Well, awesome, man. For. For people who want to learn more about you. How can they do that?

LinkedIn's probably the best way to do it. My hand under this shagility.

A little bit on Twitter, but I tend to be rent ready. Okay.

I always wondered what that was. I thought it was gonna be shagility or something.

And yeah, so. So it's Shane and agility, but it's also a little bit of Austin Powers, baby.

The international theater nerd of mystery. It's funny.

You got a podcast too, right?

Yes, I do too. I do. The Agile Data podcast. What that's about is where I've found somebody that has a set of patterns that they can actually describe without pitches. I get them on the podcast and that's actually really hard. It's really hard to get somebody who goes, here's a pattern that you can implement and here's the context of when you should and shouldn't implement it. So that's that one. And then I co host one called the no Nonsense Agile Podcast where we have agile and product people come on and for me, I've kind of. It's really interesting. It's almost like free training. I mean we get, you know, experts on who know their stuff and then they spend an hour talking to us and I get to ask questions to kind of understand, you know, how continuous discovery works or those kind of things. And then I'm like, cool, I learned something new and how do I apply that to data and how do we apply that to our startups? So I don't think we've ever told them and hopefully they don't listen to your podcast. But yeah, I just treat it as a free hours training. It's awesome.

You know, I always do the same thing with my podcast too. Like, I feel like I learn a ton. You know, even today I feel like I learned a ton. I took a lot of notes on the side and. But yeah, I mean, it feel. Feels like podcasting is sort of this cheat code where you can, you know, talk to really smart people and learn a lot if you're paying attention. That's. I think it's awesome.

So yeah, and, and because my co host on that one's Australian, he's really, really grumpy. And so we're going to call it the, the no podcast. But we got flagged.

You get for that.

Yeah. So what Murray's really good at is, is. Is really just getting down to those examples. Right? Because what we found was there were a lot of fluff merchants and there isn't data as well. Right? There's a little. Like you said. Yeah, there's a lot us talking to ourselves a bit. And so, you know, getting back to that. That meat and potatoes, right. Which, you know, you know, writing a book, that's what you got to get to, right? You can't write a book of fluff. You've got to put some meat and potatoes in there. So could.

It would really suck. Yeah. So you'd like Salt Lake City, too. I mean, not too far from here is where the Agile Manifesto is penned, actually. Up at Snowbird. What was it, 22 years ago now or something like that. So, yeah.

Yeah, I've got to go to Austin next, apparently, because all roads lead to Austin. But, yeah, definitely. Yeah, we'll be over in the US probably sometime next year, so we'll do a bit of a tiki tour around. Maybe come and visit.

Yeah, please do. You got a place, so. Awesome. Well, great chatting with you, and I guess I'll see you back on Slack, so. But yeah, fun, fun conversation. I learned a lot on this one, so, yeah.

Yeah.

Awesome. Well, we'll talk to you soon, man. So take care.

Excellent. Thanks for having me.

All right.