Diagramming to  Understanding Your Data Estate with Rob Long - Episode #76

Foreign.

Welcome to the Agile Data Podcast where we talk about the merging of agile product and data patterns in a simply magical way.

My first book, An Agile Data Guide to Information Product Canvas, is published on Amazon. If you hear your stakeholders say, hey, that's not what I wanted, and then you hear your team reply with but that's what you ask for, then you probably have a problem capturing data requirements. The Information Product Canvas is a pattern template that is a quick and simple way to solve that problem. Head over to agiledataguides.com and click on the Green Book link to buy a copy now.

Welcome to the Agile Data Podcast. I'm Shane Gibson.

I'm Robert Long.

Hey Rob, thanks for coming on the podcast today. I wanted to have a chat to you about this article you wrote that I really loved. It was called Data Producers, Consumer Diagrams and Understanding your Data Estate. But before we rip into that, why don't you give the audience a bit of background about yourself?

Yeah, I'd love to. So I got started with data in academia, so I actually have a PhD in Applied Mathematics and geophysics. So I was doing mathematical modeling and numerical simulation of planets and stars, things like sunspots and storms on Jupiter. And without realizing it at the time, what I was doing was bringing machine learning and data engineering to that field because these simulations generate big data and we were implementing supervised machine learning, but it was still a step up for that disciplined group of people. Since then I've dabbled all over the place. So I had a couple of years where I ran a software as a service helping schools in England improve their national ranking. So again, very data centric. And then throughout my career I've gone through and I spent a couple of years as a senior data engineer in defense in the uk. So really helping them upskill in using cloud platforms, how they do data and was using Gen AI to help with content creation, so scripts, audio, imagery, things like that. And now I've moved from the UK to the US at the start of this year and I run a consultancy where I focus mainly on enabling analytics. So lots of data engineering work and data strategy and that later part is really where this kind of producer consumer framework came from as I've been helping companies new to data build the foundation. So really getting them set up on the right path and that's how this kind of came to be.

Let's do a little bit of anchoring around terms because you use slightly different terms than I do, so we'll anchor those. So just talk me through this idea of how you define a data producer, a data consumer, and, and the concept of a hand loop.

So I think a producer, very broad strokes, is an individual, team or system that produces, which means to generate or capture data. So that could be thermometer reading, temperature, it could be CRM, system, whatever it is. A data consumer is someone who takes in that data for a purpose. And so there you're thinking they need data to look a certain way so they can use it to answer certain questions. And then a handover is really this interface where the two meet. So that's where a producer is literally handing over, in this case, data to a consumer. And so there you might think about terms like data contracts, you might have quality checks, compliance, whatever it is. And so I build these networks on those three ideas really.

So the language I tend to use is I was a great fan of a TEDx video called how to Make Toast and it's around system thinking, so very much the same. And he talks about nodes and links. So for me, I talk about nodes and links. A node is where a job is done, something happens, and then the link is the handover to another node. Data producer, in terms of data is created in a CRM, or a sensor is creating an event that is a node, and when it moves to somewhere else to get worked on or consumed is a link to the next node. I'll jump between your terminology and my terminology all the way through, as I always do. So we've got this idea of producers, consumers and handovers, or nodes and links. How do we use them? Why do we care? Where's the value?

I think the value is really in unifying other things that already exist. So lots of companies, for instance, might have an organization chart which gives you an almost vertical view of the roles or teams and where they sit. But as we know, lots of data work happens horizontally. So there again, it's not to replace org charts, but it helps enrich them because these producer consumer diagrams are very workflow driven, trying to capture a flow from start to end. So that's one use case where it lets you overlay these different artifacts you already have. As I said, an org chart is typically representing people. You might have your architecture diagrams which represent technology. You might have data lineage or flow diagrams which capture how data's moving through these systems. And all of them, to some degree, you can represent as producers and consumers because they're either generating or using data. And so I think this is really powerful because it helps give you a unified view of your data estate. All of these artifacts are really good. But together they give you that workflow driven kind of diagrams which help unify everyone and reduce organizational friction.

I agree. I think the idea of using a map or a diagram to visualize a concept is really valuable as long as we don't overload it, we don't make it too complex in itself. So those maps should be as simple as possible. And so we can use it to map systems. We can, like you said, we can use it to map all charts, we can use it to map flow of data, we can use it to map data layers and the rules on each of those layers. We can use it to map team design, we can use it to map a flow of work. As you say, somebody works on something and then hands it off to somebody else who uses that. I think if we go back to some of the original data mesh thinking there was this idea of input and output ports, so we're talking about that somebody does a bit of work and it gets handed over to somebody else who does another bit of work and we just visualizing that workflow or that flow. Is that how you see it?

That's exactly it.

Let's give an example, concrete example. Talk me through an example of one of these maps.

Going through an example is really useful because there's also this concept of you can look at different granularities and I show diagrams of that in the article, where at a very high level you might just have your external system which is producing data and then you bring it internally into your architecture where you have your own processes for data. So there we've got the external being the producer, the internal being the consumer. But if we look more deeply internally, we might have a data preparation layer where we ingest, do quality, create our data models and then the reporting function which takes that middle block which is now consuming data and producing data. So you get this cool thing of as you add nodes, things can turn from just being a producer to a producer consumer. And so I think there, that's an example where these networks are dynamic, but they also can have as much information as you want them to have. So to your point, at a very high level, you might want a very few number of nodes and keep it almost like a conceptual model. And then as you go down, you might want to get into the real nitty gritty detailed view to understand exactly, for instance, which technologies are doing what, which teams are responsible where, and things like that.

I think the level of detail can vary. So for example, if we think about layered data architectures which have been around for donkeys, but are now hot and popular again because of the medallion architecture. We can have a very high level map of nodes and links where there's five source system, whether it's produced, data is generated, bronze, silver, gold, and then the consumer. And that's only five nodes. And it tells a very simple story. Now what we know as data people is there's a whole lot of extra complexity. Where's the data quality rules being applied? Are there data contracts across each of those nodes? There's nodes within that. Whenever we get into silver, how are we conforming data? How are we mastering data? How are we bringing in reference data? A whole lot of things. Where are we defining those metrics? So we can take that very simple diagram and then with only the five nodes, and then we can break it down to even more detail when it makes sense. So we can present the same system with two different stories, depending on who our audience is and the story that we want to tell.

I think that's really powerful. That's exactly right. Is as you said, you might want the data engineering team to understand what goes on within this layer, but your business analysts who are consuming at the end, they don't necessarily need to know, care or even have access to the data that's there. And so that's exactly right. It's powerful because you can create variants very simply which tell different stories for the audience and it lets you again unify all those other bits as needed.

One of the other ways we can use it is this idea of when there's a handover or when there's a link, what's actually involved? So what is handed over? Is it just a blob of data? Is it a blob of data with a schema? Is it a document? What does the person hand over to the next person? What does the producer hand over and what does the consumer expect? What do they need to be handed over to them to do their step in that process? And a lot of people don't think about that. We don't bring in that lean system thinking to say actually where's waste in terms of where the producer generates some stuff that's never used, but also waste because the consumer needs some information that's never provided. And now they've got to go and do all that work again and introduce that wasted effort that's already been done because nobody told them that it has been done or what it was.

It plays into a bigger story of kind of, in a very ideal case, if everything's perfect, these are Very unidirectional networks. You take data from source, process it, it's perfect, so it's consumed in reality. It's more complex because you want to empower consumers at each handover to be able to report errors and feedback and say, you've given me this with a bunch of columns that I don't know what they are because they're undocumented or they're new in the API or for whatever reason. And so it's really important to have both the producer driven flow, which is in my mind left to right from source to target, but to have the feedback loops that go in the other direction. And so then you can have both the functional benefits of fixing errors quickly, knowing that they exist, what have you. But also from a very almost people level, remove some of the idea of playing the blame game because you're no longer throwing things over the fence. It's about having well defined processes so that everyone can work together to fix these problems. And I think that's really important.

Yeah. And if we look at it again from that system thinking, we can have different lenses on what those processes are. So for example, if we look at the flow of data, we can look at data contracts and we can look at data contracts more than just system of capture to the people doing the data wrangling. When we get it into that rule layer, we can actually have data contracts between every node and link. So if I'm transforming some data and creating a metric, what's the contract for that metric? Is there a standard YAML format? Is there a certain descriptive metadata I've got to store? We can actually put data contracts in between each of those steps. If I look at way of working, we can bring out some of the patterns from Agile and Scrum where we can do definition of ready and definition of done. So the consumer can say, hey, if I don't get given these things, then you don't meet my definition of ready. Therefore I'm not going to start work because I don't have the tools I need to do the job I need to do as a producer. I can have a definition of done. Here's all the things I would expect myself or anybody else in my team to have done before. I'm saying the work's done, it's ready to be handed off to the next node or the next consumer. So we can bring that idea of gates and patterns and pattern templates into both the way the data works, the way our data platforms work, and the way our teams work.

Exactly that. And that's why I really like this idea. And I think that it's always going to come up like the idea of data contracts, which is that acceptance criteria between producer consumers for data sets effectively. But producer consumers are really important for telling you what needs to go into a data contract, because it's far more than just technical. As you said, if the analysts are receiving data from an engineering team, they need and expect a certain level of documentation formats, whatever it is. And so it's really about getting, removing friction and unifying those requirements and expectations from both sides. So that's exactly why I really like these as a framework to use.

And I think one of the things about data contracts is everybody looks at it from a technical point of view. They look at it as what is the schema, what is the field type, what is the grain? But they forget the actual word contract, which basically means an agreement between two parties. So it is a negotiation between the producer and the consumer about what, what's needed for both sides to do their job well. And that's how we should treat it. We should actually negotiate the contract, not just treat it as a technical task.

I 100% agree and I think that's probably improving. But at least in lots of cases I've seen data contracts are taken to be technical and not have the surrounding pieces, which I think I agree with you, they're missing.

I've never seen a data contract that has the level of documentation involved in the handover, unless it was an API contract, which is in theory, self documenting. But we don't put APIs between our bronze, silver and gold layers. We should probably, we should definitely put contracts in between each of those moving parts. We don't put data contracts in between the way we capture requirements as a business analyst and hand them off to a data team. If we're running separate teams that do requirements gathering and a separate team that are doing bill, which again, I don't actually suggest you do, put them in the same team solves a whole lot of problems. So one of the other things you've done is you've used that idea of consumers and producers to actually map it to tools and technologies that have been used. So just want to talk us through how you do that and where the value from that pattern is.

As I said, I view these as a way to unify these different lenses. Technology people process data. And so when we talk about, for instance, ingesting data from an external system, we're going to use a technology to do that. So I'm going to use the example within the Azure ecosystem, that's why I mainly work with but any ETL tool, for instance Azure Data Factory, you can then say this edge of this diagram, this handover is going to be performed by Azure Data Factory. And then internally, once you've ingested it and gone through your medallion architecture, that's going to be some data warehouse, postgres, Synapse analytics, whatever data platform you're using through to once you get to the reporting stage, you're going to have a BI tool, typically, whether that's Python, Power BI Tableau. And so really the technologies in general map to the handovers. There are cases where they can be the nodes as well, the producers or consumers. And then I think what also that lets you do, again, the benefit of that is it lets you have an informed decision about do you really need this technology? So when you're thinking about cost, performance, capability, when you're talking about does your team have the skills to use this technology or do you have to learn something new? Again, by bringing it all together, you can make an informed decision about that. So it's much less the preference of the architect or listen to a vendor pitch and it lets you get that view of what you need to use.

I think the other thing I liked in the diagram you used as an example is you also bringing in the Personas. So you're mapping out the flow of the data effectively, then the technologies or tools that have been used in each step of that flow, and then the Personas you're expecting to use those tools. So you're differentiating between the tools the data engineers will use and the tools are a Persona of data analyst.

Yeah, and as I said, a lot of the people I work with and my clients are less mature with their data capability. And so for them, a lot of them start with the Google definition of data engineer or data analyst. So it's much more useful then to use these types of diagrams, map to technology and really distinguish the responsibilities of the Personas. So then they can hire the right people, get the right skills. But it also again reduces friction because if you're brought in as a data engineer and you've got clear roles and responsibilities, you're not going to kick up a fuss because you've been given something that you deem as pure analytics or whatever the case is. And so I think again, that's really useful to have because it gives you that unified view of your data estate and lets you make informed decisions about technology, team structure, how data flows from.

Source to target the thing I like about it is it tells me a story, that it's a map at a high level and I can understand it, I can infer some stuff and I can get a bunch of questions. So the one I'm looking at, you start off by saying there's a source system that goes across to a data lake, that goes across to a data warehouse, that goes across to Analysis BI and then across to a report. So I can look at that and go, I think you're running a three tier architecture data lake, data warehouse and an analysis BI layer. And then you've got source systems of data coming, being produced and then reporting content on the right hand side where people go and use them and then you're breaking it down to tools. So you're saying that effectively Azure Data Factory has been used to extract and load the data into the lake. Synapse analytics has been used to munge and wrangle the data. Power BI has been used to create the analysis PI layer and then PowerPoint Excel has been used for the primary reporting. And then you've got the idea that data engineers sit on the left and do half the work in data analysts. And so I can look at that and I can go, good, there's no omni, there's no thought spot. I can get an understanding quickly of the technology. And I go, yeah, you're a Microsoft stack. I'm not seeing Snowflake, I'm not seeing databricks. I know that effectively you're gonna be on Azure, not GCP and not aws. I can see you've got two sets of Personas in there. And that raises a question because in the way you've laid it out, it looks like the data analysts are reaching back into the data warehouse. So I can have a question of are they read only or are they actually able to write transformation code in the warehouse? Your analysis BI layer is Power bi. So I'm just going to assume that it's DAXI semantic Y thing and I'm going to ask you questions around where metrics defined, are they only in that layer or are they somewhere else? Yeah, your data lake, I'm going to assume it's azure data lake gen 2 or something like that, even though you haven't specified it. So I'm going to ask you those questions. So again I can get a really quick map of that environment of that estate and I can ask you a whole bunch of questions for things that I just want to know because I need to know or I'm just nosy and I want to know. And that's the value of these maps.

Yeah. And I think what you've just touched on is really useful in that you can create different views like we discussed about earlier. So you might have this, which is your more executive focused story, and then for your technical users, your power users, who you might have that more finer grain that you're talking, where you go into more specifics about technology, about the rules. And again, you could also go more in detail. For instance, in my experience, a lot of frustration and friction comes from the handovers is that mismatch of expectation, definition of ready, definition of done. And so this lets you account for some of those. It lets you see quite clearly who owns which parts, is it within one team? So the analyst BI to the report layers are both owned by the data analysts. So the part of the story there is that they should be able to do that internally. Whereas if you look at the handover from data engineering to data analytics, there we might need some cross team and you have to work up the processes for that. But again, you're informed of that with this diagram, so you can start to flesh out how that's going to work, who's responsible for what, how communication happens, if it's a slack conversation, a JIRA ticket, whatever you want to use for that kind of problem, tracking the extra layers of information you can get when you go to finer granularity.

And I think it's the idea of a boundary, because my eye is naturally creating boundaries and saying these things. These nodes, these links are in the boundary of a data analyst and these other ones are in the boundary of an engineer. And now I'm going to worry about where the boundaries look weird to me or where there's the touching of the boundaries or an overlap of the boundary. One of the things you do though, when you diagram this is you're effectively bucketing, putting together the consumer and the producer node with no line. And I'm assuming that's your visual way of saying that a person consumes something and then produces something. And that in my head is within the node before they hand off to somebody else. Is that just a visual style of the way you do it?

It is. So it was trying to capture the logic that for instance, the data warehouse it consumes from the previous node, the data lake and that same data warehouse produces to hand over to the BI tool. So again, you're right, you can look at this and view this in different ways. For me, it's really just about having the node labeled as either Producer or a consumer or a producer, consumer pair. And yeah, there's different ways of doing this. This was one of the simplest ways that I could fit quite a lot of information into an easy to read diagram. And so it's a complete design choice that I've had success with and I.

Think it's just a choice of language. Now I understand that when I look at those, they're effectively an ensemble and I'm looking for ensembles where these consumer producers, which means these in and outs within that node, or I'm looking for one where there's only a producer or a consumer task. Effectively, again, it helps me tell that story.

And in general, obviously you very rarely want to produce data that's not being consumed. So typically they will come in pairs. And obviously there's always the question of where do you end the diagram. So in the case we're talking about, the reporting layer says consumer producer, but producer is not connected to a consumer. So again, the bit that's missing from there is that there is a decision maker, an executive who takes that report and makes a decision, takes an action. And so that's also where I say it's as much art as science when you make these kind of diagrams is there's no rule for granularity or your start and end points. Most of the time there is a logical start point, but where you decide to end the flow, as I said, it can vary a lot and it really just depends on what you're trying to convey with that particular story.

And that's the important part is you are telling a story. So I could take 5 node 1 you've got of source system, data, lake data warehouse analysis, BI and report. And I could extend it to the right and actually say actions that are taken off those reports and outcomes are delivered by those actions and value that's delivered from the outcome. If that's the story I wanted to tell.

Yeah, and then you'd obviously sell a third Persona, the executive or whatever it is. Exactly. That's how easy it is to iterate these and see the difference between where you are and a proposed change to your process framework, technology step. Whatever it is, it directly gives you the implications and new connections you need to manage and prepare for.

And that's the other part is it helps with the story. So for example, in this scenario, if I said, oh look, I want to introduce a new BI tool, consumption tool, last mile tool, I'm either going to have to find one that is compatible with the power BI semantic BI layer. If not, we know that the blast impact of that decision is we now actually have to have two semantic BI layers or replace the power bi one with the one that actually serves multiple LAS models. So again, I can point to something and say if we do this, then we're going to affect that and visually point to the map so people understand, oh, holy shit. Actually that's one third of our data layered architecture that we're replacing. If we've only been working on it for six months, maybe if we've been working on it for 10 years and we can't programmatically migrate it, we've got a lot of work coming. So again, it's really valuable to be able to point to parts of the map. And so we're talking about that, we're talking about the uk, not the us. That's the value of the map.

Exactly, that's it.

And then the other thing we can do, so we can extend left and right and we can also change the grain effectively so we can add more rows. So for example, at the moment we're saying that all source systems producers and they're the same, but if I wanted to, I could create different rows. I could say we've got relational versus no SQL sources or we've got streaming versus batch sources. So if I wanted to, I could just add more rows, more nodes and links as a row to add more complexity or tell a different story to this map, couldn't I?

Exactly. You can add more rows in terms of different sources or even like I said, making it modular and taking out the SYNAPSE piece and say, what if I replaced it with databricks? And then you can obviously also account for new data. So in this case the data warehouse is just producing data sets to give to the analytics and BI tool. But you might decide that you want to also capture system data from your warehouse. So from Synapse, maybe how long your Spark clusters have been running, how many outages you've had, how much it's cost. And so you can also kind of not add new rows, but add an extra dimension where it's not just going left to right, but you have metadata that kind of goes up and down as well. So you can add different directions or paths possible in your network.

So do you tend to design the complex map first and then simplify it to tell different stories, or do you tend to start off simply and then add the complexity as different versions as you go? Which way do you work?

So I do something very weird where I start with a super high level simple flow, so I really just want to get the conceptual model right and then I skip the middle bit and jump to the perceived finest grain and do the real in depth. These data quality rules, these data contracts, these technologies, and then I backfill the middle bit because I find it really easy to question the requirements at that very high level and then by jumping to the really low level, find the real complexities. So like I said, if there's a skills mismatch with the team versus technology, which then I can propagate upwards. I've got a few horror stories where I did go from top to bottom, got to the bottom and then realized something didn't work and you have to backfill the whole thing then. And even though you're only changing one node, it inherently sometimes changes the nodes that it's paired with. Obviously because things work differently and connect differently. So that's how I work. I think it can work both ways, starting at the very high level or very nitty gritty detailed level. I think it's a matter of preference and how good your initial requirements are. I find the better the requirements, the easier it is to start at the high level and then go from there.

And then you are handcrafting these, you're drawing them as if they're pictures. You're not using a tool to help you with this.

A little bit of both. So I do start very conceptual hand drawing in a tool like Draw IO, but I do automate some of this with Python in terms of just treating it like a knowledge graph. And you can create a simple CSV file with your nodes and edges and get it to populate a graph. So I do a hybrid approach to get these ready.

Yeah, it's interesting that there's a gap in the tooling I think at the moment. We used to have things like Spark Enterprise Architect, which was a horrific product to try and use. I used to do enterprise architecture and I'd go into an organization. I was forced to use that tool if I knew in advance I wouldn't take the gig. Just made you so slow and such waste. And like you, I just use Draw IO, but it's a graph problem. We're talking about nodes and links effectively, ins and outs and relationships. So surprising that actually there's not a great tool for defining it. But then also creating the simple stories, I would say.

I think the closest I've seen is Mermaid js. So it's diagramming as code, so it works, but you just don't get the customization to make it more user friendly. So it's great for spinning up dirty diagrams that a technical team would love. But to sell this to an exec, it falls short exactly as you're talking about.

You're telling a story. So therefore it helps if the stories attractive to look at, not ugly. And yeah, I've done ones where I've used some of the LLMs. So I'll put it in as a CSV, get it to give me the mermaid syntax and then put it into a mermaid viewer. But it's ugly, tells a story, but not in the most attractive way. I think the other thing is again over complicating at your risk. So if you can keep it left to right in an English speaking country, people will understand it as soon as you start branching off to if then else statements where you're coming along and you've got to go up and or down. And again that just increases the complex complexity of the diagram and the story you're telling. So if that's the story you need to tell, then do it. I think the other thing is how many lenses or dimensions you bring to it. So for the one that you've got here, there's a flow of the data and the data layers, there's the technology and then the Personas. And I find that three is normally the maximum you get to before again you start bringing complexity in. So if you add another two in there, you've got to do that consciously. You're consciously saying I want to make this a more complex technical diagram than a simple map.

Fully agree. And again, it's all about knowing your target audience and the story you want to tell. So again, I think these are, these are not your super technical architecture diagrams and they're not meant to replace them, they're meant to complement them by giving a view which is consumable by your executive team, your managers, whomever else, decision makers who don't know what the icon for databricks is, for instance. So when you show them an architecture diagram with networks and tool specific icons, it's noise. And this is about almost filtering that into something useful for that audience.

One of the ways I use this nodes and link format and this idea of maps is as a workshop with data analytics teams when we want to change the way they work. So the way it works is I get the team together, I basically put something on the left and something on the right, either on a wall if it's in person, or on our virtual whiteboard if it's not. So an example would be data sources on the left Information consumers on the right. And I asked them just brain dump, brain dump all the data sources on the left and brain dump all the people that use whatever you produce on the right. So I start giving a bit of a map and then I say to them, just use some stickies and do a sticky a node for everything that you do to get the data from that left to that right. It's always interesting. So some people do very high level stickies, some people do very detailed. At this stage I don't care, I don't give them any boundaries. And then once they've done that, I get them to group those stickies together. So effectively where you're doing the same task, put it in the same area. So your idea of where it's in and out and it looks like it's the same. And from there we now have a flow of work. I can get them to use the dot pattern to say where do you think it's broken? Where do you want to invest in change? And a whole lot of other things. But I find it a really quick way of getting a team to document their processes without endless interviews and documentation. And also it's really interesting when a leader sits in the room and they use the words, but that's not how we do it, right? We do this and the team just laugh at them and go, no, this is always how we do it. Or when you have two teams that actually have completely different processes, there's some things that are shared and there's some things that aren't kind of looking at from that lens. Give me some examples of how you actually create these diagrams. Is it just you? Have you ever done it in a teaming environment?

So there's two ways I've had good success. One is where I've been brought in as a almost contractor and so they've given me the requirements and a few hours of someone's time. So I go through the requirements with them and then create this myself. And then the value is really in the playback and discussion session. But like you, I've also had success doing a whiteboard activity where we just go through with maybe one of the data teams, the kind of technical, hands on person, to create the flow and then we go with their manager independently to create the flow. Then we go with the exec level and create the flow and then you get, like you said, very different viewpoints of how they think things are working. And that's how you get real change at a process level because it's not what people thought it was. And so I've had great success with that. And then it tends to end with a session where we all come together, put them all up and go through and discuss differences, similarities, and often we come up with a new truth, which is then the one that's adopted and implemented. So it's normally some Frankenstein's monster of all of them, but it turns out that's the one that's useful because it brings together the different views people have and brings together the best of each way of working. And so that's really where I think the best lessons learned are and how you actually make change using this as a tool.

And I can imagine when you're doing that again now you've got say three maps, right? To keep it really simple. So the three expectations of how the system works to help get agreement, you can point to parts of the map, you can point to the executive part of the map where it's got the word AI agent, and you can say nowhere in the other two maps does it have any idea of LLMs or agentic behavior. So we need investment, we need to add that node because it's just missing. We're not investing in that right now. So by pointing to things in the maps, you can get agreement to add things and then you can get agreement to take things away, I'm assuming.

Yep, exactly that. And another similar example is where you might just need to end up updating their business glossary. So it might just be, and this is real case where the ic, so the technical contributor versus the executive had different definitions of data set and so their flows look very different. And it's just because dataset was taken for granted to mean data set and they. We just went into the business glossary, added a new entry and moving forward communication was easier because they were talking the same language. So you can both modify things and update how they work.

Yeah, actually that's a really good point, is that when you're doing these maps, you need to be iterating bunch of definitions, a business glossary so that if there is a box, if there's a node and it's got a word, that word needs to be defined because otherwise everybody's going to look at the map and go Christchurch, I guess Christchurch, New Zealand. No, it's Christchurch in the UK end Dorset. Oh, okay. Actually it needs to say Christchurch, New Zealand and Christchurch uk. Otherwise we're going to look at different part of the map thinking it's the same thing. I think two other areas of seeing value during this process, when I get bought in to do data blueprints for organizations. I will create these as a way of articulating a story like you said. I will help it for me to understand the thing I'm trying to map the system I'm trying to define as a blueprint and then use that to test and iterate and get feedback on whether I'm on the right track for what the organization thinks they want. And the second one is when you go in and do a review or a stocktake, I do exactly what you said as well. I will talk to the technical people and get them to help me draw the nodes and links diagram or get them to do it. And then I'll go read the technical documentation, the solution design, any documentation and see how well it conforms to their understanding. Because that means one of two things. The documentation is out of date, which is typically the case, or the people have an impression of how the system's working, but that's not actually what's happening. So those two use cases for this pattern I found is really valuable as well.

Yeah, I think we've got a very shell experience with those. The last one which just came to mind was there's an educational piece which is really interesting in that when we talk about these nodes and edges, producers, consumers, as I said, I like to use them for different use cases. But for instance, executive level people may not know what a data flow diagram or data lineage is, but if they're happy with this as a conceptual map source, data warehouse analysis, so on, then you can give the example, for instance, of if you instead make the nodes data sets that then if you think about it, it's really just a data flow diagram or data lineage diagram. And so there's a nice educational piece where something that they're comfortable with can help them understand something more technical that they don't run into. And I think that's actually never the intent, but always a nice side effect of using these.

I think we need to be careful there though, because all data lineage diagrams look awful. They look incredibly complex, they look like a Frankenstein version of the London Underground. And therefore we need to be really careful that telling that story has some value because it's a tool for data professionals to be able to go into the detail to find the bit of nodes and links that they need to solve a problem. I think it's like data models and ghosts of data past. The enterprise data modeler would print everything out on A or 0, have it up on the wall and be very proud at the size of their data Model the number of nodes and links and how complex it was, nobody else gave a shit. In fact, it actually did them a disservice because it was like, I don't understand that. And I think we've got to be careful with data lineage as well. That actually it's a tool for us, for data professionals, it's not a great map for information consumers. So I'm with you. And understand the concepts and people can look in and go, oh yeah, it looks like a really complex version of what I understand. I'm probably not going to go near it because I don't need to.

So yeah, exactly that. When I say to help them understand what it is at the conceptual level, it's not an excuse to start showing up with those lineage diagrams. However, it does mean if they know what a lineage diagram is and consequences for contracts, observability, things which can affect cost value, have implications, it's then easier to get time or money assigned for those types of projects because they have some awareness of it versus you. Rocking up out of the blue. Our observability is a mess. So I fully agree. I'm definitely not saying to try and turn the execs to data lineage experts, but again, just to increase awareness. Something I say a lot is just as companies want to become more data driven, data engineers should want to become more business driven. And so it's all about that. Just I'm not expecting data engineers to know the business through and through, but we should be aware of key metrics, processes, streams of revenue, what have you. And so I think it's just about the conceptual awareness more than anything else.

And I think the other thing is the complexity of the map also tells me a story. In our product we run a relatively simple three tier architecture. We have history, design and consume. And then within our design layer there is effectively three objects you can create. You can create a concept object, which is a list of keys for a thing. Customer, supplier, employee, product, order, payment. You can create detail about it. Customer, name, product, sku, order, quantity, payment, dollar amount. And you can create an event, a relationship between them. Customer, order, product, and that's it. There's only three types of objects you can create. And what that means is when I look at our lineage graph, it may have lots of left to rights, but the number of columns in it is very light. Which means when I have to troubleshoot, I have a very short conversation with myself. Is it the history layer? Is it one, the concept, detail or event and designed, or is it the consume layer? When I go look at other organizations and we're creating transformation code that has lots of create tables, temporary tables in between, and I've now got 16 columns for a relatively simple transformation. That complexity comes with cost, and we've got to really understand that. And then the other thing we can do is, if we think about it, if these maps become context, if they become metadata, if they become something we can query, we can actually put a boundary around a map and say, hey, if we replace the source system, how many of those nodes and links, how many of those consumer and producer ensembles need to be touched? 250 out of. How many out of a thousand. Okay, so what we're saying now is we actually have to refactor 25% of our entire state and we can get a sense of the cost of that change at a really high level. Not in detail, but we can start to really understand how much of an impact on the system we're going to make when we make these types of changes.

Yeah, I don't have anything to add. That's exactly right.

Which again comes back to actually turning this context, this metadata, into actual data we can use is probably something that we should think about a lot more. Because I'm like you, I just draw them. I've thought about automating them to make my life of drawing them easier, but I haven't actually thought about using them as a global repository to help me make better decisions.

Yeah, I think there's something really interesting there because as I said, I see these very much as a complement to those other artifacts that already exist, your Org charts, your architecture diagrams, data dictionaries, business lotteries. So I think there is a really powerful layer there that if you could bring all this together as your context layer and then use that again just as context, I think there's something really powerful you could do with that. I don't know of anyone or anything that's implementing that or even thinking about that yet.

And it comes back to EA Sparks and all those standards and the tools that complied with those standards or applied those standards. That's what they were doing. They were bringing all these different dimensions of lenses about everything we know about an organization. The problem was the tools were just horrible to use. They weren't friendly, and then the diagrams they produced were ugly. So I think that's the key, is you've got to make it easy to create and you've got to make it easy to consume producer and consumer, but do it in the way you define your system, as much as the Way you define the way you work, the way you define your technology, your architecture and your data flows. So yeah, I think that's a good.

Point because that's really what I try to do. I'm a big fan of Dylan Anderson's posts about people, process, technology and data, the four pillars. And I always use these in that context of bringing those four together and how you build a data capability or data strategy, which in turn helps you achieve your business strategy. Right. So it's all about that kind of hierarchy of do what we can and hope that we've done enough for it to propagate up into something more meaningful.

I agree. So when I do the data blueprints, I focus on team design and ways of working flows of work as much as I focus on architecture because I got sick of strategies or designs that were just a bunch of technology boxes and none of the other things that were important. The other thing I do add though is I start up with measures of success, which is if we're going to spend all this money changing what we do or implementing this new platform or whatever, what does good look like? How do we measure the investment was worth it? Is it increase in the number of self service being done by people outside the data professionals? Is it data or information being delivered faster or of a higher quality? What does it actually look like if we are successful after we spend all this time and money? Because I'm surprised the number of people that don't even think about that. They just deep dive straight into the architecture map.

Yeah. Whereas I think again I'm fully aligned with you. Stuff like the architecture map are almost a means to the end. It's how you achieve the goal, it's not the goal. And so I also start with the people and process driven part. What do we want to achieve, ways of working to achieve it, how do we know we're successful? And then as I said, things like then your team design or responsibilities, they then inform my architecture choices. A lot of times I'm never going to suggest to someone who's got two data people on their team to try and pick up, you know, enterprise level data platform plus storage plus bi, whatever the case is, it's all about almost the minimal system to achieve the goal and to hit success. And that's again why I think this just helps my thought process. Keep it simple, keep it tractable, just have impact versus bells and whistles which you can always add on later as and when you need them.

And again I go back to that diagram you did. I can Also see ways of identifying who we're going to hire. And the one that you drew talks about Power BI being as the semantic BI layer, PowerPoint, Excel being as the primary reporting last mile tool. Then you're going to hire analysts that are used to using GUIs, draggy, droppy, those types of things. Maybe a lot of Excel, which makes sense going back to the DAX formulas. But if you get somebody that's hardcore Python coder who wants to just use a cli, there's going to be a mismatch between the system that's in place and their expectations. Now, you can deal with that by giving them a different set of tools. But now we're going to have this conversation of how does the hell does a CLI with Python code talk to the semantic BI layer? Because sure as shit, they're going to want to punch back into the data warehouse layer and use the data, or even back into the lake, because that's what they're used to, the way they're used to working. And that's okay, as long as you understand there's a mismatch and you're going to have to change something. But if you go in there thinking you don't, now you've got a problem, and I can point to you where that problem is. It's a mismatch between skills of your analysts and the system you've built. So I think that's important. Again, taking these different maps, different dimensions, and being able to compare them.

Yep. And I think as a consultant, it gives me some validity that I've not just given them a list of tech, I'm not just giving you an architecture diagram that you probably don't understand. I've given you not just the tools, but the Personas, the workflows. I've talked through how I went from your requirements to this proposed solution. And again, it's transparent. It lets you have useful discussions with people, it lets you align their priorities, whether it's cost, people, performance, whatever it is. And that's why I really like this. And as you said, that kind of mismatch, identifying it early, you then get to make an informed decision. I know I've said informed a lot. That's what these diagrams give you. They give you informed decisions from the start before you're too committed to anything.

And it's also a decision that can be challenged because again, I can point to a box, I can point to a node, I can point to an ensemble, I can point to a line, I can point to a consumer producer Peer and say that doesn't make sense to me. I can point to a handoff and say that looks like it's missing something or that handoff looks like it's waste. I can now start to challenge some of those informed decisions with an informed opinion and I think that's important. Again, it becomes less conjecture, still an opinion, but I can actually try and get some clarity on where I'm disagreeing.

Yeah, it gives you the scaffolding for meaningful conversation, decision making. It's less about opinions or people, they're still opinions, but it's less about opinions without context and more about how opinions fit into a workflow which involves technology, people, data, all the different parts. Yeah, so I fully agree.

What I remember back in the ghost of data past when we were doing big requirements up front, so I used to hate it. Every now and again there'll be one of these mega projects, transformational things and you used to get a list of requirements and number the bloody things and that was effectively the input into any of your system plans and your blueprints. And I used to map the requirements to the nodes. So this node supports requirements 54B, 27C. That helps mitigate some of the arguments that where did this come from? But I'm not sure the juice was worth the squeeze. I kind of found it waste. What about you? Do you actually map any of these back to requirement statements?

It depends on the level of the requirement statement. So I definitely don't religiously apply it to all of them, but I think they are the requirements in general. Give me the context and that's how I treat it. In general, if the requirement is you need to ingest data from system A that informs some choices. If your requirement is less specific and a bit more hand wavy or conceptual, then I'm definitely not going to slog to try and assign it to a node. I'm going to use it as broad context so that then if when the discussion comes up I can describe my choices in context of that, but that's it. So I think yeah, like you in the early days because this came from experience. The reason why I'm attached to this idea is I was working a job where none of this was in place and so we had to do this just out of necessity. And so I think yeah, I'm like you. I started by very rigidly trying to was one to one map the requirements to the flow and it doesn't work or you end up with very rigid workflow and you don't have any kind of Freedom to make something better. And so now I'm selective over which ones I directly incorporate versus users context.

I think actually as I was thinking about it as well, I often use the requirements to identify where I have complexity in my map. For example, if I have a requirement that data has got to be able to come from the source system, the system of capture or production, and be available in a last mile capability to an end consumer in less than 2 seconds I'm going to have some kind of streaming architecture. And so if I look at your map, that is a typical batch orientated architecture. Like I could stream it, but I'd be really surprised if it would be streaming with those layers. So if I then have to do another flow, if I have to do another row on that diagram, which immediately makes it more complex, and that's only so I can stream, then I got to justify where that came from. And then potentially I could re architect the layered architecture. You have to be stream only if it meets every other requirement. But now can go back to what's forcing me to have that complete complexity and is there any way I can remove the complexity without introducing more complexity? Because sometimes having only one row means it's trying to do too many things and therefore it's even more complex. You're just hiding it. So yeah, I think actually thinking about it, those broad brush requirements help me again put a boundary around things and say I have to do this for these reasons. If those reasons aren't valid or important, then I can stop doing those things.

Yeah, I also think some requirements are very specific and like we talked about earlier, they might only be applied to a finer grain version of this flow. So when I worked in defense, you can imagine there was quite a lot of strict rules about data encryption and personal information masking. So there I wouldn't worry about showing that for instance the high level workflow, but in the more data engineering focused one, that's where I would include it. So again there's a trade off there in decision about it's almost twice the work, but it can be twice as impactful to have those two grain views of this one system. One for the kind of exec level, one for the technical level, but again.

That helps the collaboration conversation. So if you tell me that we need to mask people's personally identifiable information in that scenario, their names, their date of births, maybe some of their deployment information, I'm going to ask you where are we masking it? Am I masking it in the lake, the warehouse and in the bi semantic layer? Or are we saying actually the lake can hold the raw data? When we get to the data warehouse layer, then we're going to mask it, which means we now need to control access to that data lake layer that only certain people with certain authority can see the data in there. So now I have a boundary. Nobody's allowed into that layer unless they pass a certain security level. So again, they're just helping us make Tracoff decisions and also have a conversation about what happens where and what doesn't. And what does the contract look like?

Yeah, exactly. And then again, it gives you that extra view, like talked about kind of access control of if you're in the cloud, there is a different view of this diagram where you might talk about networking or access management. And again, the exec level probably don't need to see that, even if it's a requirement. You might just have that as a bullet underneath with a check mark. But then in the more technical view, you really say, we've made this group which has these permissions and that's how we've satisfied the requirement. So again, it's all about controlling the information you present and which grain you capture those requirements.

So this idea of nodes and leaks has been around for ages. This idea of system thinking that came out of lean manufacturing, it's been around for a long time. The idea of business process mapping and understanding the flow of work been around for ages. The idea of enterprise architectures and diagrams that hold the ability to tell different stories at different levels been around for ages. Why do you think in the data domain it's very rarely used.

It's a great question. I don't know the answer. I think in my experience, data people have always, for whatever reason decided not to learn from, say, software engineering, data engineers. And so data folks tend to be, can be very technical, but they're almost very cultish and like they do data. And I think there's very few people relative to the number of data folks who actually understand the business and how processes work and how to communicate that. So I think one of the biggest things I learned from academia that I've brought to my career is communication is presenting to multidisciplinary audiences. And I think that's something that is just skipped. If you go from undergraduate degree to an internship and you just. That piece of learning is missed a lot of the time. And so I'm not saying that's the only or the main reason, but it's a practical reason I think that I've seen in my career that prevents this kind of thing from picking up traction.

I think we segment the work into hyper specialization and then we start at the lowest level. So we introduce the idea of a data engineer that's going to write code using Scala or we bring in that you're going to be an analytics engineer that's going to write a model in dbt. And we don't teach end to end system thinking as a framework, as a set of patterns that you've got to understand first before you can go and do the work. If you walk into a factory, and again, I worked in the factory as a kid 30 odd years ago, but you walk the line, you'd understand the flow when you got on boarded, the flow of work from the beginning of the factory to the end so that you understood where your station was, what your part of it was. And I don't think we do that in the data work. I think the other thing is, and one of the reasons I wanted to get you on is that article you wrote, it was simple to understand for me at least the thinking is really aligned to the way I think and that probably helped. But what I often find, especially if I look at academic writing, it's really research based, it's quite technical, it's lots of complex ideas that aren't distilled in the story I can understand. I read them and I'm like, ah.

I don't get it.

And so to take that complexity and write it with simplicity is actually really, really hard. Again, big ups to you for writing an article that distilled what is quite a simple idea but can be complex down into something that's easy to understand in the written word.

Yeah, I think part of that comes from the kind of. Even though I've jumped fields a lot in my career, the one consistent has been an interest in mentoring and developing others. And so to do that you really need to be able to make information accessible. Whether that's just choosing the right type of diagram, using the right vocabulary, whatever it is. So a lot of these ideas that I'm drawn to, yeah, it's often something which I've not seen someone else explain quite simply. So I take up the challenge to do it because I think it has value to. Even if only two or three people read that and say I now get it, I'm happy with that, that's worth it. But I think that kind of thinking, as you said, the systems thinking, the context, being able to understand how these systems come together, the modular parts that they're made up of and then how to communicate that not just between technical teams, so not just between data engineers, but also data engineers and data analysts, but also data engineers with managers with C suite. That is, I think, one of the biggest gaps. And especially now with lots of buzz that you see about we don't need junior engineers. There's not just a technical deficit there just because you can outsource some work to LLMs. There is a real issue of new starters not picking up in quotes the soft skills, communication skills and learn how from the start to talk about these ideas, to communicate these ideas. And I think that's something which is a concern for me and something which I'm. Podcasts like this, I think do a really good job of starting to address the gap of giving people another avenue to learn something like this quite easily.

Yeah, I think there's the whole argument right now about role of the junior or the role of the senior. When we all get 10x, it's going to be interesting because I think education has to change because the skills that we learn, the technical skills of how to code, are going to be supported by the tools of the future. Which actually just leaves us the system problem, actually understanding how to daisy chain those tools and that code together to become far more valuable. Just go back to that writing process. I find it relatively easy to write complex words and you can just brain dump and I can just write lots of complex stuff. But to then refine it down into something that is clear and simple and reduces the complexity and increases the cognition when you're reading it. I find that the hard effort, that's where I've really got to focus and iterate time and time again and spend my time. Is that what you find or is it like the way you do your diagrams? You find it slightly different in terms of the process you use?

I think it's quite similar. Again, from my academic work, I would write quite technical scientific papers, but then I would convert those into PowerPoint presentations for conference presentations and stuff. And so I carry writing stuff like this the same way. I go from a technical idea that I understand and then I always start by creating the diagrams that capture what I'm thinking. And then the words I can just naturally sit around that because once I have the core concepts in images, I can just walk through the process of what do you need to know to understand this image? And then what do I want you to take away from this image? And so that's always my writing process of taking some complex and making it more accessible.

I think again, that idea of writing some complex words and then drawing yourself a simple map and see what's missing or what needs to be added, that helps that visual to written and back and forward.

That's why I always have a notepad on my desk, because I always find, even if I'm reading or learning something, if I can draw it out, I can understand it. Because I'm a very visual learner, like you said. I think so, yeah. That's how I go through this.

So if people want to follow you and read what you're writing and get some more of these cool ideas, these cool patterns in a way that's simple to understand, how do they find you?

So the two best places to find me are on LinkedIn. So Robert S. Long, I think is my handle, and on Substack At Long Last Analytics. So that's the name of my consultancy. My surname's Long and I like cheesy things, so I like the at Long last I've solved your problem aspect, so that's where you can find me.

Yeah, I've been reading your stuff for a little while and before you mentioned that, I only just got the joke as we're doing this podcast, I was like, ah. Actually hold on, that's your last name. I like Cheesy as well, but normally I pick it up a lot quicker. Excellent. All right, anybody wants to read what Rob's writing, go to Oglast analytics on Substack, hook him up on LinkedIn. Otherwise, I hope everybody has a simply magical day.

And that Data Magicians was another Agile data podcast episode. If you want to learn new patterns that you can apply to craft your own agile data way of working, head over to agiledata IO.