AgileData #39 - Data Lineage Patterns - Tomas Kratky

Welcome to the Agile Data Podcast, where we talk about the merging of Agile and Data ways of working in a simply magical way.

Welcome to the Agile Data Podcast. I'm Shane Gibson.

Hello everyone. This is Tomasz Kratky. Or for simplicity, just Tomas.

Hey, Thomas, thank you for coming on the show. I think this one we're going to go into that whole conversation around Data Lineage. So you've worked in that space for a long, long time. But before we kick into that, why don't you give a bit of background about yourself for the audience so they can understand who you are and where you came from.

Thank you very much. And let me start by saying thanks for having me. I'm a big fan. I must say that I listen to the podcast and I'm one of the fans following your post at LinkedIn. So I'm just really happy to be here. A little bit about myself. I'm very traditional software engineer. I started coding in Java and building small systems and larger systems and actually get to data or go to data very, very soon when building some enterprise applications for my customers back in Czech Republic. It was basically 2000 and I am in love with Data since that moment. It took me a while to really start an independent company. I had an opportunity to work for a growing professional services custom development business back in Europe and then I got an opportunity to start something completely independent. I took that opportunity. It was 2017 or end of 2016. I'm on my Menta journey since that moment.

Yeah. And so if I understand it right, Mentor is 100% focused on data Lineage. That's what it does. That's. It's meat and potatoes.

I always say. Data Lineage basically not the end. Right. Data Lineage is just a mean to something. When I started the company, I saw Data Lineage as a fundamental piece of information that if it's collected and if it's used in a proper way, can unlock a lot of power for my customers back then, which is large enterprises. Doesn't matter if it's finance, retail, healthcare, it's all the same. They have large, complex environments. So I saw that as a way how to help them, how to give them more visibility, more transparency. And also that was most important thing for me, how to give them or how to enable agility for them. Because that's one of the secret things about Linux.

If I remember many years ago when I was running my consulting company and we were helping out large organizations, large in New Zealand terms, which is probably tiny in European and US terms, but they were the big banks, the Big insurance companies, the big government agencies. And back then, we were at the stage where, yes, we were buying suites of products. They were implementing the informaticas, the IBMs, the oracles of the world back then. But even though we were implementing suites, often those suites weren't really integrated. A lot of the vendors at that time pretended they had integration in the PowerPoint slides, but they'd acquihired or bought products, different technology stacks under the covers. The tools weren't really integrated. And so we were often either buying or custom building a lineage capability to get that visibility from beginning to end of all that data movement. And so it became really important around that time to have that lineage capability to give us that single pane of glass. And then what we've seen is we've seen the modern data Jenga stack. We've seen this idea that actually, rather than buying suites of product, we now buy lots of little components and cobble them together ourselves, make our own Lego blocks. We end up with somewhere between three to 20 different products that we're integrating to move data around. And we still don't see lineage in that space. We see lineage within those tools, but we don't see that single pane of glass. And so lineage, from my point of view, has been relegated to just a feature, not a category. Which is funny, given the modern data stack is all about carving out new categories that used to be features. Profiling data is somehow a new category. So what's your view? Why has lineage been such a strong capability required in enterprises for so long and then all of a sudden it gets relegated as a DBT feature? If you use dbt, but you can only see the lineage within dbt, you can't see the beginning to the end of your entire platform.

It's an excellent question. So there are a lot of different answers. Okay, like, what is data lineage and what does it represent to me? Because when you look around, you see different people thinking about lineage in a very different way. For some people, when they say data lineage, they think about traceability, how a specific data record is moving in my environment. So when I see number seven on my report, show me specific data records used for that calculation. Some people think in terms of what we call runtime lineage or operational lineage. So it's basically runtime information about a specific workflow being executed, connecting two tables or potentially connecting multiple columns together and producing some results. So that's very useful, for example, for incident management, because when it comes to incident management, you really care only about the workflow executed just A minute ago. But then if you switch your view and you start thinking about more difficult things like change management, that's the most critical thing for every organization. Ability to change things fast and in a safe way. So for change management, what do you truly need is record level lineage or traceability. What you need is runtime lineage. What you need. No, you are missing something. What you need is you need to understand all possible dependencies in your environment. You need to understand how things may go or flow in your environment, what may happen depending on the conditions. So what you actually need is what we call design lineage. So it means what are all possible ways in your environment, how data can flow? Is it the same as runtime lineage? No, you may have very critical, exceptional workflows, but they are only triggered, I don't know, once per year. But if you miss them when, for example, migrating or changing something in your environment, you can cause huge incidents that will be seen in a few months, maybe down the road, but they will cost you like 100, 200 million. So that's the beauty and also the challenge around data lineage and one of the reasons why many people, they simplify the topic significantly and they focus on something that's easy and simple and now it's just a feature because it can do SQL on Snowflake and tell you the traceability for your data. It's super easy, it's super simple. So the other part of the issue is when it comes to LinkedIn and communication about modern data stack, we think in terms of small and mid sized companies, or let's say startups with 500, 1000 employees and they run business in a very different way than let's say big banks or big insurance companies. That gap is really significant. And I must say I have almost zero experience with smaller midsize companies. I mean, I live every single day, every minute of my day working for those giants like Fortune 100, Fortune 500. And that gap is really significant. We keep talking how complex our environment is, but we technically have one platform, one or two reporting tools. Yeah, we are really lazy developers, so we can't build proper DBT models. But now think about those really giant companies. They have, I don't know, our customers, they have 10,000 to 20,000 applications and multiple business units. Every application is like data system of its own. That complexity is just a completely different universe. Which is, by the way, one of the reasons why we were able to succeed back then. Competing with big companies like Informatica you mentioned, simply because it was so difficult that no one was able to do it. And it's basically still the same thing today. That gap, thinking about small environments and applying this to any environment, this mindset is actually the second reason why we see Lineage being seen as a feature, more or less. And then of course, the last piece is how super successful data catalogs are in terms of the narrative, how they can pretend and present that basically this is the one place where you go for anything you need with data and metadata. So I always make fun of them. I say to my customers, I mean, okay, go for it. You are telling me that you can buy one piece of software, they can collect all metadata, they can provide metadata to all possible use cases you can benefit from and help all possible users you may potentially have in your 100,000 employee environment. So, yes, if there is a solution like that, just tell me, I'm going to buy it myself. This is the last piece. How some companies are able to, let's say, influence the market and the way how we think. And I admire that. It's amazing. But unfortunately, it's also, I would say, damaging the market a little bit.

So when you talk about those patterns, you talk about data traceability. So I'm on a report, a dashboard. I've got a number. The number's 42. Where the hell did it come from? Which source system was system of capture? What was done to it? What business rules were applied before I can see that number. You talked about runtime, lineage, observability. Effectively, I'm running some workflows through my system. Tell me what's running, what's not running, where the problems are. You talked about change management, release management, CI, cd, type of things, of. Okay, I'm about to make this change. What's the impact? What's the blast radius you talked about? Effectively, you call it design Lineage, but I call it architecture. Tell me all the blocks I've got, all the different technologies, how do they integrate, where's the data moving, how do they talking to each other? You take those things and you call it Lineage. Whereas in the market now, we would call it data ops or observability or heaven forbid, data mesh. So again, it's. This use of language is important because your view of Lineage is not just a graph, a DAG that says this bit of data goes from here. It is visibility of everything that's happening to data across that organization. Is that what I'm hearing?

That's a very good point because that's one of the issues I have. If you think about it, just show me less successful market in past 20 years than metadata. There was so much promise, so much promise. And I think that every time we did it, we failed. And I have a lot of comments about that. But finally we see things changing and we were one of the first companies just saying please stop thinking about stupid stuff like metadata warehouse and one ultimate metadata repository. We are obsessed and I have the same issue with data. We are obsessed with, with idea of collection. I don't know if it's this western type of thinking that we need to have more and more and more, but we are obsessed with collection instead of being obsessed with usage. I would rather have 1% of data most companies have, but all the data I have should be in use. And there are many patterns how we can put data in use. We always think about that's the whole self service and all that crap. It's important. But I always use my mother or my father as an example. My father is actually a very important manager working in a nuclear power plant. So he needs data on daily basis, but he will never log in to any interface to what we call search for data. He needs data to be integrated, embedded into his workflows. So that's how he can benefit from data. And that's much harder for us data engineers because it means we need to first understand what his needs are. Then we need to understand how to really deliver data when it's needed, how it's needed and how to really embed it into his workflow. Not just force him to do things differently, force us to help him. And the same with metadata. And what you said like a super deep and super clever. Because one of the changes we see, and we were one of the first companies removing the direction is stop thinking about collection. Collection is important, but you don't need to have all metadata. So Manta is super well known for having an open ecosystem of partners. So we work with companies like Collection, BigID, IBM and we not only provide metadata, we also receive metadata. So quality scores or privacy scores or PI, data classification and all that stuff. And now the question is how to use it. And I actually like how Garten call it, they call it like active metadata. So how to activate it. And there are many ways how metadata can be activated and that's the only moment when you actually see use of it and you stop thinking about building one huge place where all metadata, I mean who cares? Who cares? I don't care if I have all metadata in one place or If I have 20 different repositories with some type of metadata. The key thing is to integrate and to activate. And that's, I think, where I see the market finally moving.

So many things to unpack there. I've got a whole lot of notes. So let's take it from certain different angles and we'll keep looping back on this one because for me, this is really, really interesting. So that idea of your dad. In the past, when I've been consulting with companies, they go, we want self service, self service for everybody. And I have a conversation with them about there are some people who want to serve themselves with data, but the majority don't. And Nickerson actually had a really nice article they wrote on it. They talked about self service versus silver service. So this idea of I just want to be served a beautiful meal so I can eat it, or actually, yes, I want to go up to the buffet or go into the kitchen and do the work myself. And when you look at the majority of the people that work, they don't want to play with data. That's not their job. They've got a job to do and they need data to inform that job. And that data needs to inform what should they do next? Or I've got a blocker, what are my choices? And then they just want to get on and do their job. So bit like your father, he uses data on a daily basis, but he's not going to go and play with it. He's okay, if the reactor is getting hot, alert me, don't make me go and do self service to say, how's the reactor going today? It's like, I know that if it gets hot, that's probably a bad thing. Flash a red light. And so this idea that, yes, self service is important, but I agree with you, we're fixated on this idea. Everybody's a data person, everybody's like us and wants to play with it. Where actually the majority of the world just want to use the data to do the next thing they need to do and get on with their day job. And so if we take that to Lineage. So in our product, we've got Lineage built in because we think it's a core feature. Our view is any product that uses data should provide a form of catalog. If it's using data, it should be able to surface a view of that data. It should be able to surf what the data looks like, any context within the scope of what it's using it for, and we'll talk about interoperability in a single view later. The second thing is it should show Lineage if it's doing something to the data, it should be able to visualize what it's doing because there's value in there and that should just be table stakes. I shouldn't need to go and buy a third party tool to be able to see those two things when I have a single system. So as we're building our lineage out, we think about it the same way you do. We think about what action we could take. So one of the things that was really interesting for me was when we built our first version of it and we call it Data Map, we worked out that actually because we're transforming the data, if I could click on a node in that lineage map and say run, run forward or run backwards, it was really valuable because I didn't have to go and find what we call a business rule, but think of it as a transformation. I'd have to go find the search for the transformation and go, that's the transformation I want. Now run it, run it forward, run it backwards. I could go into that visual map, click on it and get it to do a task for me and save minutes, which is what we're about. And we're currently working on this idea of column level lineage. I love the sexy demos every vendor does when they talk about active column lineage or we've got column level lineage, because what do they do? They show you a pretty graph with two input tables, one transformation, maybe one aggregation node and a report. And you go and look at even a small customer, you go look at their production environment and I call it mad persons knitting. There's hundreds of nodes, they're like spaghetti. We thought about, well, there are three actions we want to take and they kind of align with yours, which is somebody's using a number on a report or a dashboard and they've asked that famous word, where the hell did this come from? And what the hell did you do to it? So we want to start with that and just get given a story, a map, a lineage of where that data come from. Second scenario is somebody's got a systems of capture that we're bringing the data in and they're going to do something bad to it. They're going to add a column, drop a column, change a column, remove a table, split a table. They're going to do some bad things that are going to impact us. So if they do that, what are all the things that might be impacted? And then the third one is, I'm about to make a change. I have a transformation or a rule and I'm about to Make a change to that rule. So I'm in the middle of the spaghetti. Tell me everything on the left and everything on the right that could be impacted by this change. And so for me, those are the three actions that a traditional lineage serve. What's your view apart from those three? Do they resonate and have you seen any other ones?

Definitely. That's a very powerful start. When I think about a lot of customers, basically the first thing they need and what they want, and it helps them with many different use cases and in many different scenarios. Let's assume you have really detailed column level across simple technologies. You understand semantics of those transformations. You understand what's actually happening to your data. You're not just connecting two dots together. You actually really understand what's the calculation, what's actually happening to this data point and how it's transformed. So if you have all of this, I believe me personally, the most powerful subset of metadata you can get. And you can start thinking about a lot of very interesting use cases. Maybe you have tables in your solution. You have labeled what are really like key data sources. This is really key data source. Or you can label it automatically by using Lineage. Because you see that 90% of reports is using data from this place. And now in your solution, I bet that you have some profiling capabilities, so you have access to information about, let's say quality of data going up and down, or potential some data quality issues on the horizon. So what you can do is immediately act on it and you can immediately notify the user of your report and tell them, be careful in next 60 minutes, I would not use this report. Or you can completely shut down the report and just do something instead. Okay, give us 60 minutes. We are working on it. So you can be proactive and it can be fully automated. And that's what we see with data as well. You use data in your workflows and processes to fully automate them. That's what we want. So we are doing the same with metadata. You can think about optimizations. You actually understand all possible ways how data can flow in your environment. You understand density, you understand the example I mentioned earlier. You understand that like most of the data for your dashboarding, analytics and reporting is coming from this place. And you understand critical path for your data and you understand maybe overloaded components in your architecture. And you can use all of that to optimize because you have all that information. So you can do suggestions and recommendations how those workflows can be done maybe differently, or you can proactively notify One or two people who are responsible for the architecture that there may be an issue with something, because again, you see that the thing was changed 17 times in last one week. So something is off, so someone should check on it. Or another really powerful example is you can simply say, what are the workflows that you can completely decommission? Because you may have flow of data and there may be a table in the end, but there is no outflow, no one is accessing the table, There is no report connected to the table. It may be sitting there. Last week we had a very interesting discussion with one of our customers. And just applying this use case to a subset of their environment, they are saving something like US$100,000 per month. Just a stupid thing like that. It's stupid, but that's exactly how, if datalaunch is used properly, how you can unlock a completely new universe of use.

Cases that cost one's really interesting because in the old days we'd have a server sitting in a data center, or heaven forbid, in a cupboard. It would have a database on it. Would you be using Oracle SQL Server? Or it could even be teradata. But it was a sunk cost. So if we were running transformations, if we were running workloads, the impact was other workloads ran slower, the reports ran slower, or if we segmented the warehouse out properly, our transformations ran slower, and we had to optimize the schedule. Nowadays, every time we run something, it costs us because we're using Elastic compute. And so we talk about the cost of these cloud analytics databases. And yes, there are some simple things you can do. The wait time, the ability to shut it down after a minute of inactivity rather than 30 minutes. So that way you're not giving away free credits. But the other one we often look at is, okay, we're doing all this compute on this data that's not being used. And so it comes back to this idea that you've got that actually when we do anything around that data ops, that lineage, that observability, it should be based on a use case that has an action that we want to change. And what I'm seeing is most of the conversations around Lineage right now in our market are around data engineers or analytics engineers. We're talking about lineage for ourselves, not lineage for our consumers or our stakeholders. And so it's not a case of let's identify the data that's used the most and see how we can make it more accessible or faster on the end report or actually, what are they clicking on the report because maybe we can alert them when that behavior happens so they don't have to go check it. Move from silver service to that silver service. So I think that's an important pattern. Whenever we do this work, we should be thinking about our consumers and our stakeholders more than we think about ourselves. That seems to be the way that you think about it with your customers.

Yeah. And I think it gets me back to the pattern thinking and for a second getting back to data itself. How we make use of data, One very simple way is we give data to people and we give them an opportunity to search in data. So I can build my SQL or I can do something drag and drop, doesn't matter, I'm searching in data for answers. Or it can be a sexy Google type of interface. Doesn't matter. That's one very simple way how we can make use of data. But then when we really want to help people, we need to think how exactly. It was the case with my dad, how to transform data and prepare information they need and how to deliver it. So it's part of the workplace and workflow. So they do not need to leave it, they do not need to do anything extra. So we all have this like beautiful examples when we open our mobile banking application and we have some really useful recommendations or insights into our spend from our bank instead of just giving us access to all transactions, which of course we can do. So thinking about it differently, thinking about the end user, thinking about who can benefit from from data and how to really help them. And the last case is this process automation thing. Data is powerful, we can use data to automate a lot of things and then when we move to metadata, it's exactly the same scenarios. So we can either have metadata somewhere and let people use it. And that's exactly the catalog use case. You use catalog and you use metadata in it to search for data to find data you need. And that's very powerful use case. For some user groups they need new data sets, they need to build new analytics and they are using catalog to find data they need. But in the same time the examples I gave you, it's more like don't even show metadata to anyone, use metadata to do something for other people. And that's the real power. That's what I believe we must do more and more and that's how we can truly fully become data driven organization. Otherwise we are just like bullshitting people. Sorry for my language.

That's right, I call bullshit all the time. So it is a complex problem though. I Remember the previous generation of data catalogs? So these are the ones, the elation and the calibras. Their whole premise was back in the days before it was called AI and maybe it was called machine learning. It was definitely after data mining. Problem was we have a catalog and when we install the catalog, it's empty, there's nothing in it and we've got to populate the catalog. And we learned that going there's 3,000 tables and 30,000 columns, or heaven forbid, you bought SAP in. And it's just a plethora of mess cataloging all that information so it was useful, was never done. We would never throw a thousand people in it for 10 years to do that work. So we ended up with these empty catalogs and then there was a pattern that came out which is we'll just throw the machine in it, we'll throw machine learning at it. It will auto tag everything. And it was a great idea. But every enterprise that I worked with where they went to implement one of those, we never saw the value out of it. What happened was for some reason the catalog was never populated. And there's a whole raft of reasons. It wasn't the vendor's fault, but it was the organization, the process, a whole lot of reasons. So that's hard. So that problem of saying, I've got this noise, how do I get the signal out of it? And then we look at the complexity of all the technology stacks that if we were providing a single plane across all of them, they all use different languages. SQL, Python and even within SQL. I remember we used to laugh when I was at Oracle about everybody follows the standard, we just follow our version, we follow the Oracle version of SQL and Microsoft, those evil people follow their version and then we get into complexity. So people love to automate things. So when they're writing their transformations, they'd bring in templates that are looping. When I was at SAS with a macros, when you tried to get the lineage, it was like, well, the lineage says coolest thing and. And then 50 columns turn up because we were using config driven design, which I think is what Gartner calls active metadata now. And so there's all this complexity and if I look at what you've done, the fact that you've got large vendors actually licensing your product to do the work that in theory they could do themselves, just shows that actually that problem is so complex that often they will just insource it to get that kind of value out of it. So why is that? Why is building this visibility, this observability of Lineage. For those use cases, we talked about traceability on a report, watching the runtime Lineage and seeing what it does, change management and that architectural view of all dependencies. Why is it so hard to build a solution that delivers these?

That's actually a very good question. I keep thinking about this last couple of years. The reason number one is if you think about basically anyone who is successful in metadata market today, they either exclusively focus on smaller companies and by smaller I mean 500, 1000, 2000, maybe up to 5000 employees more or less. And these companies, they run on one Snowflake instance or one Databricks instance. They have few reports or maybe a few hundreds reports. But it's similar, simple, what I call vanilla SQL environment. And we pretend now every organization out there will adopt DBT and now everyone will migrate to Snowflake. But hey, what Snowflake is already supporting Python and if not today will be soon. And today it's Snowflake. But around the corner we already have two or three I bet really promising technology is going to replace it. I did it myself four times already. Cobalt Teradata, Teradata to Hadoop, Hadoop to Snowflake. Right now it's just, it's happening again and again and pretending that it will be different this time, it's just being stupid. So one part of the issue is that we have technologies evolving very quickly and no matter how hard we try, it's just very, very difficult to agree on anything that would be one step standard. So we will all every company around the globe will do transformations this way. And now it's easy and simple. But even like thinking about Oracle or Microsoft, we were discovering while building scanners, we were discovering how and just I'm switching to computer science language for a second. How the published grammar of Oracle PL SQL is incomplete and how you can write stuff in Oracle that's not in line with the grammar itself. So if you start speaking English and using completely made up words and it's working and people using it, and that's not uncommon. People think about these very old technologies, everything is stable, but it is not. Oracle is one of the most successful companies out there. They keep developing the software and they are doing crazy, crazy stuff. So that's one part of the problem. The variety of technologies and lack of any standards. And it's really what you said, it's beautiful idea. Every data system, no matter how big or small, should provide its metadata, including lineage. Unfortunately, if you really think about it, for anything that's even close to custom development, it's extremely difficult. So once you allow anyone to write a Python script and use it as part of your transformation, you are doomed unless you work with someone like Manta. That's one part of the complexity. The other part of the complexity, especially lately, is how we merge together, which is great, I love it, because I think there is a lot we can learn from software engineering and data engineering. Not just copy it, just learn from it and adjust and use it the way we need. But as we merge these groups together, we are moving to completely different type of languages and environments and it's creating even more complexity for data processing and for any form of visibility. And one thing I really deeply believe we heavily undervalue is we expect a lot to be done with data. We have great ideas, we have a lot of requirements, and these things cannot be done with few simple SQL statements, no matter how powerful your database is and how you can avoid joins and whatever. It's just a minor issue. Doing joins or not joins, just a minor issue. We are trying to express complex things. So why we are surprised the end result is complex. We use terms and I do it myself and I hate myself for it. We say spaghetti and it creates an impression that the way data architectures are designed and built for most systems is bad. It sucks. And in some cases it does, but in many cases it does not. It's just very complex requirement or very complex set of requirements from thousands of different users, thousands of different requirements and you just try to help them. And now you are surprised that your environment is complex. So that's one piece. Even if we have standards, even if we have one platform and we run everything on it, which is just very hard to see for larger enterprises, you still have the last piece. So we want a lot, we have it.

I do a lot of work in the agile space and one of the questions I constantly ask myself is why are the patterns we use from Agile and data and analytics so far behind software engineering? And why do they kind of fit, but they kind of don't? And it comes around to that complexity. If we wanted to reduce the complexity of our data platforms, we would force our systems of capture to store the data the way the end user wants to consume it. Not for data entry, but for data consumption. And if they could do that, then we don't need to do half the work we do. But the organizations are making trade off decisions. That's why when I look at Data Mesh and I go the core four principles, they're talking about data as a product, federated data governance, they're all core principles we've been trying to do for years. The challenge is if we push that data work down to the software engineering teams where it should be, the question we've got to ask ourselves is will the stakeholder who's funding that work, that change agree when the software engineering team comes to them and goes, so we finished that feature, we're ready to push the feature out to the customer to test it and get some value back. But we can't do it for three weeks because now we got to do the second bit, which is actually make the data for purpose for our internal consumers and sure as shit the product owner is going to go push it. I want the value. So unless we can find ways of taking what we do as data professionals and embedded in the software engineering practice, so it just disappears. This idea of a magical data library like any of the other development libraries they use, it is always a two stage process and it's a trade off decision that every organization makes if we look at that idea of standards. So I do remember many years ago we had the common warehouse meta model. The idea was again, this shows how old I am. We had Business Objects, we had Cognos, we had Oracle Discover, we had all these really great tools back then and we wanted a way that they could share their semantic layer. So the idea that I'd done some semantic definitions in Business Objects universe, why couldn't I actually share that semantic definition with the end user layer of Discoverer? And we're seeing that in the metadata world now with Open Metadata, open Lineage and blah blah, blah. But there's no incentive for any of the vendors to actually play nicely together. In fact, there's a disincentive. I know that last year for the modern data Jenga stack, we saw some startups play with each other nicely, but that's only because it was free money, they didn't need to compete and actually working together made sense because you build this little moat around everybody else. We'll see how it goes this year when they start running out of Runway and they need to actually eat each other's lunch. But in technology we've had a few standards like SQL which even though they are variations, it has become a de facto standard for a lot of data work. What are you seeing in that metadata standard sharing domain? Do you think we're going to get there this time or do you think we're going to fall back to the age old behavior of I don't want to share my stuff because I'm opening up, I'm preaching the moat effectively.

So it's an excellent question. I think that we have couple companies at least trying to play this game from really quite selfish reasons. But that's good enough reason for me. So we are actively participating in Nigeria, that's IBM organization or started as an IBM project. There are many like big players involved. But one disadvantage of this, let's say movement is its complexity. On the other side of the spectrum we have Open Lineage or Open Metadata. In both cases, if you look like at Open Lineage, it's just funny thing, it's so simple that it doesn't help. That's actually like one of the really key challenges. We designed the standard in a way that we basically kill every possible use case except for incident management. And it's because people designing the standard are thinking about this use case in the first place. So runtime Lineage, runtime view and potentially addressing issues when they happen. And we are also actively participating in this and we want to do all we can to move it to the next level. And I really truly believe in standards, especially open standards and open projects like this one. And we dedicate a lot of resources, we try to help as much as we can and in our product we have a lot of planned investments how to really support the Open Lineage standard, as I said, as much as we can. So it's a really big thing for me and I really truly believe that we can make it happen. Maybe it's not going to be perfect, but way better than like just think about today we basically build integrations or we have partners building integrations with every possible data catalog. It's just very typical use case. We love this Lineage, but we want Simplified View to be available to data stewards and users of a data catalog. Yes, I love the case. I'm more than happy to integrate. I believe in open ecosystem. But now it means you go one by one and you are building these integrations or your partner is building the integration. It's like everyone is so tired of it, it's just wrong. It shouldn't be like that.

When we look at the pattern of an enterprise integration hub, when we said we have 150 different applications in our organization and we need to integrate the data for operational reasons, not warehousing reasons. And we said look at the spaghetti that person's knitting. It's a peer to peer interface between 150 and 150, which was just chaos. And then we looked to bring in these integration hubs and somehow they became expensive. That you often hear of a five or $10 million project in large corporates to put an integration hub that nobody ever used. So it's the same problem when we talk about metadata sharing. And it's not even just about the metadata of what table holds. It's not just the metadata of the lineage of how that data moves. For me, it's about the rules we apply. Often we'll get asked if somebody wants to move off our platform to another platform, how do we help them? And I go, well, we're not going to give you the code we execute because effectively that's our one thing that we treat as proprietary to us. But what we'll give you is the rules that you've applied to execute that code. Because we effectively operate a layer system. So we have a language that then converts into another language that converts into the actual SQL Python that we execute. So we won't give you the SQL, the Python, but we'll give you the other layers. We've been experimenting with exposing that language using a framework called Gherkin. Right. So given this and. And then do this, and so it becomes a quasi natural language. But the challenge is we can give you back your dictionary of every rule you have applied in a way that you can understand it, but you still then got to rewrite it. And it cracks me up when people go, don't go for proprietary platforms because they're closed. And I'm going, that's cool. But you know, you run on Snowflake, that's not open. It does execute SQL.

Yeah.

You've now got 3,000 DBT models. Well, let's call them blocks of code because they're not actually models. And language annoys me, and that one annoys me a lot.

I agree with you. It annoys me as well, very much.

Just call it a code block. And so I've got these 3,000 blobs of code that five or 10 people have worked on that actually five of them have already left to go and get the next job. And somehow you're going to move that across to databricks. No, you're not. Because it's not about the execution platform, it's about the replumbing of that logic. And so just like metadata, where we don't have open standards, where we can share, it's the same with transformation. We don't have open standards where I can say, here's my transformation rules, go and import them to another platform and run them if you want to, because that's fear it's not where our IP lives. So I hope we move to a stage where we become more interactive and more open. Not convinced we will, but while we're talking about that idea of this complexity, and you talked about start off with 500 to 1000 people company as being small, for me that's large, especially in New Zealand. But if we look at the complexity we have in those really large organizations, those enterprises, the number of transformations they run, the number of platforms that they have to integrate with, often I say it's just a matter of time. You might be a 500 employee company and you think you're tiny, but give yourself two years and you're going to have 10,000 DBT models because you're just going to keep adding to them. And it's the same. We see a lot with teams building out lineage or catalog capabilities. So they often think about the short term cost. I talk to organizations about the first time you hire your first aid engineer. You're effectively making a half a million to a million dollar investment. And they're going, what do you mean? I'm like, well, show me an organization that runs with one data engineer. There are every now and again a unicorn, but not often. So you're going to end up with four or five. Your fully loaded cost of those will be between half a million, a million dollars a year. So to build your own, you're starting off with that, assuming you get a good team of five. And they actually are incredibly effective. And we see that with lineage and observability of the metadata in that teams will often start off building their own. They'll start by building their own parsers or scanners off the log files, grab some open source stuff and build that. Or I've seen teams do it where they are actually notating their code with notation language and then using something like Sphinx to scrape it out and use UML diagramming tool to show the stuff. And they're all good starts, really quick and really easy to get some value. But when you start then going, okay, now you want to expose it to a thousand users, okay, we're going to plummet into a wiki or a static site generator. Okay, now they want to augment the context of it, or they want to be able to click on something and see what it turns up. Or we want to do some alerting like you said, that actually we've seen some stuff that's happening in the lineage that we need to tell you. Start adding in all these things that are important to take action and you become a product development company. And so given you're a product development company, that just does lineage. Although your definition of lineage for me is DataOps and you've been working on it for years with some really large customers and seeing all those use cases. If you had to guess for a team to build a standard set of capability that meets the normal use cases. Team of three, team of five, how long would it take?

Ah, that's a very good question. I think for simplicity we try to rate companies based on employee count, which is not accurate. And we have couple customers with few hundreds employees but their environment is just very complex and massive. If I think about average company, average small company, nothing exceptional. As you said, five data engineers, one data platform running on something like redshift, snowflake, bigquery, whatever, maybe looker, tableau. So I see couple different things. First one is it's always very simple to do first 50%. In this case we are not even talking about 80 to 20 rule. It's just easy to start. And especially if you are computer science guy, you love the idea of building parsers and now you are finally using that second semester knowledge and you feel really great. So you build your first parser or you get grammar from Internet and you build your first parser and it's plain SQL and it's just super. And then it gets more difficult and then maybe you need to cover stored procedures. And now you are moving from SQL to procedure language and now you have one calling another. You have parameters and you have parameter context and now you need to move from the second semester to fourth semester and especially in systems programming. So that's one part tend to see it as very simple. And then you move from plain SQL and you start dealing with more complex things. We hate Excel sheets as data engineers and we don't understand it's the number one tool everyone who is working with data want to use. But you need to do lineage for that as well. You can't ignore it. So now you are moving from super sexy SQL parsing to wow this mess in Excel sheets with everything you can imagine. And yeah, actually by the way our risk team, they are actually using SaaS analytics because SaaS analytics is by the way really good tool for risk analytics. But SAS is like how do we do that? It's macros and it's wow. It's a completely new universe. So I believe we have spent a decade so far building variety of scanners. And of course in your company you only have subset of those technologies so you don't need 10 years and 100 people to build that. But even with the basics you have, if you have five people building it, and you really want to do it in an automated way, and you don't want to do most of the stuff manually, you will spend definitely way more than a year with five people trying to even cover SQL more advanced use cases. Way more than one year and not even talking about the other stuff. That will take you even longer. And you still have just the scanning part. And now no one will actually use your metadata. Maybe you will use it yourself for impact analysis and incident resolution. Okay, fine. So now you invested a few million dollars just to have a tool for yourself. And you need to keep investing because those SQL things, they develop new features, you need to cover those, unless you are okay with not using them at all. And it's still just very basic, plain use. So what if you actually want to see some value from those millions invested? So you start thinking about activation and real analytics. But that's a completely new discipline. You need to understand very hard stuff like network algorithms and graph algorithms, and you need to really think about business value and about tens and hundreds of different use cases and you keep building and building and building and you need 5,100 people. And it makes no sense to me.

I think for those companies it's like you said, a bunch of smart people that can build smart stuff, they have teams that are big enough that allow them to do it. I'm guessing, because I don't know for sure, but it's easier to hire people than it is to buy licenses. The theory of sunk cost comes in, but it's not visible. A bit like technical debt with Agile Team. The debt sits under the COVID and most people don't see it. And for me again, you talked about changes in technology over time. We've seen centralization, decentralization, we've seen mainframes and then client server and then network apps and then cloud. And we're seeing the kind of rise of DuckDB and people like Mother Duck and that which effectively is for me, another wave of decentralization. We're talking about moving the data down to the desktop, to the browser. Actually, most of the patterns we use for linear genreservability at the moment are based on servers, right? We expect that there is a server that is logging what it's doing and we can pass those logs, or we're expecting there is a server that is transmitting the information we need. As soon as we move into that new world where we're Doing local compute again, there ain't no log, right. We're not going to be harvesting the log files off my PC because it's designed to not outdo that. So unless DuckDB and those tools are now transmitting the logs back to all the metadata back to a central server, which it should do, we've got a whole new architecture to worry about in the next generation that's going to hit us. We're talking about edge node computing now. Again, that's another set of patterns. The cool thing is we've seen it before, so we know what we need just to close it out. I think for me, language is important and when you talk about data lineage, actually you're talking about a whole lot more. And I go for rants about buzzwashing where people take what they used to do and then call it the new thing. Oh yeah, we're a data warehouse company, but we do data mesh. Even though we have central storage, central team, everything's centralized, but we're data mesh. I think you're probably on the opposite is actually you've gone and you do a lot of the things that we talk about in areas outside of lineage, but you're just going, no, we just do lineage. And for you, lineage is traceability of the data that turns up in a report back from where it came from. It's about that runtime lineage that what broke, why did it break? How do we fix it? It's about that change management. It's about seeing what's about to change and what's the impact of that. It's about that architecture. What are all the moving parts and how are they put together and how do we see that? That covers observability, it covers data ops, it covers a whole lot of stuff that the market talks about now. But for you, it's that focus of the next piece, which is, okay, now I can track it and now I can see it actually, what do I do with it? And that's the next generation is what action do I take? How do I use this lineage, this metadata, to actually reduce time or reduce risk or reduce cost? How do I get value out of it? Because I don't just want to store it, I want to actually use it so we get some value out of it. So for me, those are the important patterns that we should look at. We've done some work. What was the value we got out of it? Apart from as a data engineer, I can see my own work because, hey, that's got some value because it makes you faster. But that's not it. What do we get for our stakeholders? What do we get for our consumers? How do we make their life easier? So just to close out, is there anything else from you, your point of view that's really important when we talk about this whole world of data lineage.

Yeah, I think I love the discussion. Thank you again for having me. I think we can continue for hours.

I would probably get back to that thing with standards and actually love how you see metadata and lineage and how you think about it. Basically you try to make it part of your product. And actually that's something we really, really discuss internally right now. How we can change the market so we help people like you and people building software in big banks and big telco companies and healthcare companies or small mid sized doesn't matter. And how to help them to build software in a way that metadata and Linux is provided. I think it's super powerful, super difficult and challenging, but it's something we are becoming super passionate about because that's how we can move from where we are today to the future. Where we think about usage, activation and analytics and not just about collection. The collection sucks. There is no value in collection.

We collect it for a reason. We collect rainwater because we want to water the garden and we want to drink it. Not just so we go, oh look, there's a big bucket of rainwater. So think about the use, think about the value we get out of it. Don't just think about that first step. Hey look, that's been great. If people wanted to follow you, get hold of you, see what you're doing, see your latest thoughts, what's the best way for people to get in touch?

I think the best one is LinkedIn. It's just quite easy. I was one of those early guys, so I don't have any numbers or any strange letters attached to my name@LinkedIn. So you can find me there easily. And I really try hard to be active but as I said, like sometimes hard. We are a bit consumed by our customers. We are not so visible from multiple reasons and we live in the bubble of large enterprise companies. And I just try to share as much as possible, especially because I believe there is a lot we can learn from those companies. We call them in ugly ways but there is a lot to learn from those really experienced enterprise architects, data architects and data engineers working there. So you can find me there. LinkedIn.

I'll put your contact details in the show notes but I just want to re echo what you said is 10 years ago we did things like data modeling because we saw the value of it when we hit the complexity of the data that we dealt with. And then somehow as a profession or a domain we lose it for a while and we just write code and blobs of code and big tables and no model and then it comes back again to return all the data models. We can take the lessons of our past and we can apply them in new way, innovative ways to make it better in our future. But we shouldn't lose them, we shouldn't throw them away, but we shouldn't implement the way we used to. It's that balance of innovation doing new things with old lamps. So look, that's been great. Thank you. I really appreciate your time and hope everybody has a simply magical day.

Thank you very much. Have a great day.

And that Data Magicians was another Agile data podcast. If you'd like to learn more on applying an agile way of working to your data and analytics, head over to agiledata IO.

Sam.