AgileData #33 - analytical team topologies - Ashwin Kamath

Welcome to the Agile Data Podcast, where we talk about the merging of agile and data ways of working in a simple, in a simply magical way. Welcome to the Agile Data Podcast. I'm Shane Gibson.

And I'm Ashwin.

Hey Ashwin, thank you for coming on the show. I've been looking forward to this one. We had a bit of a chat beforehand and we talked about lots and many varied things, but before we rip into that, why don't you give a bit of a background about yourself for the audience?

Yeah, my name is Ashwin Kamath. I am the CEO and founder of Spectre, which is a new data platform that I've been working on for the last year. I've been working in the data space for about 10 years now, primarily in the fintech world, first in a lending underwriting setting and more recently in a systematic and quantitative trading setting. More recently, I started Spectre to really focus on improving the production critical grade data space where, you know, we start to look at more automated decision-making frameworks to optimize and improve different business functions for companies.

Cool. And before that, you've worked for some interesting companies. Do you want to tell us a bit of a background about your journey from 10, 11 years ago when you first started into this kind of data world and the types of teams you worked with and the types of work they were doing and some of the interesting things that you've done over the last 10 years?

Yeah, so I've mostly been focused on the data engineering and data infrastructure side. Done kind of various projects on the data science side as well to get a bit of understanding. So Kind of rewinding back to 2015 time, I joined a company called Affirm out in San Francisco. It's a buy now, pay later company started by the legendary Max Levchin from PayPal. And I used to work on the data engineering side for what we call the bank engineering team, which kind of optimized our and figured out how to take our billion dollar, several billion dollar loan portfolio and distribute that between different capital markets partners that worked with. And this was kind of my first introduction into the big data space. We used a lot of Apache Spark to implement our solutions. It was essentially just a very massive, like, optimization problem to take all these little small microloans and figure out how do we distribute them between different capital markets partners like Morgan Stanley, like Jefferies, in a way that didn't break the covenants that were established in legal documents. And so we had to make sure mathematical constraints were in place. We had to make sure that data quality was exceptionally high because of course, if you sell the same loan twice to two different firms, you're going to end up in a situation where you have a regulatory agency down your throat. And so it was a very high stakes environment, very interesting to kind of see the problems that we had to solve in that kind of a setting. And then more recently, I was at Two Sigma, quantitative hedge fund here in New York, where I'm currently based. And I used to work on the alternative data platform where we focused on ingesting data from thousands of different data vendors, preparing that for the trading modeling teams and doing all the cleaning, the preparation, normalizations that standardization and feature engineering required to really put this data to use in a very automated setting. And the kind of challenges that we faced at the scale of managing 50,000 to 100,000 data processes and still wanted to move quickly on the next set of projects is actually staggering. And there was a whole strategy around how to actually develop infrastructural tools that would allow us to move quickly.

I started my journey in the data world pre-big data. So I was back in the old generation of what we probably call data warehousing or OLAP. And what we saw back then was we saw a real disconnect between what we called the data miners back then. So data scientists of today to a degree and the ETL developers, BI teams. So there was always a disconnect between BI and ETL. The ETL developers would always create the code and create the data structures and hand it over the fence to the people that would visualize it. But they tended to talk about, there was a dependency there. And the data mining teams tended to do it on their own. They may treat the data warehouse as a source, but often they wouldn't, and they'd be in their own bubble. They wouldn't really talk to anybody else. And what I saw with the big data wave come through is a change of team topology. This idea of a data scientist turning up and a data engineer. And I saw some organizations try to find that magic unicorn, that one data scientist that can do data engineering, facilitation, and understand the business requirements, and the statistical engineering to find the right answer of the model, and then the DataOps engineering to be able to deploy the models and build the platforms and all that kind of stuff. And very few people in the world that could do that. So we ended up with pods. Ended up with a pod that had one engineer, one scientist, maybe one platform-focused person, and they just started working together. Is that what you saw when you were in the fire there, building out these big data platforms and using them within those organizations?

Yeah, almost exclusively. I would say basically very similar to the idea of pods, we would have these data science teams that, that really were the most connected. They were almost like ops people, acted as strategy folks for the C-level staff. Within the organization. And then we had data engineers, which acted more as almost support for the data science teams and factored in two ways. So the support was both at kind of the front end of preparing the data and making sure it's ready so that some sort of exploration and ad hoc analysis can be done. But then also after that ad hoc analysis is done, having someone to be able to productionize data science workflows that are developed within a notebook environment and quickly put that into some sort of production environment so it can run in an ongoing fashion. Was actually much larger part of what the data engineering teams that I've been a part of have focused on. And then going even beyond that, in the hedge fund world, we also had this concept of a data support team where the data engineering team was actually so overloaded with these, these productionization requests for bringing in new data, for productionizing data science workflows, preprocessing tasks, and delivering different types of production model, ML-based modeling workflows that the data support team was actually designed to almost kind of handle the triaging of alerts and data quality issues that might come up over time. This structure actually gave the data scientists more confidence that they could move on to another project and 6 months later they could look back and say, okay, my previous project is still healthy, it's working, it's still producing good insights. I can comfortably keep working on my next stuff with next set of projects without having to worry too much about maintenance and that sort of thing.

And that pattern, that team topology of build something and then hand it off to another pod squad team, whatever you want to call it, to productionize it and maintain it, that's an anti-pattern compared to DevOps or data Ops these days, because the DevOps, DataOps pattern is you build it, you release it, you maintain it, it breaks, you fix it, it's your baby. And you're then focused on those things that you're building, that domain, and you're consistent. Whereas that pattern back in those days was build it, throw it, come back and retrain it. But my view was that was because of the scarcity of those data science resources. So you couldn't have multiple teams of data scientists that built things within a certain domain and then maintain them. Is that what you saw?

Yeah, I think that's pretty similar to what I've seen as well. It's a good point you bring up, which is that the DevOps side of things have not really taken off within the data world. You're starting to see it a little bit more, and actually some of the newer kind of tooling that's coming out usually has some side of DevOps associated with it. So that deployment and the infrastructural side of using those tools is actually getting easier and easier over time. But I think there's going to be some similar transformation. I like to equate it similar to what Vercel did for frontend engineering. With Vercel, you literally can push code to a Git and they have their own CI/CD system already set up, and that will automatically deploy on new versions of your frontend application. We have yet to see that really happen within the data world. And I, that's probably what I look forward to the most.

For me, yeah, there's been an interesting team topology change as we jump from the big data wave to the modern data stack wave, where actually we've gone away from pods. We've gone back to teams of one. I mean, an analytics engineer is able to, to spin up a Snowflake, they're able to spin up a dbt on their own, and then they effectively seem to work on their own. They're an end-to-end process amongst themselves. So they're effectively becoming unicorns for that collection of data, the transforming of it, and then the pushing it through. Maybe they don't do the presentation, so maybe the visualization's still done by another team and there's a handoff there, back like the old ETL/BI days. But we are starting to see the, that team of one again, right? And that's where the analytics engineering role seems to have gone. And that's been empowered by technology. But it's interesting that we've seen in the past when we have teams of one, chaos reigns. So people build code, they deploy it, it has value. There's nobody to hand it off to, to maintain it, so they keep maintaining it. Then they leave the organization, their knowledge goes. Or they're doing such a great job, they just get pumped more and more requirements, build more, and they end up with 3,000 bundles of code, one they wrote 18 months ago, and they got no clue what they wrote. They go look at it and go, oh. Why did I apply that business rule? Why am I excluding all our customers that have shoes that are blue? I can't remember why we did that. So did you see that in that transition in the market right now from that idea of small pods of people that work together to teams of individuals, or are you seeing something different?

I think I see it more at kind of the top of the stack at the data science level and less so at the data engineering level. So within the data science level, usually I do see that most data scientists operate more autonomously and take on individual projects. There's still collaboration. In the datasets that need to be maintained that might feed multiple different data science workflows. But absolutely you're seeing that individual data scientists and data engineers are just so much more efficient today than they were even 5 years ago. And that is, I think, really good for the scene to kind of prove out this concept, but also there comes with a lot of baggage. When you have a codebase that has 3,000 different Airflow DAGs or dbt models, you don't even know which ones are being used, which ones are not being used, which ones are required for the highest level SLAs. It's very hard to keep adding to that stack and that foundation without taking a step back and saying, okay, we need to deliver on some human-level change management system, make it easier to move quickly and know which alerts we can ignore, which tables are most important, which ones are less important.

And we see that a lot. We've seen that with data mesh buzzword. It's interesting to see whether my waves of technology and team topologies, I start talking about the data warehouse OLAP waves, the big data wave, the modern data stack wave, and the data mesh wave. And what I see is as data people, we love to focus on technology. We don't tend to focus on team topology, organizational structures, our ways of working, our processes, or what we do. We just, we go straight into a technology solution. And I'm seeing that again. And one of the areas that, yeah, there's been quite a vocal piece on the Twittersphere and the LinkedIn sphere at the moment around the loss of data modeling. And yeah, it's interesting because it made me go back and say, what is data modeling? And well, actually, yeah, people say nobody models, but they do. As soon as you write a piece of code, you're effectively modeling because you're structuring a piece of data as a model, physical or typically a model. But people are talking about this loss of logical modeling, this loss of understanding the business context and doing that design work up front before we get into code. And in the data warehousing wave, we were strongly modeling. I mean, we weren't good at it. We weren't, well, we did good models, but we weren't fast at it. The way I describe it is there was often a person stuck in a room for 6 months scratching their chin or their bed, coming out with this most beautiful model before we could start developing anything. And that wasn't a great process. But then we went into the big data wave. We seem to have lost modeling altogether, but I'm not sure that's true because I'm sure people were modeling, but what did you see? Was there any step in the process back then when people were modeling the business process or modeling the data before they ripped into writing Spark, Scala, Python code to do the cool, the cool feature factories and stuff like that?

Yeah, it's an interesting dilemma. I would say that's been going around the space recently. The event kind of factors into the whole ETL versus ELT debate as well. If I take a step back, looking kind of like way into the past from where I started my engineering career, which was actually not in the data space, but more in application development, these ideas of resource diagrams where you kind of model out your database and kind of show the links and foreign keys and whatnot. It's almost like that has kind of come back in full force into the data space. And I would say like most application engineers have kind of internalized that and there's like very core to how they develop and deliver. Usually the technical design docs will actually like lay out, okay, here's like our database tables, schemas and whatnot. Here's what our APIs are going to look like. I see that as becoming kind of what this whole data modeling space turns into as well within the data space. And the idea being that. There's a different type of, is it ERP diagrams?

Yeah, Entity Relationship Diagrams.

ERD. There's a different way of kind of thinking about it at the service level where you're trying to provide, you're using an API to provide a service to an end customer, an end user, and then using it in a more analytic setting where it's actually an internal representation of how the company thinks about the world it belongs in, right? The environment, the competitive environment, what have you. And being able to separate those two concepts and have some sort of layer that actually translates from one to the other, I think is going to become very key. Whether or not that transformation happens at the E, the L, the T phase of the data transformation is actually really just an implementation detail. But I think whenever some sort of production analytical workflow needs to depend on some type of data table, that it's very important to have that layout properly specified and schematized so that it doesn't change on you as, as that data or those data outputs get used.

Yeah. And I think for me, it's about the patterns you're going to use. So yeah, you need to think about your architecture very lightly at the beginning, the patterns you're going to adopt. Ideally, just sketch out a diagram, like you said, something really simple. So typically, if you get a technologist to deploy you a bunch of services, they're typically going to draw a piece of paper, boxes for each of the services. They're going to go, okay, so you want some kind of data repository service. Okay. That can be a cloud relational database like MySQL, or it could be a NoSQL database, or it could be a graph store. We kind of need to know what that box is going to be. There's a box here, of course, store some data. Oh, and you want to talk to it? Okay, there's some kind of API layer, GraphQL, REST, there's a lot of technology choices, but there's a thing you're going to talk to that's going to talk to the data, right? So there's a box here. And so they'll draw out those boxes so they can have a conversation about how many of those boxes you need, what they look like. They're doing that work. They're modeling that environment. And so everybody does it, but sometimes we kind of do it lightly and badly, and sometimes we do it lightly and well. Just picking up on the ELT one, it's really interesting. Again, I'm becoming micro-focused on terminology now, because what I find is we use words and they mean different things to different people. And I'm kind of looking at, so there's been an announcement, and this will date the podcast in a year or two's time. So Snowflake and Salesforce have just announced a partnership where it looks like Salesforce is going to deploy Snowflake instances next to your Salesforce tenancy. So what that means in theory is a lot more money for Salesforce and Snowflake, whoever's got the paywall. It's almost like data virtualization. It means that we could actually pass our queries down to that Snowflake instance, zero-copy clone it anyway, and grab that Salesforce Salesforce data without having to use an ETL. We don't have to extract and load using Fivetran or Stitch or Airbyte or Multano, any of those ones. So we're effectively getting the ability to query that operational system via code, not extract and load it in, which funny enough, in the data warehousing stage, we had things like reporting replicas, which were near real-time instances of the database that wasn't, was sitting next to production, but stopped us taking production out with our big queries. Now, what's going to be interesting about that is if that pattern sticks, then that means we actually end up with TEL because we're going to transform it and then we're going to extract it and load it. Yeah. Or transform it and copy it. So what's that going to be? TCT, transform, copy, transform. So same pattern though, we're going to transform the data to make it useful, make it fit our business model or how we think, and then we're going to copy it or move it or make it available to somebody else to use. So kind of interesting world around that. I don't know, is that the postmodern data stack?

I don't know. I think there are people already trying to take that term, right? At the post—

I think I'm one of them because I'm not a fan of the modern data stack.

There you go.

There you go. And so one of the really interesting things around that analytics space, that data science space, was this idea of feature engineering. So for me, there are a bunch of patterns embedded in that term. So one pattern is this idea that we create a series of columns that have flags. Yeah. So we go and say, okay, we want to run a model, some form of statistical machine learning model. And we know that those models like big wide tables, they don't like doing joins. We know that they like indicators of yes or no. So high-value customer, yes, no. Between $10 and $20, yes, no.

And more than $1,000, yes or no, right?

That's right. So a bunch of flags that tell us a feature of a person, which is an interesting idea, and then the models run against that. But within the feature engineering space, there's more to it than that, isn't there? Because there seems to be a whole category around those type of capabilities now. And back in the big data Wave, I could see the engineers and the data scientists creating those feature sets, creating those tables of flags and refreshing them before they got to the model. What are you seeing in that space? Back when you were doing it in that Wave, was there a lot of work creating those feature set tables? And what are you seeing now for it?

So I think, well, I find feature engineering to be a positive thing, I would say, in kind of establishing an intermediate step step between kind of the more raw versions of the data that might have come from your ETL tool into dumped into your data warehouse. After that, you normalize it and then you might create a set of feature, you know, data points, flags, what have you, that kind of identify here to our organization, this is what's valuable to us. And so it kind of factors into the whole data modeling space a little bit more. It reduces the dimensionality significantly because instead of looking at a more scaled number, you look at like a 0 or 1, which almost like simplifies what your next set of things are going to do in the machine learning space, whatever kind of machine learning model you might train on top of those feature flags. And I think it actually allows you to separate the development of these feature sets and feature flags from the actual training and modeling stage. And so what I've actually seen in some companies now is that these are actually two completely separate teams. One that kind of just delivers on these feature sets, says, okay, they'll literally store that in a data warehouse project that they call their feature store. And then you scale up this machine learning team that just kind of reads from that store and can be relatively autonomous. Sure, they can make requests to the feature engineering team, but they can also kind of take what's already there and kind of move forward with that.

Yeah. And so again, I see, if we look at team topology, we start to see again, breaking it down as we scale, breaking down to separate teams that do handoffs to it. And we've seen that in the data engineering world. So back In the big data wave, the data engineer did everything, in my view. They did the collection of the data, they did the transformation of the data, they worked with the scientists to make the data denormalized in these feature stores. They did all the work end to end.

They were designed to be like very focused on that specific project.

Yeah, yeah, yeah. So we always have to have boundaries, otherwise we're boiling the ocean, as Juan from Catalog and Cocktails like to say. So, and people can't handle that. So our domain boundary wasn't the work we were doing, it was the data we were working or the people we were working with or the type of questions we were answering to give us that boundary that made us able sort of focus. And then what we've seen though with the analytics engineer is we've seen hyper-specialization. So the data engineer now seems to be relegated just to the extract and load. I mean, oh my God, what a boring job. Go take data from there and make it turn up over here. Where's the craft in that? And then the analytics engineer seems to get all the fun of transforming, understanding the business logic and all that kind of stuff. So we see these team topologies where people do the end-to-end and the real, they use their brains and do the hard craft. And then hyper-specialized as a way of scaling. So did, so that's what you saw back then, right? Was that there were people that created the feature sets, which were the engineers, and then there were people that used the feature sets, which was data scientists to do the models on top of it. And then there was a handoff between them. So how did that handoff work? How did they have a conversation of, I need 6 more flags and 2 of them are based on age, 3 of them are based on shoe size. I mean, how did that conversation happen? How did that handoff happen?

You're not going to be happy with the answer, Jira ticket.

Oh, you're right.

I'm not happy. That is actually the state of the art I've seen so far, which is basically communicate through some sort of project management tool.

And good, thank you for saying that because Jira is not an agile management tool. It is not enabling agile ways of working. It is a ticket management tool for a service desk where we can fire and forget. We can throw those requirements over the fence and make it somebody else's problem. And 9 times out of 10, not have a conversation. I listened to a podcast ages ago and they had the best Jira management process I've ever heard. And what it was, when you raised a Jira ticket, the person who got allocated that ticket, it automatically booked a meeting in both your diaries for half an hour. And if you did not turn up and have a conversation with that person who picked up the ticket, the ticket got closed because you weren't committing to actually helping the person understand your problem. You were just firing and forgetting. Now that was the coolest Jira management process I ever heard. I love that one.

Yeah. Yeah. I mean, it's an interesting problem because there's almost a misaligned incentive in a way where the team that wants the new feature flags to be created often doesn't have the capability to do so. They literally don't have the permissions to push data into the feature store. And so you get this scenario where there has to be some sort of communication handoff if you design your organization this way to be able to say, hey, sister team of mine, Can you please like prioritize these tasks? But then the whole prioritization becomes like a big thing around, okay, we've got this endless queue of tasks that we're never going to get through. What do we prioritize? And then that starts to factor into other kinds of methods of scoring which tickets go up and down and that sort of thing, which have not really seen it work well in practice. It's definitely an area that I think companies will need to improve on over time.

Yeah. And again, it depends on the scale and the size of the organization. So I had Sean McGarry on the podcast a while ago, and I've known him for many years. Cause we worked together previously in previous lives, and he's based out of the UK now, and he was presenting at a conference around some analytics stuff, and he attended one of the sessions, and I can't remember the example he used, it was either Uber or Airbnb, one of those big FAANG companies presenting their way of working and their analytics workflow, and Sean talks about how he asked the person who's doing the modeling up there, how do you deal with the problem of getting access to the data? Cause it's always hard getting access to new data. It's just a beast to solve. And the person's turned around and said to him, well, it's easy. I just log a request and 15 minutes the data turns up. And Sean's like, well, how's that work? And it was like, well, there's a team of 1,000 data engineers just sitting there waiting for the request. So they don't have a capacity, they have a capacity problem, but not what we see in small organizations. So for me, it's always about the context of your organization. How many people do you have? How much work do you have? What's the best way of optimizing the workflow for the team? And if you're big and large, yeah, that hyper-specialization, fire and forget might work. I'm still not a fan of it because Did you find that you started to really focus on the way that Jira ticket was written, the way that the data scientist described the features they needed so that the engineer could understand what to build and get it right? Or was it kind of loose?

So we, we started tightening that over time and made it to the point where the Jira ticket actually became more of a form that needed to be filled with here's 10 questions that need to be answered about how often does this thing need to be run? What are the downstream dependencies that are going to be leveraging this feature that kind of gives us a sense of what sort of priority or SLA that should be applied to this. Usually oftentimes the data scientist will actually provide some sample code too. And that actually was probably the biggest accelerator to moving forward because at the end of the day, they know what they want. They're, it's not like they don't have the capability to create the feature. A lot of times it's just, there's kind of doing the initial modeling of the feature, which is like, okay, here's a SQL query that I need to run. and then there's all the other stuff around how do I productionize this? Where am I going to run this thing ongoingly? What sort of schedule am I going to run it on? Does this need to run after some upstream dependencies? If I have 3 chains of dependencies and then that data that like 3 chains up, it's coming from a data vendor. I want to make sure that if my transform is running, that every step above it was healthy. What does data quality and health even mean in the first place? It really depends. It's very contextual on what's happening with that data. And so being able to kind of understand those things actually gives the data engineer a lot more to work with to be able to kind of deliver quickly on these feature sets and feature flags.

One of the interesting things about that for me is when I coach teams, they often want to jump into automation on day one, because that's what we're trained to do. We're like, just automate the snot out of that, get that boring job out of my face. But what I say to them is you can't automate something that isn't predictable, understood. If you don't actually know how it's going to work, then automating it early is going to cause you a pain in the butt. Come. So go through stages, maybe do it manually, then maybe create a bunch of forms that people fill out. And then once you got that, then automate it so they can self-serve. That sounds like that was a right process for that. You start off where it's just a conversation, what do you want? Then it's fill out a form because here's the things I know I need to know. How do you want to schedule it? And then potentially we could automate that scheduling so you can self-serve. You can actually figure out where in the slot to add it. And it gives you a safety net where it goes, okay, you do realize that's going to run before the data turns up. So you might want to rethink that. So that's for me is how we should approach this, this idea of automation. It has to have a level of maturity and sustainability before we can automate it. Otherwise, we're just asking for a hiding.

I mean, automation is dangerous. You lose touch with what's actually happening under the hood. So you better be sure it's correct. And I would say this applies even beyond just data and data engineering, even just in terms of what type of internal processes do you want to focus on, especially within the startup world. We only have so much capacity to be able to do things, right? And being able to kind of get a touch and feel of something and whether it's working is almost more valuable feedback and information than having something automated that's self-service, but you don't really know if it's a good thing to have.

That's kind of where we come into this idea of platform as a service or platform as a product. So again, as we're scaling teams out, we will often split off a pod of people who work on the platform capability that other people can use. So we're effectively becoming internal software engineers again. We're building platforms that the engineers or the analytics engineers or the data scientists can use. And what I often see when people start to do that is they don't adopt that product thinking. And what I mean by that, I mean, there's a whole lot of practices around product management roadmaps, and there's a whole lot of product thinking that's really useful for a team. But the key thing they don't understand is who their customer is. And so what the platform team start doing is building features they want, features they think might be useful, not features that actually their customers, the software or the data scientist or the data engineers are crying out out for. And if, is that, as teams are scaling, is that what you saw? As you saw pods become more specialized where they should be creating something that serves other pods, did they actually focus on what those other pods wanted and needed next? Or did they go down the Kool-Aid, build what they thought was cool and everybody would use and never actually monitor whether anybody actually used it? So did we create value or did we just spend some time building something we wanted to build?

Yeah, I would say when companies have excess capacity in terms of human resources and headcount, especially within the engineering domain, you start to see more of that kind of, let's build this because it's cool. It's very technologically advanced feat. And like, how cool would it be if this existed without really thinking through the actual implications and the actual value proposition to the end user at the end customer. The companies I've worked at have always been very strained. And so the types of work we always prioritize were almost was entirely based on pain points that we ourselves were feeling. And so I think your mileage may vary depending on the company you're at and how much they've kind of poured people into growing a certain team without kind of thinking through, okay, do we actually need all of them?

Yeah, and that company context is key. So I still, for me, I see a lot of large corporates trying to go on the data mesh bandwagon. When I look at them, my question to myself is, are they adopting the idea of decentralizing the data work down to a software engineering team? And I look at the organization and I go, they don't have software engineering teams. They've implemented Salesforce and SAP and those kind of things. So they're not democratizing the data work down to the software engineering team, which is one of the goals, because we know that the software engineering teams treat data as an exhaust. They create the applications that people need and data's generated as part of that, and then their job's done. They don't care how the data's used after the fact. Act. If I was them, I wouldn't at the moment either. I have a bunch of features or actions I need to be able to deliver in the product to the customer, and the data just supports that. It's not the key.

Good point. I almost feel like data mesh is something you graduate to once you've gone through the centralized data story. You start with your data silos, you go, you centralize it, and then you're like, wow, this is slowing me down too much. So then you migrate to a data mesh kind of architecture. And if you try to skip ahead, It doesn't work from what I've seen. You just end up with more data silos.

Yes, I agree. If you're a small company, start centralized, and then as you grow, you're going to have a scaling problem. You're going to have a centralized team that can't handle the work that's coming. So you're going to add more people. And we know that more than 8 or 9 people working in a pod becomes inefficient. So you're going to split the pods up. So now it becomes a scaling problem of how do you have 3 teams of 8 that are working independently, but still managing the conflict when they are working on the same data or they're working on the same platform, right? That whole kind of collaboration whilst trying to keep separate. And so data mesh has some ideas around that. A lot of them come out of team topology behavior in that book. There's a lot of scaling frameworks from Agile. So there's the way Spotify described the way they scaled their team, the so-called Spotify model. There's Jürgen's process around unfixed which take some of the team topologies and some of the Spotify kind of patterns and articulates them in a way that I understand a lot more. It's just the way he describes them I find really useful. So really what we're saying is we don't want a team of 27, we want 3 teams of 9. And so therefore, how are we going to make that work? A natural behavior is one of those pods becomes a platform pod and they serve the other teams. The downside of that, now what we're saying to those teams is you have to use the platform somebody else is building. You have to wait for that centralized platform and that centralized team to give you the features. You can't go and build them yourself. So we're going to slow you down. Now, as long as you're happy with that trade-off, that's okay. And so that idea of scaling from a data mesh point of view, definitely on board for that. But this idea of passing it to the software engineering team, that's where nirvana is because there's data people, we don't need to care anymore. They're giving us data that's well-formed, high quality, observable, but I don't see many people getting to that level because the tools and techniques that we have right now are based around data people and our data skills, not around software engineering people and their engineering skills. They're two different domains, they're two different behaviors. Is that what you see?

Yeah, I would say that is how I have seen a lot of organizations mature. The way you're describing it, you kind of break off this platform team and say, okay, their role or their goal is to accelerate the other two teams. By building out a platform, except the trade-off there is that by kind of pushing the agenda to them, if they are not moving quickly enough, and which inevitably happens at scale, you end up with like almost a slowdown. Now, the counterpoint to that is that you have more standardization and consistency, which makes it easier to maintain long-term. Having 5 different data pipelines written in 5 different languages, written in using 5 different tools is arguably be much more difficult to deal with when the 5 people who wrote those leave the company and then you have a new set of 5 who, and you just pass on ownership. And I think that it's an area that I think about a lot and it only really affects bigger companies, I would say, where you have people leave the company and then now you have to figure out who's going to own this thing. How are you going to distribute all the work that they have worked on? Which there is so much tribal knowledge and domain knowledge required to understand what happens in a data pipeline. Pipeline, like I can't even remember what hap— what my data pipeline I wrote 6 months ago is supposed to do. And documentation isn't necessarily going to solve that issue. I see it both ways. I see the pros and cons. I think in general, standardization is a good thing at a certain scale, just to make sure that you actually are able to kind of function and you don't get just completely burdened by tech debt. But at the same time, you still want to be able to move quickly and still get all the bells and whistles that are coming out every other month.

It's a trade-off decision. Like you said, you're trading for standardization because you get some benefits in the future out of standardizing. But the thing you're trading off is your speed to market, your ability for a team to iterate fast. And so you just got to make that call. What are you going to standardize and what are you going to leave as a part of the tribe?

And go further on this. I like to almost think there is the exploratory aspect of data science and the production aspect of data science are like two completely separate things. Once you kind of get into the production side of things, where you know something has signal, you're trying to apply more automation. I think at that point, that's where you should start thinking about the standardization. These are the things that need to live on longer than any individual data scientist within that organization. That's what I see. If you kind of lose the ability to quickly iterate on at the exploratory stage, it's actually very difficult to keep innovating, keep bringing value to the company.

I'm going to agree and disagree, and it's around the patterns that we use. So I agree that you need the ability to be able to innovate and explore do new things quickly, and you need the ability to be able to standardize and productionize and make things run in an automated and trusted way. And the pattern that everybody seems to do is they break those two things up. They have an exploratory team who go and do the cool stuff and the innovation, and then they hand it over the fence to some poor team that productionizes. And actually there are people that like, that they have that mindset that they like their production. It's like testing. There are people, weirdos out there that love testing and yeah, good on them, but I can't do it. So we tend to separate it. We say there's a pod of people that innovate and a pod of people that productionize. And the pattern tends to be handover. Over the fence. So I've gone and created some cool stuff and now I'm gonna hand it over to somebody to productionize it. I've seen that fail every time because you're just creating this friction. Either you put too many guardrails in of what's handed over that the innovation team are effectively writing the productionized code anyway, or they've written some stuff that's cool, handed it over, and then that team that gets it just gets swamped in trying to rewrite it. And the way I've had the most success with teams is that the pod that's innovating are doing the first cut. Up, they're exploring, they're like pioneers, they're going out and they're finding the fertile ground where they could build. And they're doing some testing, they're putting a tent up, they're digging some stuff, they're planting some seeds, they're killing some animals. And then they don't just leave the camp and go off again and wait for somebody else to come and like, oh, there's a camp, cool. No idea what that tent's doing, let's pull it down and build it a big building. Um, what happens is the second team actually go and observe what the first team have done. And then they find the products that they could build. That automate some of that work. So they're looking for things that if they automated it, somebody else would have value out of that. And then the third part of it is really the test. If another squad, pod team, picks up that piece of product and reuses it, then you've nailed it. If somebody comes and lives in the house that you built, they saw some value there because they've gone to the effort of actually moving it. And so for me, that's the process, right? Is people explore. We have to be able to lose our baby. You have to go, well, that team built some stuff and it had no value, so just We're not going to do anything with it. It's going to go away. It was a good test, but there was no value. So don't keep investing. And then that's for me, that pattern of doing it that way. Explore, somebody else looks and says, I think there's value there. It's a marketable product. They invest a little bit more. Then another squad actually picks it up and uses it. And that cycle is the cycle that's worked for teams I've worked with.

So what I'm hearing from you is essentially that we should treat our data engineering teams, not as handoff for here is my 'Custom data pipeline, I want you to do the productionization,' but rather, 'Here is my custom data pipeline. I want you to give me tools to productionize this instantly, and here is how I need it productionized.' And that is exactly what I think companies need to start aspiring for when they kind of break off this team, and which you're right, I have in practice, I have not seen it done well, but I think part of it is that the space is still kind of immature when it comes to what does production data really look like. Every company's kind of designed their own production data stack in their own completely separate ways with their own completely separate tools. And most of them aren't even really hitting the mark for, okay, we catch 90% of data quality issues, or we have an SLA of 2 hours after a data pipeline fails to understand what happened and remediate the issue. Sometimes there are several day outages. Another question we have to ask is whose responsibility is it when there are issues, right? Does it go back to the original data scientist who created it? Does it go back to the data engineering team, which is actually effectively where things do go today, which I think itself actually puts so much burden on the data engineering team that they can never innovate their way out of those issues. Or is there a separate data support team that's almost like, we're going to outsource talent to just like triage issues?

And you know, what we see a lot, I see a lot is people focus on the tools. They sometimes focus on the techniques and patterns, but they don't focus on the team topology or the way of working. They don't focus on that handoff and where they're at now and what happens when they're going to scale. What's their plan? What do they think they're going to do when they add another another 10 people when they're successful. And so when I go in and help, uh, as a consulting gig, when I go and help a customer, one of the first things around technology choices or data blueprints or some kind of strategic kind of thinking around where they could go. First thing I ask them is, let's benchmark your current team topology, how your teams currently work together, and let's benchmark what you think you're going to do. What are you going to decentralize? What are you going to centralize? And once we understand that, then we can find tools, techniques, patterns that may work for you. But if you don't understand that, you're really just Firing into the wind, you've been ad hoc. And so, yeah, I agree with you. You have to understand that how we make toast, that nodes and graphs of how work's going to happen. And that really is bringing a lot of lean thinking. If you look at lean from agile, that's what it is. It's observe the flow, figure out where a bottleneck is, go, that bottleneck's causing us problems. What can we change to remove the bottleneck? Experiment with it and then do that. But data teams don't do that. They don't look at their own processes and way of working. As a system. It's just all tribal behavior.

I think it'll change. I think it's going to change out of necessity as well. The hard truth is that people leave companies. And I think today in this environment, people leave companies more than, I'd say, 10, 15 years ago, where people would actually stay with a company for more than 5 years on average. And so you almost have to design everything about your business and your organization around this idea that that people might leave and tribal knowledge needs to transfer hands before they do so.

And especially if you look at technology acceleration, the way we are getting new technologies out faster and faster every day, what we're also seeing is team topology acceleration. And the example I use is if the way the funding worked for a lot of companies over the last couple of years, we saw masses of people being hired into the data space, into a company pretty much overnight. I mean, that's a nightmare. How do you onboard 100 people into your data engineering practice if it's not well formed?

I mean, the analytical engineering title is probably like 2 years old.

Then we see the funding dry up and we see massive cuts and I feel really sorry for those people, 10, 20% cuts of people in an organization overnight. So you think about the tribal knowledge of those people who have been doing work, who understand the work they were doing, they've just walked out the door, or they're pushed out the door. So how do we survive that?

It is terrifying to think that, wow, every single person who knows about these 500 tables in my data warehouse is gone. And so I don't even know if I can turn it off. My Snowflake, it just continues to rack up bills, but I don't even know what's being used and what's not. And I don't even know who, which of these 500 people I could go to to even figure that out.

And we've definitely seen, I think somebody termed it a while ago and I've picked it up because I like the concepts that a lot of features in a product have become categories. So, you know, you take data lineage and I liken data lineage to a seatbelt in a car. I'm not going to buy a car without a seatbelt. Built because it's just table stakes. You hop in a car, there has to be a seatbelt there. It keeps me safe. Yet now you get a tool that moves data around and has no lineage. How is that not table stakes? You gotta tell me what you're doing with the data. And I'm moving data. You've gotta give me a picture of what I've done. How could you not do that? And so I think we'll see these categories become features again and a consolidation as we always see in the market as we go through different waves.

When an ETL tool and a reverse ETL tool are completely separate. Systems, it's actually very hard to even understand lineage, even if each one is giving their own separate versions of lineage.

I'm not a great fan of the term reverse ETL, but then we've started to see the term data activation or operational analytics. And it kind of brings me back to that big data wave that you were part of, because one of the changes that I saw, so when we were in the data warehouse wave, we were internally focused. A lot of the time it was internal reporting. It was KPIs, balanced scorecards. There was a bit of operational reporting in there, but it wasn't really using data to to affect hardcore process changes in our organization or targeting of our customers. Like I said, the data mining team were typically separate. And in the big data wave, I saw a change of that. I saw this volume of data being used to create models that invoke change either by recommending to a human internal to the organization what action they should take next or codifying that action. A lot of the time, the share trading models, they were saying buy, sell, buy, sell, and actually invoking the action of buying And so that was a change in how we use that data, what the action we took. And then we lost it again. And for me, the reverse ETL or the data activation is trying to go back to that way where we use the data that we have to actually execute an action or recommend an action to be taken for the customer or the person, the employee internally, of what they should do next. Is that what you see? Is that back in that big data wave, it really was around operationalizing the action that should be taken from that data?

Yeah, I think so. The difference with how I see things applied today versus before is that before it was oftentimes used for single decision-making points, right? Like, I want to understand, okay, for this next ad campaign, I'm going to do some analysis, maybe crunch a bunch of data, and then take the CSV output and manually do what reverse ETL is doing. Whereas now you see a lot more automation around how this data is flowing. How it moves in and out of the data warehouse. In the fintech space, there's applications in both the front office side of things where you're doing— you're underwriting a customer who just came to your website within 2 seconds. And then you're, you're overnight during the times when people aren't really using the website as much, you're retraining all the models, you're pushing them back into your fraud and underwriting systems. And then on the very back end, you've got your reporting data stack that's basically crunching all this new data that's shown up during the day, creating loan servicing tapes, loan origination tapes, and sending those out to your banking partners, your regulatory agencies and whatnot. And being able to manage that data flow, it's actually just one very large system around your product and your service that, that needs to cover such a vast breadth of different types of kind of analysis and data crunching and processing that it's arguably more important to make sure the automation is in place. Is because humans make more mistakes than even data pipelines do and make sure there's enough quality control in intermediate stages of each of these kind of processes to ensure that if you're sending data out to a regulatory agency, you don't want them to come back 15 days later and say, hey, we noticed an issue with your data. Now you need to replay the last 15 days for us so that we can have the accurate statement. And that actually introduces such a larger burden. I equate it almost to recalls in a manufacturing setting. Where you've already sent out the product and now you have to bring it all back in and then resend out a new version of that product. And the burden of doing that is so much higher than just catching the issue in the factory.

I like that, our recall analogy. That's great. Because what happens in the data world, we don't, because it's a virtual product, we don't have to recall it. We just change it. Oh yeah. If you look under the bonnet now, the engine's different. We didn't tell you that. If you were relying on the fact that it was a 4-cylinder, it's like, you've got a V8 now. Yeah, but we didn't tell you that. It's driving a bit faster. Didn't upgrade the brakes. So that recall, we need that practice of recall. And actually, as you're talking about that, that FinOps side of things, financial services, you know, it does remind me 20, 30 years ago when I went to get a credit card, it would take days. I had to fill out paper forms. It went to a human. The credit risk team spent ages. And now we're into that immediacy where I can log onto a website. I can request it. It does a model in the background, in theory, and it credit scores me and says, low risk, here you go. I have a credit card. So that automation using data has been been valuable to organizations and valuable to consumers. And that's what we should be focused on.

One story about that, actually. Quicken Loans, for example, and most of these really quick underwriting systems that will give you a credit card, a loan from the website itself, they actually make their decision in under maybe half a second. And they'll put this little spinner and make you think they're taking time. Usually they'll take about 10 to 20 seconds to make you feel like they're doing work. Because if they just gave you a response that quickly, you'd be like, whoa, something's off here.

That's pretty cool.

I wonder what the model's doing.

I wonder if it's actually doing a lot of scoring. Scoring or it's just going, you're asking for $5K, we don't care. Oh, hell, you're asking for a $50K limit, we care. We're actually going to run a model against you. I wonder if there's a little bit of theater there around when there's actually something happening or when there's not.

It's a couple of things. The first is usually there's model scoring happening even before you've clicked submit on the form. So the way that you move your mouse, are you copy pasting into the form? Like these things actually factor in to whether or not you're a fraudster and how credible you are. And then there's kind of a second aspect of this, which is actually probably the bigger chunk of it, which is a lot of this stuff happens offline. Most of the online processing is a very highly trained model that has been kind of picked out because you match the profile of what that model is trying to score you on. And so how they determine that model is the one to be used is actually a much larger offline modeling system that's designed around this concept.

And that's a pattern that people often don't understand. It's this idea of real-time. Time, because people talk about, oh, I want real time. Okay, what part of real time do you want? Do you want real-time data feed? Do you want a real-time model where it's actually going to train the model on the fly in real time? Or do you just want to use a score that's been pre-calculated in real time to give a response? So, you know, that example, I didn't even think about that example of the way you fill out the form. That interaction behavior infers a lot about your risk, but that's a real-time thing. As I type these things in, it's scoring the model of my behavior to say yay or nay. But my credit score is probably pre-calculated. There's probably a model in the background that's taken some attributes about me and pre-scored me to say I look like this type of person, therefore I'm in the safe bucket or I'm not. The model itself isn't real-time, but the use of that data is real-time.

Exactly. And it, there's like a whole kind of area of engineering that I've been finding very interesting in more and because a lot of our customers kind of adopt this methodology of, okay, we have offline data processing, does all the big data workflows there, and then an online serving layer, which is usually just like Elasticsearch or DynamoDB or something very quick that you actually run an ETL process from your data warehouse into one of the serving layers and then put an API on top of the serving layer. I'm seeing this as a more and more common pattern to getting and delivering on these kind of real-time real-time data ML workflows.

And it comes back to that concept of which, which pattern needs to be real-time pattern, which doesn't, and what's the whole supply chain of your data and where are the choke points, where are the things you need to improve on. And that understanding is really important. And the other one that I, from an analytics point of view or a data science point of view, the other one that I find people get confused about is the difference between a core business process and an admin process. And I'll give you an example. So I use a methodology called BEAM from Laurence Core a lot when we're working with organizations who understand their data requirements. And so that works on our core business processes. Customer orders product, customer pays for order, store ships product, customer returns product. That flow of, you might call it a customer journey, but it's that core business process. And as we run those workshops, we will always get taken down to administration processes. So customer orders product, somebody reviews order, somebody approves order, you know, when it's not automated. And those are admin tasks and they're valuable, but we need to differentiate an admin task process versus a core business process. And I see that the same in analytical modeling is that some of the features are actually driven, feature flags are driven off admin processes and some of them are driven off core business processes. And we have to be really clear which ones we are talking about, which part of that data workflow, which part of that process are we focused on to achieve the task we've got? Do you see that? Do you see a confusion often between those two things?

I wouldn't say confusion. I think it's more in terms of an investment. Spent by the company. And I think it's obviously pretty contextual to the domain that you're looking at. Within the financial services sector, you need to make sure your reporting is really good. You're dealing with a much bigger headache from bad data in your reporting stack, potentially, than giving out a few fraudulent loans. Just because that's just how the financial sector works. You've got regulatory agencies that are double-checking everything you do and that sort of thing. I think in, in other applications, it's less more. Being able to come back and say, from an admin perspective, I can understand, okay, here's everything that kind of happened and I need to like review this thing. It depends almost like what is the ROI associated with a bad decision? If that's very high, you probably want to invest more time there and make that something that is very robust, has significant quality control around it. Whereas I would say for the most part, you see more, most attention coming on the consumer side or the actual user-facing side. Whereas this is a recommendation system that actually affects whether or not someone's going to click an item on my e-commerce site and buy the thing. You're probably going to put in more effort into that than just back-office manipulation around, around reporting of the recommendation system itself.

That's a really good concept. I have a bunch of patterns around this idea of an information product, and it's really just a way of creating a boundary of requirements. So it's a way of saying, okay, here's a small set of requirements that require some data, some code, some delivery mechanism, visualization, a service, and we're trying to get down to a size that we can deliver in an iterative 3 weeks, we're trying to get a product that's small enough that we can produce it and give it to our stakeholder for feedback and value. Give them some value, get some feedback. And that's the challenge, to break things down into those smaller chunks that we can do them in a smaller amount of time. And as part of that, we've got the information product canvas, which asks them a bunch of questions, a bit like your Jira form that you mentioned earlier. So we go and say, what's the outcome you want to achieve with this product? What's the action you're going to take if this information turns up? What questions do you want to answer? What core business processes? There's a bunch of things that give us context around what they're asking. And what we know is when we run those sessions with people where we go, okay, let's describe this information product. You often used to always start and say, what's the outcome? And what we got was an admin task. Oh, I need a list of customers. So what I've learned is we typically start off and say, what core business question do you want to ask? And it's like, how many customers we got? Okay, cool. Once you understand that, what action are you going to take? Oh, well, I'm going to go and do a sales campaign to go and get some more if the number's below our target.

Target.

Cool. And what's the outcome of that action? Oh, we're going to have more customers and more revenue. So we kind of work backwards. And then with the data quality side though, we have a lot of conversations in the market around observability, data quality, data trust, all this. And they're very technical conversations. It's the frequency of the data, is it cleaned? But taking your point, what we don't have a conversation is if that data's wrong, what's the impact to the organization? If it's just on an internal dashboard that somebody reviewing at the end of the day, The impact is not as large as if that number's given to a regulatory body, then we've kind of lied. And so we don't ask that. We don't say, what's the blast radius of that data being wrong for this information product? And that's really interesting. We should. It's definitely a pattern we should be picking up.

It's almost a side conversation relative to the agile concept and moving quickly, but it's one of these things where it's once you kind of understand that, you can prioritize things much better. Okay. If two alerts go off, which one am I supposed to take care of first? And being able to say, okay, this is in some sense, it's, this is noise, this specific alert that I don't need to worry about right now. Maybe I want to even go ahead and turn that off for the future because no one, literally no one cares about it. I think in today's world, we haven't really gotten that far. I've seen this at numerous companies where it's, oh, let's just throw a bunch of data quality, but we're not going to invest the time to tune it and calibrate it. It to what that use case actually is. If a data quality issue is detected, that is actually a non-trivial engineering burden or data science burden to go in and figure out what went wrong. Is there something I have to fix? Do I have to write a cleaning pipeline to remove these bad data points? Am I going to go manually remove the bad data points? And then depending on how the upstream is syncing, it might literally just keep adding those bad data points back. I think like understanding the impact and the use cases matter a a lot. Even going back to what you were saying though, it's even just understanding what is the specific use of the data that's being handed over is very important. Someone might say, I want a list of customers, but if they don't tell you they're going to be running an email campaign off that list, you might not include the data with the emails. There's no guarantee that your, the data you pull has the field they care about.

That's a really interesting point you make. We think about flowing the data all the way through to an action and outcome and a consequence. We think about that as a flow diagram. Program. We always focus on the left. We focus on the data and the technology because that's what we do. We sometimes start to focus on the action and the outcome. I've never really thought about focusing on the consequence. And there's been a whole thing in LinkedIn for a little while now around this idea of data SLAs. I can't remember who raised it first, but this idea that that last mile, that thing that we put in front of our customer, our information consumer, why don't we set the SLA at that and then reverse engineer what everything it needs to move to get to that change, that data turning up at that time. And so that's something that we've been thinking about. And then the same as with the data quality stuff, in our product, we've got data trust rules and we've got notifications, but we haven't solved the notification problem because what we know is exactly what you said. There's just a noise of notifications, the things going wrong. There's no impact understanding. So we're just going to quieten them. We're going to turn them off. And then we don't know which one we can put on mute and which one we can't. And so that idea of yours of saying, here's the impact at the end of the supply chain of something going wrong, then that shouldn't inform us about what we make quiet. You can't make that one quiet because the blast radius to the organization is so high that, yeah, no, you can't turn that one off. This one, yeah, that's all right. Nobody really cares. There's no real impact. I don't even know why we're capturing it.

You have to work back to an SLA from this because otherwise everyone asks for the highest SLA. I've seen this. This every time, right? No one wants an SLA that is not the highest priority because they don't really care. As an individual data scientist in a 1,000-person company, I don't know what else is going on, but I feel like whatever work I'm doing is the most important. And it's hard for me to at least balance that between everyone else's work. And so I haven't seen this solved in any fashion. I have some ideas around how to kind of think about SLAs, usually in the term of if I don't get this report by 5 PM, bad stuff happens. You set 5:00 PM as your target time. If 4:00 PM comes around and a bunch of your data pipelines are backed up upstream, you probably know you're going to miss it. You can quickly send that notification, but even then it's okay. Too much notifications is actually just as bad as no notifications.

Oh, and observability, right? We often know that buyers are liars. So let's monitor that they actually are grabbing that information at 5 o'clock. Because if they don't, then like you said, everybody's information is the most important. Again, observability should be two-way. It's observability of what we've done to the data and whether it's working or not, and it's observability out whether that data's been used for the action, because if they're not using it, we've just— low value, we've wasted our time.

As I said, the hedge fund world, it's actually— there's an easy paradigm to go by, which is when the market opens, the data needs to be ready. So we didn't have too many problems around understanding, are they being used? But then when models were turned off because they, say, weren't performing as much, it's unclear what can be turned off upstream. Lineage, well, first of all, is not used in a way to deprecate things. Lineage is actually used more as a debugging tool. I think companies also need to take a better understanding of the full lifecycle of their data into the death of a data pipeline, turning that off, deleting the code, or at least commenting the code out so it's not continuously running when it doesn't need to be.

And you're right, we never turn data pipelines off because there's a low cost of keeping them running in theory. And even when we can see the lineage, we don't know what impacts the user or the decisions made. Yeah, what we need is data chaos monkey. Netflix used to just turn servers off to see what happened. Happened. That's what we need. We need data Chaos Monkey that just stops DAGs running and then see who screams. Actually, in a lot of companies I've seen, we do that by default. They just don't run. No compgen.

Well, what's interesting is with Chaos Monkey, it's meant to make sure that your servers are fault tolerant. So a service engineer will actually design their service in a way to outperform Chaos Monkey and will spin up new containers when their containers get killed.

Killed.

There's no real equivalent in the data world. The data doesn't show up, that actually is going to have a materially bad impact. But the cost of finding out what the bad impact is might be greater than the knowledge of the bad impact.

Yeah, but it's an experiment. Just turn some data off and see if anybody even notices. And then you're like, yeah, no, nobody noticed.

That's cool. I like that.

As I said, our hour goes pretty quick. So just before we close out, is there anything else that you wanted to cover that from Agile and data and that way of working? Is something you've been passionate about or experienced?

Yeah, I think I'm very passionate about the intersection of the data orchestration with data quality and being able to quickly iterate on— I have a data pipeline, I know it's producing value for the company, I want to push this into my production system, and then I want to wrap it with some quality control. And that set of steps is, in my opinion, completely ignored in the industry right now. That I think it's going to get more and more attention over time. And it's very important for this concept of being able to move in an agile fashion and kind of move on to your next set of projects without having this level of anxiety of, are my old data pipelines working? Are my old ML models still performing? That sort of thing. And I would urge everyone to pay more attention to what their productionization handoff looks like. Don't put the burden on data engineers to just manually do it, but find ways to automate it and keep improving on that start.

I agree. Bring that lean thinking in where you actually understand the system, you identify the bottlenecks or the risk areas, and then you do something about iterating the way you work or your system to see if you can automate it, reduce that risk, fix that problem, and just rinse and repeat, constantly updating the system so that it's better. That's what we should be doing. Hey, look, that's been great. Thank you. We kind of went all over the place as we often do, but for me, key takeaway really is that Blast Radius is understanding the impact of something happening or not happening, how it's used. And from a data quality point of view, if it's bad data, really, do we care what bad things happen? Sometimes we do and sometimes we don't, and we should prioritize the work based on that. If people wanted to get ahold of you, what's the best way of people connecting with you in the modern world?

I'm available on LinkedIn. That's probably the best way. Our website is spectradata.com, which you can always reach out to us through the contact us form there. But otherwise, yeah, LinkedIn is probably the best It's an absolute pleasure to be here and it was a great conversation. Thank you so much for having me.