AgileData #33 - analytical team topologies - Ashwin Kamath

Foreign.

Welcome to the Agile Data Podcast where we talk about the merging of agile and data ways of working in a simply magical way.

Welcome to the Agile Data Podcast. I'm Shane Gibson.

And I'm Ashlyn.

Hey Ashwin, thank you for coming on the show. I've been looking forward to this one. We had a bit of a chat beforehand and we talked about lots and many varied things. But before we rip into that, why don't you give a bit of a background about yourself for the audience?

Yeah. My name is Ashwin Kameth. I am the CEO and founder of Spectre, which is a new data platform that I've been working on for the last year. I've been working in the data space for about 10 years now, primarily in the fintech world, first in a lending underwriting setting and more recently in a systematic and quantitative trading setting. More recently I started Spectre to really focus on improving the production critical grade data space where, you know, we start to look at more automated decision making frameworks to optimize and improve different business functions for companies.

Cool. And before that, you've worked for some interesting companies. Do you want to tell us a bit of a background about your journey from 10, 11 years ago when you first started into this kind of data world and the types of teams you worked with and the types of work they were doing and some of the interesting things that you've done over the last 10 years?

Yeah, so I've mostly been focused on the data engineering and data infrastructure side, done kind of various projects on the data science side as well to get a bit of understanding. So kind of rewinding back to 2015 time, I joined a company called Affirm out in San Francisco. It's a buy now, pay later company started by the legendary Max Levchin from PayPal. And I used to work on the data engineering side for what we call the bank engineering team, which kind of optimized our and figured out how to take our billion dollar, several billion dollar loan portfolio and distribute that between different capital markets partners that we worked with. And this was kind of my first introduction into the big data space. We used a lot of Apache Spark to implement our solutions. It was essentially just a very massive like optimization problem to take all these little small micro loans and figure out how do we distribute them between different capital markets partners like Morgan Stanley, like Jefferies, in a way that didn't break the covenants that were established in legal documents. And so we had to make sure mathematical constraints were in place. We had to make sure that data quality was exceptionally high because of course, if you sell the same loan twice to two different firms, you're going to end up in a situation where you have a regulatory agency down your throat. And so it was a very high stakes environment, very interesting to kind of see the problems that we had to solve in that kind of a setting. And then more recently, I was at Two Sigma Quantitative Hedge Fund here in New York where I'm currently based. And I used to work on the alternative data platform where we focused on ingesting data from thousands of different data vendors, preparing that for the trading, modeling teams and doing all the cleaning, the preparation normalizations that standardization and feature engineering required to really put this data to use in a very automated setting. And the kind of challenges that we faced at the scale of managing 50 to 100,000 data processes and still wanted to move quickly on the next set of projects is actually staggering. And there was a whole strategy around how to actually develop infrastructural tools that would allow us to move quickly.

I started my journey in the data world pretty big data. So I was back in the old generation of what probably called data warehousing or olap. And what we saw back then was we saw a real disconnect between what we called the data miners back then. So data scientists of today to a degree, and the ETL developers, BI teams. So there was always a disconnect between BI and etl. The ETL developers would always create the code and create the data structures and hand it over the fence to the people that would visualize it. But they tended to talk about there was a dependency there, and the data mining teams tended to do it on their own. They may treat the data warehouse as a source, but often they wouldn't. And they'd be in their own bubble, they wouldn't really talk to anybody else. And what I saw with the big data wave come through is a change of team topology. This idea of a data scientist turning up and a data engineer. And I saw some organizations try to find that magic unicorn, that one data scientist that can do data engineering facilitation and understand the business requirements and the statistical engineering to find the right answer of the model and then the data ops engineering to be able to deploy the models and build the platforms and all that kind of stuff. And there's very few people in the world that could do that. So we ended up with pods. We ended up with a pod that had one engineer, one scientist, maybe one platform focused person, and they started working together. Is that what you saw when you were in the fire there, building out these big data platforms and using them within those organizations.

Yeah, almost exclusively, I would say basically very similar to the idea of pods. We would have these data science teams that really were the most connected. They were almost like ops. People acted as strategy folks for the C level staff within the organization. And then we had data engineers which acted more as almost support for the data science teams and factored in two ways. So the support was both at kind of the front end of preparing the data and making sure it's ready so that some sort of exploration and ad hoc analysis can be done. But then also after that ad hoc analysis is done, having someone to be able to productionize data science workflows that are developed within a notebook environment and quickly put that into some sort of production environment so it can run in an ongoing fashion was actually much larger part of what the data engineering teams that I've been a part of have focused on. And then going even beyond that in the hedge fund world, we also had this concept of a data support team where the data engineering team was actually so overloaded with these, these productionization requests for bringing in new data, for productionizing data science workflows, pre processing tasks and delivering different types of production model, ML based modeling workflows, that the data support team was actually designed to almost kind of handle the triaging of alerts and data quality issues that might come up over time. This structure actually gave the data scientists more confidence that they could move on to another project and six months later they could look back and say, okay, my previous project is still healthy, it's working, it's still producing good insights. I can comfortably keep working on my next stuff, next set of projects without having to worry too much about maintenance and that sort of thing.

And that patent, that team topology of build something and then hand it off to another pod squad team, whatever you want to call it, to productionize it and maintain it. That's an anti pattern compared to DevOps or data ops these days. Because the DevOps data ops pattern is you build it, you release it, you maintain it, it breaks, you fix it, it's your baby. And you're then focused on those things that you're building that domain and you're consistent. Whereas that pattern back in those days was build it, throw it, come back and retrain it. But my view was that was because of the scarcity of those data science resources. So you couldn't have multiple teams of data scientists that built things within a certain domain and then maintain them. Is that what you saw.

Yeah, I think that's pretty similar to what I've seen as well. It's a good point you bring up, which is that the DevOps side of things have not really taken off within the data world. You're starting to see it a little bit more and actually some of the newer kind of tooling that's coming out usually has some side of DevOps associated with it. So that deployment and the infrastructural side of using those tools is actually getting easier and easier over time. But I think there's going to be some similar transformation. I like to equate it similar to what Vercel did for front end engineering. With Vercel you literally can push code to a git it and they have their own CI CD system already set up and that will automatically deploy on new versions of your front end application. We have yet to see that really happen within the data world and that's probably what I look forward to the most.

For me, there's been an interesting team topology change as we jump from the big data wave to the modern data stack wave, where actually we've gone away from pods, we've gone back to teams of one. I mean an analytics engineer is able to spin up a snowflake, they're able to spin up a DBT on their own and then they effectively seem to work on their own. They're an end to end process amongst themselves. So they're effectively becoming unicorns for that collection of data, the transforming of it and then the pushing it through. Maybe they don't do the presentation, so maybe the visualization is still done by another team and there's a handoff there back like the old ETLVI days, but we're starting to see the that team of one again. And that's where the analytics engineering role seems to have gone. And that's been empowered by technology. But it's interesting that we've seen in the past when we have teams of one chaos raids. So people build code, they deploy it, it has value, there's nobody to hand it off to to maintain it. So they keep maintaining it. Then they leave the organization, their knowledge goes, or they're doing such a great job, they just get pumped. More and more requirements, build more and they end up with 3,000 bundles of code. One they wrote 80 months ago and they got no clue what they wrote. They go look at it and go I why did I apply that business rule? Why am I excluding all our customers that have shoes that are blue? I can't remember why we did that. So did you see that in that transition in the market right now from the idea of small pods of people that work together to teams of individuals, or are you seeing something different?

I think I see it more at kind of the top of the stack at the data science level and less so at the data engineering level. So within the data science level, usually I do see that most data scientists operate more autonomously and take on individual projects. There's still collaboration in the data sets that need to be maintained that might feed multiple different data science workloads. But absolutely, you're seeing that individual data scientists and data engineers are just so much more efficient today than they were even five years ago. And that is, I think, really good for the scene to kind of prove out this concept. But also there comes with a lot of baggage when you have a code base that has 3,000 different airflow DAGs or DBT models, you don't even know which ones are being used, which ones are not being used, which ones are required for the highest level SLAs. It's very hard to keep adding to that, that stack and that foundation without taking a step back and saying, okay, we need to deliver on some human level change management system, make it easier to move quickly and know which alerts we can ignore, which tables are most important, which ones are less important.

And we see that a lot. We've seen that with data's mesh buzzword. So I've been trying to see whether my waves of technology and team topologies, I start talking about the data warehouse OLAP waves, the big data wave, the mondata step wave and the data mesh wave. And what I see is as data people, we love to focus on technology. We don't tend to focus on team topology, organizational structures, our ways of working our processes or what we do. We just, we go straight into a technology solution. And I'm seeing that again. And one of the areas that there's been quite a vocal piece on the Twitter Sphere and the LinkedIn sphere at the moment around the loss of data modeling. And yeah, it's interesting because it made me go back and say, what is data modeling? And well, actually people say nobody models, but they do. As soon as you write a piece of code, you're effectively modeling because you're structuring a piece of data that has a model, physical or typically a model. But people are talking about this lot. This loss of logical modeling, this loss of understanding the business context and doing that design work up front. Before we get into code and in the data warehousing wave, we were strongly modeling. I mean, we weren't good at it. We weren't. Well, we did good models, but we weren't fast at it. The way I describe it is there was often a person stuck in a room for six months, scratching their chin or their bed, coming out with this most beautiful model before we could start developing anything. And that wasn't a great process. But then we went into the big data wave. We seem to have lost modeling altogether. But I'm not sure that's true because I'm sure people were modeling. But what did you see? Was there any step in the process back then when people were modeling the business process or modeling the data before they ripped into writing Spark, Scala, Python code to. To do the cool. The cool feature factories and stuff like that?

Yeah, it's an interesting dilemma, I would say, that's been going around the space recently. Even kind of factors into the whole ETL vs DLT debate as well. If I take a step back, looking kind of like way into the past from where I started my engineering career, which was actually not in the data space, but more in application development, these ideas of resource diagrams where you kind of model out your database and kind of show the links and foreign keys and whatnot, it's almost like that has kind of come back in full force into the data space. And I would say, like most application engineers have kind of internalized that, and that's like very core to how they develop and deliver. Usually the technical design docs will actually like, lay out, okay, here's like our database tables, schemas and whatnot. Here's what our APIs are going to look like. I see that as becoming kind of what this whole data modeling space turns into as well, within the data space. And the idea being that there's a different type of. Is it ERP diagrams.

Erd into the relationship diagrams.

Right, Erd. There's a different way of kind of thinking about it at the service level, where you're trying to provide. You're using an API to provide a service to an end customer and end user, and then using it in a more analytic setting where it's actually an internal representation of how the company thinks about the world it belongs in. Right. The environment, the competitive environment, what have you. And being able to separate those two concepts and have some sort of layer that actually translates from one to the other, I think is going to become very key. Whether or not that transformation happens at the E, the L, the T phase of the data Transformation is actually really just an implementation detail. But I think whenever some sort of production, analytical workflow needs to depend on some type of data table, that it's very important to have that layout properly specified and schematized so that it doesn't change on you as that data or those data outputs get used.

Yeah. And I think for me it's about the patterns you're going to use. So, yeah, you need to think about your architecture very lightly at the beginning, the patterns you're going to adopt. Ideally, just sketch out a diagram, like you said, something really simple. So typically, if you get a technologist to deploy you a bunch of services, they're typically going to draw a piece of paper boxes for each of the services. They're going to go, okay, so you want some kind of data repository service. Okay, that can be a cloud relational database like MySQL or it could be a NoSQL database, or it could be graph store. So we kind of need to know what that box is going to be. There's a box there, store some data and you want to talk to it. Okay, there's some kind of API layer, the GraphQL rest. There's a lot of technology choices, but there's a thing you're going to talk to that's going to talk to the data. Right. So there's a box there. And so they'll draw out those boxes so they can have a conversation about how many of those boxes you need, what they look like. They're doing that work, they're modeling their environment. And so everybody does it, but sometimes we kind of do it lightly and badly, and sometimes we do it lightly and well. Just picking up on the ELT1, it's really interesting. Again, I'm becoming micro focused on terminology now because what I find is we use words and they mean different things to different people. And I'm kind of looking at. So there's been an announcement. This will date the podcast in a year or two's time. So Snowflake and Salesforce have just announced a partnership where it looks like Salesforce is going to deploy Snowflake instances next to your Salesforce tenancy. So what that means, in theory, is a lot more money for Salesforce and Snowflake, whoever's got the paywall. It's almost like data visualization. It means that we could actually pass our queries down to that Snowflake instance, zero copy, clone it anyway and grab that Salesforce Salesforce data without having to use an EEL tool. We don't have to Extract and load using fivetran or Stitch or Airbyte or Multano or any of those ones. So we're effectively getting the ability to query that operational system via code, not extract and load it in which, funny enough, in the data warehousing stage we had things like reporting replicas which were near real time instances of the database that wasn't was sitting next to production, but stopped us taking production out with our big queries. Now what's going to be interesting about that is if that pattern sticks, then that means we actually end up with tel, because we're going to transform it and then we're going to extract it and load it. Yeah. Or transform it and copy it. So what's that going to be? Tct? Transform, copy, transform. So same pattern though, we're going to transform the data to make it useful, make it fit our business model or how we think, and then we're going to copy it or move it, or make it available to somebody else to use. So, kind of interesting world around that. I don't know, is that the postmodern data stack?

I don't know. I think there are people already trying to take that term. Right.

I think I'm one of them because I'm not a fan of the modern data stack. There you go, there you go. And so one of the really interesting things around that analytics space, that data science space, was this idea of feature engineering. So for me, there are a bunch of patterns embedded in that term. So one pattern is this idea that we create a series of columns that have flags. Yeah. So we go and say, okay, we want to run a model, some form of statistical machine learning model. And we know that those models like big wide tables, they don't like doing joins. We know that they like indicators of yes or no. So high value customer. Yes, no, between 10 and 20.

Yes, no, and more than $1,000. Yes or no, Right?

That's right. So a bunch of flags that tell us a feature of a person, which is an interesting idea, and then the models run against that. But within the feature engineering space, there's more to it than that, isn't there? Because there seems to be a whole category around those type of capabilities now. And back in the big data wave, I could see the engineers and the data scientists creating those feature sets, creating those tables of flags and refreshing them before they got to the model. What are you seeing in that space? Back when you were doing it in that wave, was there a lot of work creating those feature set tables? And what are you seeing now for it.

So I think I find feature engineering to be a positive thing, I would say, in kind of establishing an intermediate step between kind of the more raw versions of the data that might have come from your ETL tool into dumped into your data warehouse. After that, you normalize it and then you might create a set of feature, you know, data points, flags, what have you that kind of identify here to our organization. This is what's valuable to us. And so it kind of factors into the whole data modeling space a little bit more. It reduces the dimensionality significantly because instead of looking at a more scaled number, you look at like a 0 or 1, which. Which almost like simplifies what your next set of things are going to do in the machine learning space, whatever kind of machine learning model you might train on top of those feature flags. And I think it actually allows you to separate the development of these feature sets and feature flags from the actual training and modeling stage. And so what I've actually seen in some companies now is that these are actually two completely separate teams. One that kind of just delivers on these feature sets says, okay, they'll literally store that in a data warehouse project that they call their feature store. And then you scale up. This machine learning team that just kind of reads from that store can be relatively autonomous. Sure, they can make requests to the feature engineering team, but they can also kind of take what's already there and kind of move forward with that. Yeah.

And so again I see if we look at team topology, we start to see again, breaking it down as we scale, breaking down to separate teams that do handoffs to it. And we've seen that in the data engineering world. So back in the big data wave, the data engineer did everything, in my view, they did the collection of the data, they did the transformation of the data, they worked with the scientists to make the data denormalized in these feature stores. They did all the work end to.

End, designed to be like very focused on that specific project.

Yeah, yeah, yeah. So we always have to have boundaries. Otherwise we're boiling the ocean as one from the catalog and cocktails like to say so and people can't handle there. So our domain boundary wasn't the work we were doing, it was the data we were working or the people we were working with or the type of questions we're answering to give us that boundary that made us able to focus. And then what we've seen though, with the analytics engineer is we've seen hyper specialization. So the data engineer now seems to be relegated just to the extract and load. I mean, oh my God, what a boring job. Go take data from there and make it turn up over here. Where's the craft in there? And then the analytics engineer seems to get all the fun of transforming, understanding the business logic and that kind of stuff. So we see these team topologies where people do the end to end and the rule, but use their brains and do the hard craft and then we hyper specialized as a way of scaling. So, so that's what you saw back then, right, was that there were people that created the feature sets, which were the engineers, and then there were people that used the feature sets, which was data scientists, to do the models on top of it. And then there was a handoff between them. So how did that handoff work? How did they have a conversation of I need six more flags and two of them are based on age, three of them are based on shoe size? I mean, how did that conversation happen? How did that handoff happen?

You're not going to be happy with the answer and JIRA ticket.

Oh, you're right, I'm not happy.

That is actually the state of the art I've seen so far, which is basically communicate through some sort of project management tool.

And good, thank you for saying that. Because JIRA is not an agile management tool, it is not enabling agile ways of working. It is a ticket management tool for a service desk where we can fire and forget, we can throw those requirements over the fence and make it somebody else's problem. And 9 times out of 10 not have a conversation. Listen to a podcast ages ago and they had the best JIRA management process I've ever heard. And what it was when you raised the juror ticket, the person who got allocated that ticket, it automatically booked a meeting in both your diaries for half an hour. And if you did not turn up and have a conversation with that person who picked up the ticket, the ticket got closed because you weren't committing to actually helping the person understand your problem. You were just firing, forgetting. Now that was the coolest Euro management process I ever heard. I love that lot.

Yeah, yeah. I mean it's an interesting problem because there's almost a misaligned incentive in a way where the team that wants the new feature flies to be created often doesn't have the capability to do so. They literally don't have the permissions to push data into the feature store. And so you get this scenario where there has to be some sort of communication handoff. If you design your organization this way to be able to say hey, sister team of mine, can you please like prioritize these tasks? But then the whole prioritization becomes like a big thing around, okay, we've got this endless queue of tasks that we're never going to get through. What do we prioritize? And then that starts to factor into other kind of methods of scoring, which tickets go up and down and that sort of thing, which have not really seen it work well in practice. It's definitely an area that I think companies will need to improve on over time.

Yeah, and again, it depends on the scale and the size of the organization. So I had Sean McGurr on the podcast a while ago and I've known him for many years as we worked together previously in previous lives. And he's based out of the UK now and he was presenting at a conference around some analytics stuff and he tended one of the sessions and I can't remember the example used was either Uber or Airbnb, one of those big fang companies presenting their way of working in their analytics workflow. And Sean talks about how he asked the person who's doing the modeling up there, how do you deal with the problem of getting access to the data? Because it's always hard getting access to new data. There's just a beast to solve. And the person's turned around and said to him, well, it's easy. I just log a request in 15 minutes, the data turns up. And she was like, well, how does that work? And it's like, well, there's a team of a thousand data engineers just sitting there waiting for the request. They don't have a capacity. They have a capacity problem, but not what we see in small organizations. So for me, it's always about the context of your organization. How many people do you have? How much work do you have? What's the best way of optimizing the workflow for the team? And if you're big and large, yeah, that hyper specialization fire and forget might work. I'm still not a fan of it because did you find that you started to really focus on the way that JIRA ticket was written? The way that the data scientists described the features they needed so that the engineer could understand what to build and get it right? Or was it kind of loose?

So we started tightening that over time and made it to the point where the JIRA took it actually became more of a form that needed to be filled with here's 10 questions that need to be answered about how often does this thing need to be running? What are the downstream dependencies that are going to be leveraging this feature. That kind of gives us a sense of what sort of priority or SLA that should be applied to this. Usually oftentimes the data scientists will actually provide some sample code too. And that actually was probably the biggest accelerator to moving forward because at the end of the day, they know what they want. It's not like they don't have the capability to create the feature a lot of times. It's just there's kind of doing the initial modeling of the feature which is like, okay, here's a SQL query that I need to run and then there's all the other stuff around. How do I productionize this? Where am I going to run this thing ongoingly? What sort of schedule am I going to run it on? Does this need to run after some upstream dependencies? If I have three chains of dependencies and then that data that like three chains up, it's coming from a data vendor. I want to make sure that if my transform is running that every step above it was healthy. What does data quality and health even mean in the first place? It really depends, it's very contextual on what's happening with that data. And so being able to kind of understand those things actually gives the data engineer a lot more to work with, to be able to kind of deliver quickly on these feature sets and feature flags.

One of the interesting things about that for me is when I coach teams, they often want to jump into automation on day one because that's what we're trained to do. We're like, just automate the snot out of that. Get that boring job out of my face. But what I say to them is, you can't automate something that isn't predictable. Understood. If you don't actually know how it's going to work, then automating it early is going to cause you a pain in the back. So go through stages, maybe do it manually, then maybe create a bunch of forms that people fill out. And then once you got that, then automate it so they can self serve. That sounds like that was a right process for that. You start off where it's just a conversation, what do you want? Then it's fill out a form because here's the things I know I need to know. How do you want to schedule it? And then potentially we could automate that scheduling so you can self serve. You can actually figure out where in the slot to add it. And it gives you a safety net where it goes, okay, you do realize that's going to run before the data turns up. So you might want to rethink that. So that's for me, is how we should approach this idea of automation. It has to have a level of maturity and sustainability before we can automate it. Otherwise we're just asking for a hiding.

I mean, automation is dangerous. You lose touch with what's actually happening under the hood, so you better be sure it's correct. And I would say this applies even beyond just data and data engineering, even just in terms of what type of internal processes do you want to focus on? Especially within the startup world, we only have so much capacity to be able to do things right and being able to kind of get a touch and feel of something and whether it's working is almost more valuable feedback and information than having something automated. That's self service, but you don't really know if it's a good thing to have.

That's kind of where we come into this idea of platform as a service or platform as a product. So again, as we're scaling teams out, we will often split off a pot of people who work on the platform capability that other people can use. So we're effectively becoming internal software engineers. Again, we're building platforms that the engineers or the analytics engineers or the data scientists can use. And what I often see when people start to do that is they don't adopt that product thinking. And what I mean by that, I mean there's a whole lot of practices around product management roadmaps and there's a lot of product thinking that's really useful for a team. But the key thing they don't understand is who their customer is. And so what the platform team start doing is building features they want, features they think might be useful, not features that actually their customers, the software or the data scientist or the data engineers are crying out, out for. And is that as teams are scaling, is that what you saw as you saw pods become more specialized where they should be creating something that serves other pods, did they actually focus on what those other pods wanted and needed next, or did they go down the Kool Aid, build what they thought was cool and everybody would use, and never actually monitor whether anybody actually used it. So did we create value or did we just spend some time building something we wanted to build?

Yeah, I would say when companies have excess capacity in terms of human resources and headcount, especially within the engineering domain, you start to see more of that kind of, let's build this because it's cool. It's very technologically advanced feat. And like how cool would it be if this existed without really thinking through the actual implications and the actual value proposition to the end user at the end customer? The companies I've worked at have always been very strained. And so the types of work we always prioritize were almost, almost entirely based on pain points that we ourselves were feeling. And so I think your mileage may vary depending on the company you're at and like, how much they've kind of poured people into growing like a certain team without kind of thinking through, okay, do we actually need all of them?

Yeah. And that company context is key. So I still, for me, I see a lot of large corporates trying to go on the data mesh background bandwagon. When I look at them, my question to myself is, are they adopting the idea of decentralizing the data work down to a software engineering team? And I look at the organization and I go, they don't have software engineering teams. They've implemented Salesforce and SAP and those kind of things. So they're not democratizing the data work down to the software engineering team, which is one of the goals. Because we know that the software engineering teams treat data as an exhaust. They create the applications that people need and data is generated as part of that, and then their job's done. They don't care how the data's used after the fact. If I was them, I wouldn't at the moment. Either I have a bunch of features or actions I need to be able to deliver in the product to the customer, and the data just supports that. It's not the key.

Good point. I almost feel like data mesh is something you graduate to once you've gone through the centralized data story. You start with your data silos, you go, you centralize it, and then you're like, wow, this is slowing me down too much. So then you migrate to a data mesh kind of architecture, and if you try to skip ahead, it doesn't work. From what I've seen, you just end up with more data silos.

Yes, I agree. If you're a small company, start centralized and then as you grow, you're going to have a scaling problem. You're going to have a centralized team that can't handle the work that's coming. So you're going to add more people. And we know that more than eight or nine people working in a pod becomes inefficient, so you're going to split the pods up. So now it becomes a scaling problem of how do you have three teams of eight that are working independently but still managing the conflict when they are working on the same data or they're working on the same platform. Right. That whole kind of collaboration whilst trying to keep separate. And so Data Mesh has some ideas around that. A lot of them come out of team topology behavior. In that book there's a lot of scaling frameworks from Agile. So there's the way Spotify described the way they scale their team, the so called Spotify model. There's Jurgen's process around unfixed, which take some of the team topologies and some of the Spotify kind of patterns and articulates them in a way that I understand a lot more. It's just the way he describes them I find really useful. So really what we're saying is we don't want a team of 27, we want three teams of nine. And so therefore how are we going to make that work? A natural behavior is one of those pods becomes a platform pod and they serve the other teams. The downside of that, now what we're saying to those teams is you have to use the platform, somebody else is building, building. You have to wait for that centralized platform and that centralized team to give you the features. You can't go and build them yourself. So we're going to slow you down. Now, as long as you're happy with that trade off, that's okay. And so that idea of scaling from a data mesh point of view, definitely on board for that. But this idea of passing it to the software engineering team, that's where nirvana is. Because as data people we don't need to care anymore. They're giving us data that's well formed, high quality, observable. But I don't see many people getting to that level because the tools and techniques that we have right now are based around data people and our data skills, not around software engineering people and their engineering skills. They're two different domains, they're two different behaviors. Is that what you see?

Yeah, I would say that is how I have seen a lot of organizations mature. The way you're describing it, you kind of break off this platform team and say, okay, their role or their goal is to accelerate the other two teams by building out a platform. Except the trade off there is that by kind of pushing the agenda to them, if they are not moving quickly enough and which inevitably happens at scale, you end up with like almost a slowdown. Now the counterpoint to that is that you have more standardization and consistency, which makes it easier to maintain long term. Having five different data pipelines written in five different languages written in Using five different tools is arguably be much more difficult to deal with. When the five people who wrote those leave the company and then you have a new set of five who. And you just pass on ownership. And I think that it's an area that I think about a lot and it only really affects bigger companies, I would say, where you have people leave the company and then now you have to figure out who's going to own this thing. How are you going to distribute all the work that they have worked on, which there is so much tribal knowledge and domain knowledge required to understand what happens in the data pipeline. Like I can't even remember what happened, what my data pipeline I wrote six months ago is supposed to do, and documentation isn't necessarily going to solve that issue. I see it both ways. I see the pros and cons. I think in general standardization is a good thing at a certain scale, just to make sure that you actually are able to kind of function and you don't get just completely burdened by tech debt. But at the same time you still want to be able to move quickly and still get all the bells and whistles that are coming out every other month on.

It's a trade off decision. Like you said, you're trading for standardization because you get some benefits in the future out of standardizing. But the thing you're trading off is your speed to market, your ability for a team to iterate fast. And so you just got to make that call. What are you going to standardize and what are you going to leave as.

A part of the trial and go further on this? I like to almost think there is the exploratory aspect of data science and the production aspect of data science are like two completely separate things. Once you kind of get into the production side of things where you know something has signal, you're trying to apply more automation. I think at that point that's where you should start thinking about the standardization. These are the things that need to live on longer than any individual data scientist within that organization. That's what I see. If you kind of lose the ability to quickly iterate on at the exploratory stage, it's actually very difficult to keep innovating, keep bringing value to the company.

I'm going to agree and disagree. And it's around the patterns that we use. So I agree that you need the ability to be able to innovate and explore and do new things quickly and you need the ability to be able to standardize and productionize and make things run in an Automated and trusted way. And the pattern that everybody seems to do is they break those two things up. They have an exploratory team who go and do the cool stuff and the innovation and then they hand it over the fence to some poor team that productionizes. And actually there are people that like, they have that mindset that they like their production. It's like testing. There are people, weirdos out there that love testing and yeah, good on them, but I can't do it. So we tend to separate it. We say there's a pot of people that innovate and a pot of people that productionize. And the patent tends to be hand over the fence. So I've gone and created some cool stuff and now I'm going to hand it over to somebody to productionize it. I've seen that fail every time because you're just creating this friction. Either you put too many guardrails in or what's handed over that the innovation team are effectively writing the productionized code anyway, or they've written some stuff that's cool, handed it over, and then that team that gets it just gets swamped and trying to rewrite it. And the way I've had the most success with teams is that the pod that's innovating are doing the first cut up. They're exploring. They're like pioneers. They're going out and they're finding the fertile ground where they could build and they're doing some testing. They're putting a tent up, they're digging some stuff, they're planting some seeds, they're killing some animals. And then they don't just leave the camp and go off again and wait for somebody else to come and like, oh, there's a camp. Cool. No idea. What that tent's doing is pull it down and build a big building. What happens is the second team actually go and observe what the first team had done and then they find the products that they could build that automate some of that work. So they're looking for things that if they automated it, somebody else would have value out of it. And then the third part of it is really the test. If another squad pod team picks up that piece of product and reuses it, then you've nailed it. Somebody comes and lives in the house that you built. They saw some value there because they've gone to the effort of actually moving it. And so for me, that's the process, right, is people explore. We have to be able to lose our baby. You have to go, well, that team built some stuff and it had no value. So just, just we're not going to do anything with it. It's going to go away. It was a good test, but there was no value. So don't keep investing. And then that's for me, that pattern of doing it that way, explore. Somebody else looks and says, I think there's value there. It's a marketable product, they invest a little bit more, then another squad actually picks it up and uses it. And that cycle is the cycle that's worked for teams I've worked with.

So what I'm hearing from you is essentially that we should treat our data engineering teams not as handoff for Here is my a custom data pipeline. I want you to do the productionization, but rather here is my custom data pipeline. I want you to give me tools to productionize this instantly and here is how I need it productionized. And that is exactly what I think companies need to start aspiring for when they kind of break off this team. And which you're right. In practice I have not seen it done well. But I think part of it is that the space is still kind of immature when it comes to what does production data really look like. Like every company's kind of designed their own production data stack in their own completely separate ways with their own completely separate tools. And most of them aren't even really hitting the mark for, okay, we catch 90% of data quality issues or we have an SLA of two hours after a data pipeline fails to understand what happened and remedy the issue, sometimes there are several day outages. Another question we have to ask is, whose responsibility is it when there are issues? Right. Does it go back to the original data scientists who created it? Does it go back to this, this data engineering team which is actually effectively where things do go today, which I think itself actually puts so much burden on the data engineering team that they can never innovate their way out of those issues. Or is there a separate data support team that's almost like we're going to outsource talent to just like triage issues.

And you know what we see a lot, I see a lot is people focus on the tools, they sometimes focus on the techniques and patterns, but they don't focus on the team topology or the way of working. They don't focus on that handoff and where they're at now and what happens when they're going to scale, what's their plan, what do they think they're going to do when they add an another 10 people when they're successful and so when I go in and help as a consulting gig, when I go and help a customer, one of the first things around technology choices or data blueprints or some kind of strategic kind of thinking around where they could go, first thing I ask them is let's benchmark your current team topology, how your teams currently work together. And let's benchmark what you think you're going to do, what are you going to decentralize, what are you going to centralize. And once we understand that, then we can find tools, techniques, patterns that may work for you. But if you don't understand that, you're really just, just firing into the wind. You've been ad hoc. And so yeah, I agree with you. You have to understand that how we make toast that nodes and graphs of how work's gonna happen. And that really is bringing a lot of lean thinking. If you look at lean from agile, that's what it is. It's observe the flow, figure out where a bottleneck is, go that bottleneck's causing us problems. What can we change to remove the bottleneck, experiment with it and then do that. But data teams don't do that. They don't look at their own processes and way of working as a system, as just all tribal behavior.

I think I'll change, I think it's going to change out of necessity as well. The hard truth is that people leave companies and I think today in this environment, people leave companies more than say 10, 15 years ago, where people would actually stay with a company for more than five years on average. And so you almost have to design everything about your business and your organization around this idea that, that people might leave and tribal knowledge needs to transfer hands before they do so.

And especially if you look at technology acceleration, the way we're getting new technologies out faster and faster every day, what we're also seeing is team topology acceleration. And the example I use is if the way the funding worked for a lot of companies over the last couple of years, we saw masses of people being hired into the data space, into a company pretty much overnight. I mean that's a nightmare. How do you onboard 100 people into your data engineering practice if it's not well fit?

I mean the analytical engineering title is probably like 2 years old.

Then we see the funding dry up and we see massive cuts. And I feel really sorry for those people, 10, 20% cuts of people in an organization overnight. So you think about the tribal knowledge. Those people who have been doing work, who understand the work they were doing, they've just Walked out the door. Well, they pushed out the door. So how do we survive that?

It is terrifying to think that, wow, every single person who knows about these 500 tables in my data warehouse is gone. And so I don't even know if I can turn it off. My snowflake, it just continues to rack up bills. But I don't even know what's being used and what's not. And I don't even know which of these 500 people I could go to to even figure that out.

And we've definitely seen, I think somebody termed it a while ago and I've picked it up because I like the concepts that a lot of features in a product have become categories. So, you know, you take data lineage and I liken data lineage to a seat building a car. I'm not going to buy a car without a seat built because it's just table stakes. You open a car, there has to be a seat belt there. It keeps me safe. Yet now you get a tool that moves data around and has no lineage. How is that not table stakes? You gotta tell me what you're doing the data and I'm moving data. You've got to give me a picture of what I've done. How could you not do that? And so I think we'll see these categories become features again and a consolidation as we always see in the market, as we go through different waves, when.

An ETL tool and a reverse ETL tool are completely separate, different systems, it's actually very hard to even understand lineage, even if each one is giving their own separate versions of lineage.

I'm not a great fan of the term reverse etl, but then we've started to see the term data activation or operational analytics. And that kind of brings me back to that big data wave that you were part of. Because one of the changes that I saw. So when we were in the data warehouse wave, we were internally focused. A lot of the time it was internal reporting, it was KPI's balance scorecards. There was a bit of operational reporting in there, but it wasn't really using data to affect hardcore process changes in our organization or targeting of our customers. Like I said, the data mining team were typically separate. And in the big data wave, I saw a change of that. I saw this volume of data being used to create models that invoke change, either by recommending to a human internal, the organization what action they should take next or codifying that action. A lot of the time, the share trading models, they were saying, buy, sell, buy, sell. And Actually invoking the action of buying and selling seller. And so that was a change in how we use that data, what the action we took and then we lost it again. And for me, the reverse ETL or the data activation is trying to go back to that wave where we use the data that we have to actually execute an action or recommend an action to be taken for the customer or the person, the employee, internally of what they should do next. Is that what you see? Is that back in that big data wave, it really was around operationalizing the action that should be taken from that data?

Yeah, I think so. The difference with how I see things apply today versus before is that before it was oftentimes used for single decision making points. Right. I want to understand, okay, for this next ad campaign, I'm going to do some analysis, maybe crunch a bunch of data and then take the CSV output and manually do what reverse ETL is doing. Whereas now you see a lot more automation around how this data is flowing, how it moves in and out of the data warehouse. In the fintech space, there's applications in both the front office side of things where you're doing, you're underwriting a customer who just came to your website within two seconds and then you're, you're overnight during the times when people aren't really using the website as much, you're retraining all the models, you're pushing them back into your fraud and underwriting systems. And then on the very back end, you've got your reporting data stack that's basically crunching all the, this new data that's shown up during the day, creating loan servicing tapes, loan origination tapes, and sending those out to your banking partners, your regulatory agencies and whatnot, and being able to manage that data flow. It's actually just one very large system around your product and your service that needs to cover such a vast breadth of different types of kind of analysis and data crunching and processing that it's arguably more important to make sure the automation is in place because humans make more mistakes than even data pipelines do. And make sure there's enough quality control in intermediate stages of each of these kind of processes to ensure that if you're sending data out to a regulatory agency, you don't want them to come back 15 days later and say, hey, we noticed an issue with your data. Now you need to replay the last 15 days for us so that we can have the accurate statement. And that actually introduces such a larger burden. I equate it almost to recalls in a Manufacturing setting where we've already sent out out the product and now you have to bring it all back in and resend out a new version of that product. And the burden of doing that is so much higher than just catching the issue in the factory.

I like that. Our recall analogy. That's great. Because what happens in the data world, we don't. Because it's a virtual product, we don't have to recall it, we just change it. Oh yeah. If you look under the bonnet now, the engine's different. We didn't tell you that. If you were relying on the fact it was a four cylinder. It's like you've got a V8 now. Yeah, but we didn't tell you that it was driving a bit faster and a bit dangerous enough. Upgrade the brakes. So that recall, we need that practice of recall. And actually as you're talking about that, that finops side of things, financial services, you know, it does remind me 20, 30 years ago when I went to get a credit card, it would take days. I had to fill out paper forms. It went to a human. The credit risk team spent ages and now we're into that immediacy where I can log onto a website, I can request it. It does a model in the background in theory and it credit scores me and says low risk. Here you go. They have a credit card. So that automation using data is been valuable to organizations and valuable to the consumers and that's what we should be focused on.

Fun story about that, actually. Quicken Loans, for example, and most of these really quick underwriting systems that will give you credit card a loan from the website itself, they actually make their decision in under maybe half a second and they'll put this little spinner and make you think they're taking time and usually they'll take about 10 to 20 seconds to make you feel like they're doing work. Because if they just gave you a response that quickly, you'd be like, whoa, something's off here.

That's pretty cool. I wonder what the model's doing. I wonder if it's actually doing a lot of scoring. Boring. Or it's just going, you're asking for 5k, we don't care. Oh hell, you're asking for a 50k limit. We care. We're actually going to run a model against you. I wonder if there's a little bit of theater there around when there's actually something happening or when there's not.

A couple of things. The first is usually there's model scoring happening even before you've clicked submit on the form. So the way that you move your mouse, are you copy pasting into the form. Like these things actually factor in to whether or not you're a fraudster and how credible that you are. And then there's kind of a second aspect of this which is actually probably the bigger chunk of it, which is a lot of this stuff happens offline. Most of the online processing is a very highly trained model that has been kind of picked out because you match the profile of what that model is trying to score you on. And so how they determine that model is the one to be used is actually a much larger offline modeling system that's designed around this concept.

And that's a pattern that people often don't understand. It's this idea of real time time because people talk, oh, I want real time. Okay, what part of real time do you want? Do you want real time data feed? Do you want a real time model where it's actually going to train the model on the fly in real time? Or do you just want to use a score that's been pre calculated in real time to give a response? So you know that example, I didn't even think about that example of the way you fill out the form. That interaction behavior infers a lot about your risk. But that's a real time thing. As I type these things in it scoring the model of my behavior to say yay or nay. But my credit score is probably pre calculated. There's probably a model in the background that's taken some attributes about me and pre scored me to say I look like this type of person, therefore I'm in the safe bucket or I'm not. The model itself isn't real time, but the use of that data is real time.

Exactly. And there's like a whole kind of area of engineering that I've been finding very interesting and more recent. And because a lot of our customers kind of adopt this methodology of okay, we have offline data processing, does all the big data workflows there and then an online serving layer which is usually just like Elasticsearch or DynamoDB or something very quick, that you actually run an ETL process from your data warehouse into one of the serving layers and then put an API on top of the serving layer. I'm seeing this as a more and more common pattern to getting and delivering on these kind of real time time data ML workflows.

And it comes back to that concept of which which pattern needs to be real time pattern, which doesn't and what's the whole supply chain of your data and where are the choke points, where are the things you need to improve on? And that understanding is really important. And the other one that I, from an analytics point of view or a data science point of view, the other one that I find people get confused about is the difference between a core business process and an admin process. And I'll give you an example. So I use a methodology called Beam from Lawrence Core a lot when we're working with organizations will understand their data requirements, requirements. And so that works on a core business process. Customer orders product, customer pays for order, store ships product, customer returns product. That flow of, you might call it a customer journey, but it's that core business process. And as we run those workshops we will always get taken down to administration processes. So customer orders product, somebody reviews order, somebody approves order, you know, when it's not automated. And those are admin tasks and they're valuable. But we need to differentiate an admin task process versus a core business process. And I see that the same in analytical modeling is that some of the features are actually driven feature flags are driven off admin processes and some of them are driven off core business processes. And we have to be really clear which ones we're talking about, which part of that data workflow, which part of that process are we focused on to achieve the task we've got? Do you see that, do you see a confusion often between those two things?

I wouldn't say a confusion. I think it's more in terms of an investment by the company and I think it's obviously pretty contextual to the domain that you're looking at within the financial services sector. You need to make sure your reporting is really good. You're dealing with a much bigger headache from bad data in your reporting stock potentially than giving out a few fraudulent loans. Just because that's just how the financial sector works. You've got regulatory agencies that are double checking everything you do and that sort of thing. I think in other applications it's less bit more being able to come back and say from an admin perspective I can understand, okay, here's everything that kind of happened and I need to like review this thing. It depends almost like what is the ROI associated with a bad decision. If that's very high, you probably want to invest more time there and make that something that is very robust, has significant quality control around it. Whereas I would say for the most part you see more most attention coming on the consumer side or the actual user facing side side. Whereas this is a recommendation system that actually affects whether or not someone's going to click an item on my E commerce site and buy the thing. You're probably going to put in more effort into that than just back office manipulation around, around reporting of the recommendation system itself.

That's a really good concept. I have a bunch of patterns around this idea of an information product, and it's really just a way of creating a boundary of requirements. So it's a way of saying, okay, here's a small set of requirements that require some data, some code, some delivery mechanism, visualization of service. And we're trying to get down to a size that we can deliver in iteration. Three weeks. We're trying to get a product that's small enough that we can produce it and give it to our stakeholder for feedback and value, give them some value, get some feedback. And that's a challenge to break things down into those smaller chunks that we can do them in a small amount of time. And as part of that, we've got the information product Canvas, which asks them a bunch of questions, a bit like your JIRA form that you mentioned earlier. So we go and say, what's the outcome you want to achieve with this product? What's the action you're going to take if this information turns up? What questions do you want to answer? What core business processes? There's a bunch of things that give us context around what they're out there asking. And what we know is when we run those sessions with people, where we go, okay, let's describe this information product. You often used to always start and say, what's the outcome? And what we got was an admin task. Oh, I need a list of customers. So what I've learned is we typically start and say, what core business question do you want to ask? And it's like, how many customers we got? Okay, cool. Once you understand that, what action are you going to take? Oh, well, I'm going to go and do a sales campaign to go and get some more if the numbers below our top target. Cool. And what's the outcome of that action? Oh, we're going to have more customers and more revenue. So we kind of work backwards. And then with the data quality side, though, we have a lot of conversations in the market around observability, data quality, data trust, all this. And they're very technical conversations. It's the frequency of the data, is it cleaned? But taking your point, what we don't have a conversation is if that data is wrong, what's the impact to the organization. If it's just on an internal dashboard that somebody reviewing, the end of the day the impact is not as large as if that number's given to a regulatory body, then we've kind of lied. And so we don't ask that, we don't say what's the blast radius of that data being wrong for this information product. And that's really interesting. We should, it's definitely a pattern we should be picking up.

It's almost a side conversation relative to the agile concept and moving quickly. But it's one of these things where it's. Once you kind of understand that you can prioritize things, much better. Okay, if two alerts go off, which one am I supposed to take care of first? And being able to say, okay, this is, in some sense it's, this is noise, this specific alert that I don't need to worry about right now. Maybe I want to even go ahead and turn that off for the future because no one, literally no one cares about it. I think in today's world we haven't really gotten that far. I've seen this at numerous companies where it's oh, let's just throw a bunch of data quality but we're not going to invest the time to tune it and calibrate it it to what that use case actually is. If a data quality issue is detected, that is actually a non trivial engineering burden or data science burden to go in and figure out what went wrong. Is there something I have to fix? Do I have to write a cleaning pipeline to remove these bad data points? Am I going to go manually remove the bad data points and then depending on how the upstream is syncing, it might literally just keep adding those bad data points back. I think understanding the impact and the use cases matter of a lot. Lot. Even going back to what you were saying though, it's even just understanding what is the specific use of the data that's being handed over is very important. Someone might say I want a list of customers, but if they don't tell you they're going to be running an email campaign off that list, you might not include the data with the emails. There's no guarantee that the data you pull has the field they care about.

That's a really interesting point you make. We think about flowing the data all the way through to an action, an outcome and a consequence. We think about that as a flow diagram. We always focus on the left. We focus on the data and the technology because that's what we do. We sometimes start to focus on the action and the outcome. Never really thought about focusing on the consequence. And there's been a whole thing in LinkedIn for a little while now around this idea of data SLAs. I can't remember who raised it first, but this idea that, that last mile, that thing that we put in front of our customer, our information consumer, why don't we set the SLA at that and then reverse engineer what everything it needs to move to get to that change, that data turning up at that time. Time. And so that's something that we've been thinking about. And then the same as with the data quality stuff in our product, we've got data trust rules and we've got notifications, but we haven't solved the notification problem because what we know is exactly what you said. There's just a noise of notifications of things going wrong. There's no impact understanding. So we're just going to quieten them, we're going to turn them off and then we don't know which one we can put on mute and which one we can't. And so that idea of yours of saying here's the impact at the end of the supply chain of something going wrong, then that should inform us about what we make quiet. You can't make that one quiet because the blast radius to the organization is so high that yeah, no, you can't turn that one off. This one, yeah, that's all right. Nobody really cares. There's no real impact. I don't even know why we're capturing it.

You have to work back to an SLA from this because otherwise everyone asks for the highest sla. I've seen this every time, right? No one wants an SLA that is not the highest priority because they don't really care. As an individual data scientist in a thousand person company, I don't know what else is going on, but I feel like whatever work I'm doing is the most important and it's hard for me to at least balance that between everyone else's work. And so I haven't seen this solved in any fashion. I have some ideas around how to kind of think about SLA is usually in the term of if I don't get this report by 5pm, bad stuff happens. So you set 5pm as your target time. If 4pm comes around and a bunch of your data pipelines are backed up upstream, you probably know you're gonna miss it. You can quickly send that notification. But even then it's okay. Too much notifications. It's actually Just as bad as no notifications.

Oh, and observability. Right. We often know that buyers are liars. So let's monitor that they actually are grabbing that information at 5 o'. Clock. Cause if they don't, then like you said, everybody's information is the most important. Again, observability should be two way. It's the observability of what we've done to the data and whether it's working or not. And it's observ about whether that data's been used for the action because if they're not using it, we've just low value, we've wasted our time.

As I said, the hedge fund world, it's actually, there's an easy paradigm to go by which is when the market opens, the data needs to be ready. So we didn't have too many problems around understanding are they being used. But then when, when models were turned off because they say weren't performing as much, it's unclear what can be turned off upstream. Lineage. Well, first of all is not used in a way to deprecate things. Lineage is actually used more as a debugging tool. I think companies also need to take a better understanding of the full lifecycle of their data into the death of a data pipeline. Turning that off, deleting the code or at least commenting the code out so it's not continuously running when it doesn't need to be.

And you're right, we never turn data pipelines off because there's a low cost of keeping them running in theory. And even when we can see the lineage, we don't know what impact to the user or the decisions made. Yeah, what we need is data Chaos Monkey. Netflix used to just turn servers off to see what happens happened. That's what we need. We need data Chaos Monkey that just stops dags running and even see who screams. Actually in a lot of companies I've seen we do that by default. They just don't run no comp.

But what's interesting is with Chaos Monkey it's meant to make sure that your servers are fault tolerant. So a service engineer will actually design their service in a way to outperform Chaos Monkey and will spin up new containers when their containers get killed. There's no real equivalent in the data world. The data doesn't show up. That actually is going to have materially bad impact. But the cost of finding out what the bad impact is might be greater than the knowledge of the bad impact.

Yeah, but instant experiment, just turn some data off and see if anybody Even notices and then you're like, yeah, no, nobody noticed, it's cool. As I said, our goes pretty quick. So just before we close out, is there anything else that you wanted to cover that from agile and data and that way of working is something you've been passionate about or experienced?

Yeah, I think I'm very passionate about the intersection of the data orchestration with data quality and being able to quickly iterate on. I have a data pipeline, I know it's producing value for the company. I want to push this into my production system and then I want to wrap it with some quality control. And that set of steps is in my opinion, completely ignored in the industry right now. That I think it's going to get more and more attention over time. And it's very important for this concept of being able to move in an agile fashion and kind of move on to your next set of projects without having this level of anxiety of are my old data pipelines working? Are my old ML models still performing that sort of thing? And I would urge everyone to pay more attention to what their productionization handoff looks like. Don't put the burden on data engineers to just manually do it, but find ways to automate it and keep improving on that that start.

I agree. Bring that lean thinking in where you actually understand the system. You identify the bottlenecks or the risk areas and then you do something about iterating the way you work or your system to see if you can automate it, reduce that risk, fix that problem and just rinse and repeat constantly updating the system so that it's better. That's what we should be doing. Hey look, that's been great. Thank you. We kind of went all over the place, as we often do. But for me, key takeaway really is that blast radius. Understanding the impact of something, something happening or not happening, how it's used. And from a data quality point of view, if it's bad data really do we care what bad things happen? Sometimes we do and sometimes we don't and we should prioritize the work based on that. If people wanted to get hold of you, what's the best way of people connecting with you in the modern world?

I'm available on LinkedIn. That's probably the best way. Our website is spectradata.com, which you can always reach out to us through the contact us form there. But otherwise, yeah, LinkedIn is probably the best way. It's an absolute pleasure to be here and it was a great conversation. Thank you so much for having me that.