AgileData #28 - analyst vs analytics engineer - Benn Stancil

Welcome to the Agile Data Podcast, where we talk about the merging of agile and data ways of working in a simply magical way.

Welcome to the Agile Data Podcast. I'm Shane Gibson.

Howdy. And I'm Ben Save.

Thank you for coming on the show. It's great to have you. I think today we're going to talk about the world of analysts and analytics engineers.

And how we can help them do good things with data in an agile way. But before we do that, how about introducing yourself for the people out there that are listening who haven't actually heard of you.

For sure, thanks for having me. Like I said, I'm Ben. I am one of the founders of a company called Mode. We build products for analysts and analytics engineers. So it's a BI product basically built for analysts.

And so, yeah, so I've been in the space now for about 10 years, originally as an analyst myself and then as a founder of Mode. Kind of been through a number of different roles as founders are kind of apt to do, bouncing around kind of between roles in marketing and product in our own internal data work, and now spend most of my time in either the internal work or in basically like the community, trying to understand the directions that people think the industry is going, how exactly that fits into what Mode's doing. It is that we can provide the most help in the future where it is the market's evolving, that we don't want to be a part of things like that, and then having conversations like this to try to better understand that.

So if we look at the market at the moment, we've got that new specialization that's just come out, the idea of an analytics engineer. And as you said, you've got a product that's been serving analysts for a long time, from what I can tell. So, well before this hyper specialization came out. What's your view on the difference between an analyst and an analytics engineer in terms of the way the market's describing it right now?

Yeah, so I think there's like the commonly agreed upon difference, which is basically the analytics engineers are responsible for writing what amount to data transformations, that they are the ones who aren't necessarily getting data into a database, but once it's in a database, kind of applying logic to it to make something useful out of it. They get the sort of primordial soup of data and then they have to turn it into something that can in theory be made into, turned into life. Analysts then are the people who are responsible for basically making sense of that. And okay, how do we actually interpret it how do we apply it to the business? What decisions do we make with it? How do we make recommendations to other folks? I think that in theory, that line makes sense. And you have sort of the people who are the data engineers getting data into the system. You have analytics engineers who are kind of like maintaining the data in a nice clean way in the system. Then you have analysts who are trying to draw the business conclusions from that data. In practice, the line gets pretty fuzzy around things like metrics, definitions, and those kinds of stuff where it's like, is that an analytics engineer? Is it an analyst? If it is an analyst or if it is an analytics engineer, then what does an analyst do? Because are they just like sort of handing those metrics off to a business stakeholder? So I think that's where it gets kind of fuzzy.

I don't actually know what other people's opinions are on this. Most people, I think, don't sort of fully say analysts are things that should go away. But it is sort of notable that there is a lot of squeeze there. And I think that's. There's lots of potential to talk about in that. But I think that's a dangerous direction for the industry kind of is to overemphasize the value of analytics engineers at the expense of what it is that analysts bring to the table. Now, granted, I come from what you would probably call it analyst background, so maybe I'm biased here, but I think that's the sort of boundaries that we're kind of drawing. And I think there will be a little bit of a pullback from what it is analytics engineers do as we start to realize, like, hey, we actually need a lot of these skill sets that analysts have that we're kind of slowly eating up from the analytics engineer kind of bottom, if you will.

Yeah, I mean, I feel a bit sorry for the data engineers, but we can go back to that in a minute. So when I'm coaching teams, because I spend a bit of my time when I'm lucky enough to coach data analytics teams, one of the things we try to encourage is T skills, right. That people have specialization typically in some skills. So they might be strong transformation coders, or they might be strong testers or strong facilitation or getting out requirements, but they tend to have other skills. Right. So a person that's more of a business analyst set of skills will tend to be quite good at writing documentation.

So we try and get T school teams where, okay, when the person with the best specialization is busy doing a task, another team member can pick up another task that's similar and just get it done. They won't get it done as fast, but they can get it done right. So we don't block. And the idea there is that actually we now have a team that can pick up the work to be done and get it done. And what I've seen the market do is move away from that. It's moved to hyper specialization, it's moved to handoffs. Right? So you described it. A data engineer goes and grabs data from somewhere and lands it. You know, an analytics engineer. You write SQL to create a model. An analyst then comes in and tries to pick up that model and actually use it to answer the business question. So we have all this handoff and for me that's actually an anti pattern for agile ways of working. Right. That really hyper specialization, that factorization of the process and that handoff process is actually where a lot of the problems happen. And from my point of view, this whole hyper specialization has been driven by the vendor market trying to come out with a niche with a category.

And then to own that. And so by coining a term and coining a role, you now have a Persona that you target and you can focus on their specialization. So what's your view on that? Are you seeing that hyper specialization that teams now handing off rather than a group of people working together and doing the skills, the things that they have skills to do?

We don't see that that much. But I think that dynamic is there in a way that is tricky. So. Okay, so I'd make a couple points. One is I very much agree with you that vendors drive a lot of this because they're looking for their niche. And the data market now is a very, very crowded market. There are tons and tons of products in the space and there still is a little bit of a dance being done by all those vendors. And Mode is one of these vendors. So like this is not a thing that I am not guil.

A dance done by those vendors to figure out the places that they fit where they are, if not the category creator. Sort of the clear best thing in this particular use case. There are way too many companies for their own to be categories. Like it would be ridiculous for these companies to also like I have my own specialized category of like SQL driven a B testing done on top of snowflake for E commerce companies.

Though somebody's probably tried. But as a result of that I think they do look for like their places where they can kind of build fences around what they do. Part of that Comes in like defining roles, defining these different things. You know, you see a lot of like kind of the thought leadership type of content that's all angled at this sort of direction. Analytics engineering, I think is, is something that has transcended that to some degree. I mean, I think, you know, the, the folks at DBT probably did the most to promote that particular role. But, um, I'm sure that is, is obviously clearly motivated by what it is that DBT sells. However, like, the uptake in the industry I think has been such that that that was less of like them inventing something that wasn't there and then more just like capitalizing on, on a direction that actually made sense for folks. And, and you know, they, they I'm sure push it, but it's. It's a reasonable push probably to make. And, and some of that's like real. A lot of that's real in their case.

That said, I think that there are on the margins, it creates some weird tension where in the analytics engineering thing, okay, there are now people who identify this way and want to protect this space. And so you have these weird edges where it's like who's responsible for what.

And I don't think it's so much like a resistance to collaboration, but it's more of a little bit of.

Being territorial and saying, no, this is the thing that I want my job to be because I need to make sure I carve out enough of a role for what I do. And people want to have influence in the business and things like that. You know, people want to not have some tiny, narrow little niche to work on. They want to solve like, bigger problems. And I am not sure that like the analytics engineering role is quite yet defined well enough to prevent the people who are in it from annexing a little bit more than is probably right. We used to have this dynamic of like data engineers and analytics or data engineers and analysts or data scientists, what you want to call them, like sort of consumption people. Data engineers do a bunch of stuff. The place they collided is this pipelines. There's like that whole stitch fix, I think it was stitch fix the blog. Data scientists shouldn't write pipelines, or data engineers shouldn't have pipelines. So you have this sort of like friction point there. But for the most part, people stay in their other lanes. We now introduce analytics engineers in that middle point. Crowds out sort of data engineers. It crowds out some analysts. Data pipeline writer is not a particularly compelling job. Like, nobody wants the job title of your job is to be like the Manager of data pipelines. And so I think analytics engineers start to push outward to the point where they are pushing into what it is that analysts do. And my concern is basically find ourselves in a position where like that role becomes big enough that the analyst role starts to deteriorate and we start to hire for people who are good at maintaining pipelines and modeling data and sort of injecting business logic into messy data as opposed to people who have like, the primary skill is solving a problem. And so I don't know that I care that much about how this shakes out so long as we still hire a bunch of people who like their primary skill set is, I think, critically about business problems. I think about applying data to them. I am not, I don't care if I'm good at technical stuff. I am good at looking at a problem being like, okay, how do I use data to help answer this question? And while analytics engineers can do that, it's kind of the secondary skill. And so to me, the thing that goes wrong here, potentially this isn't how it's gone, but it could go wrong is if we say, okay, we're replacing data engineers and data scientists with data engineers and analytics engineers. And nobody has the primary skill of making sense of business problems with data. We pun all that over to business stakeholders who understand their domain but aren't really data experts. And now you have this gap of, well, how do we actually interpret this data to make a business decision in a good and sound way.

Yeah. So look, let's break it down to looking at from two Personas, right? So I'm going to start off with data engineer first and then I'll kind of go off into the analyst and we'll kind of, we'll kind of look at it from those two points of view. So, yeah, I'm old enough to remember ETL developer, you know, back in the days where, yeah, we had tools that had to do the transformer memory, couldn't do it in the database. So it was, you know, extract, transform and load versus extract, load and transform. But effectively an ETL developer was responsible for collecting the data, transforming and combining and conforming it and then making it available for somebody to write a report, an analyst or somebody else. And then I saw that role kind of get rebranded once the data scientists turned up. It got rebranded as a data engineer. Came out of some of the cool startup companies, but they were still solving that problem. Right. Was data had to be collected. Data had to be made fit for purpose. Data had to be given to the People that wanted to use it for analysis. Then we saw the analytics engineer come in. So, okay, I'm going to take that transformation role. And I agree, I agree. The ability to do that transformation. Writing SQL has shown us that actually we have a lot of people who are literate in SQL now who can do that role. Right. So we've democratized or made it more people can do that work. And I think that's a great piece of value that we've got in the data domain. But the poor old data engineer, now what they're left with, they're left with grab the data from Salesforce and suck it in and land it into an S3 bucket before the analytics engineer gets to do the cool stuff. And actually when I look at, you know, people in those roles, what I'm seeing them start to do is going, well, that's, you know, like you said, that's boring pipeline work. Right. I don't want to be stuck doing that. So, you know, we typically look over the fence at our software brethren and they're going, hey DevOps, that's cool. Let's, let's become data ops engineers. Right? We've got more problems to solve because technical people like to solve problems. Yeah. And so as a data ops engineer, hell, now I've got lots of technologies. I've got to get a metric store and I've got to get, you know, a, and I've got to get catalog and I've got to integrate those things because nobody does it for me. That's a cool technical problem to solve. So they're again moving from this hyper specialization of move data from left to right because it's boring and going out to use their skills to solve lots of problems. So that's what I'm seeing. But what are you seeing in the data engineering space?

I think that that makes sense. You know, it's. And I think there's two effects or two things that are different in, in the data engineering world. One is, is the, the analytics engineering piece of. Yeah, you're not writing transformations. You're not. As a data engineer, you are mostly divorced nail from the business in that it used to be you had to understand, okay, what's the point of this data that we're pulling in? Great. We're not only like getting some sort of streaming data from this thing. It's like we need to understand what it's going to be used for because we are needing to write it in a particular way so they can be used for that kind of use case. Now And I this certainly isn't applied every case and stuff like that. But like you can probably be a successful data engineer without having a clue what your business is doing. Like you can get a thing of like I need to get data from here to here. How people deal with it is on the other side. And like I don't touch that. And sure you probably are a little better better if you know what's going on and certainly like in practice you actually would. But I don't think you really need to like you're just like WR raw data from one place to another. So I think that is one way in which it's different. And so that agree with you pushes things back down to like looking for other problems. The second thing is the basically like the vendor elt, the five Trans and Stitches and stuff of the world have made it. So a lot of those pipelines and even things like what Snowflake can do and things like that. Some like AWS products have made it so those pipelines are also not things that you need a data engineer to do. Like you need a specialized problem to hire a data engineer. A startup of 20 people likely doesn't need that because all of the data need to source is going to be coming from either like warehouses or from SaaS apps that you can go buy a tool off the shelf and that'll be the one that like can write the thing into your warehouse for you. And so, and like maintaining a database is now a thing you just pay Snowflake to do. You pay bigquery or whatever. Like all of those kind of operational tasks have been sort of abstracted away by vendors too. So it does sort of leave a data engineer. Like what do you do?

The things that we typically see are one of two directions.

One is this like data ops concept which is kind of like devopsy type of stuff. It's like figuring out how to keep the systems up, maintain sort of the legibility across a bunch of different tools. Basically make the thing work in a way that isn't going to be designed for like okay, great, you have five tran and you have DBT and you've got Snowflake and you've got Mode and you've got Census and you got High Touch or whatever. And all these things like those things need to actually communicate with each other in some way. This is a problem I think in general with like new modern data tools is you have a bunch of disparate tools that can't talk to each other. And so it's like, okay, how do you actually create an experience out of that? Like what is the experience of using this platform? And data engineers, I think are kind of well suited to think about that and the ops of it. The other problem is it seems like they are becoming more specialized in that there are some companies that have true data engineering problems and that's where you have to go to do this stuff.

Your average 30 person company does not need a data engineer, or even 100 person company does not really need a data engineer. Netflix sure does. You know, the Uber definitely does. The companies that are doing things at real scale, doing things where like timelines, like latency really matters, doing things where they're trying to solve problems in unique ways that aren't just like connect Salesforce to a database and write queries on it, do really need those folks. And so I think there's probably like some upmarket draw for data engineers where it's like, okay, I need to go work at one of these companies that has true data problems rather than a company that's just trying to munge a bunch of stuff together. They can use some SaaS vendors to do.

Yeah, I agree. I think the data engineer comes into their own when there's a scaling problem. So I had Sean McGurr on the podcast a little while ago. I'd worked with Sean many years ago in New Zealand and he's now@data IQ. And he said, one of the things he said was he went to a conference, which is a data science conference, and there was a presentation by somebody from Uber, Netflix, Airbnb, one of those. And he asked him the question of, well, as a data scientist, how do you get access to your data? And the person looked at him strangely, as if it's the weirdest question ever. And he said, well, I just slapped the data engineering team and 15 minutes later they send me a link and I use it. And so a lot of us aren't used to working at organizations of that scale where actually you can have a team of engineers that just make the problem go away. I think the other one is you've identified as this idea of companies that use SaaS products. They're using Slack, they're using Salesforce, they're using HubSpot, and there are now solutions to democratize getting access to that data because it's a repeatable pattern. But the large companies, they are building their own platforms, they're not using off the shelf software as a service. And so therefore, again, we need an engineer, a data engineer to solve the technical problem of how do you make that data available? So if we take the lens and we flip it to the analyst.

So in my experience, you know, again, I'm old enough to remember when the analysts were responsible for working with the stakeholders to understand the business problem. Yeah. So what, what is our problem? They were responsible and skilled to explore the data, to see how the data might give us an answer of where the problem lies and potentially what we could do to solve the problem. They would typically create some way of visualizing or presenting a view of the problem using data, and potentially a view if that problem had been solved. And heaven forbid, the really good ones would actually be responsible for helping roll out the change in the organization to make the problem go away. Right. To close that loop and actually see the benefit. Right now what I think I'm seeing is other. You know, again, this hyper specialization means that these analysts are getting relegated to dashboard designers. What are you seeing in that, in that market?

There is some of that, though. I actually put a little bit of that. Like, I think that is in some ways the responsibility of the analyst to not do that. I think most and most data teams would say yes. Like, they don't want to be dashboard designers. Their job is to help make business decisions. You know, they're there to. To support like the strategic initiatives of the company, all that sort of stuff.

A lot more people say that, I think than are successful in pulling it off. Like, it's hard to do.

And I think it's hard to do because dashboards are now relatively easy to create and everybody wants them. And it's. There's like a sort of increase in demand because you can do it more that all these people have access to a bunch of data.

Like.

Say you're like a support team.

It used to be where it's like, okay, you use Zendesk. You look at the Zendesk. Things like that's kind of the data that you get. And the analysts are focused primarily on kind of the big company problems and things like that. Now it's like everybody has access to data. Everybody knows they have access to it. Support team wants access to, like, they want the same things everybody else wants and they should be able to have them. And that creates like a lot more demand for. We need to build dashboards on kind of stuff.

I think as a data team, it is difficult to get away from that. It's like easy to think, well, we need to build sort of the foundation and then we'll finally get to these bigger questions. And. And it's a mirage that you never reach. Because the more dashboards you build, the more people have questions, the more things they need to poke at, the more like, well, what about this? What about this? It comes up. And so I think at some point, like it is on the analysts themselves to say like, this is how far we will go and no further beyond that. Like we're going to answer these more important questions like we are, we are. Yes. You don't have a dashboard that is exactly this problem, but so what? You don't need it. What's more valuable for my time is to, to work on these questions. So there's like some discipline there of saying, like, we'll only build so many. You know, a company with a thousand metrics is, is a company with no metrics, basically. But you know, like, I don't, I don't know how much this dynamic, like the, the sort of new data engineering, ETL analytics, engineering world affects this other than it just creates so much like additional data basically that while building an individual dashboard is in theory easier, you have like this giant proliferation of things that you could do. You can, you can sort of busy yourself forever with that work, more so than you could have in the past. Maybe there's something else to it, but it feels to me sort of like this is a, this is a job of the analysts to recognize where their value is and to draw the lines in the sand and say that's what I'm going to do, rather than the business needs changing or something like that. I think it's just like the job of the analyst is to stay the same. Even though the tools have evolved fundamentally, their job is no different and people should just focus on that.

Yeah, I think when I'm working with a new team, one of the things I get them focus on is what I call how to make toast based on a really great video from TedX. And it's this idea of what is your process? Right. Who does which bit of work and when do they do it? And so we see a bunch of patterns. Normally the team will come up with. So there's the prototype production pattern, I call it. Right where they'll go, okay, we've got a problem. The analyst will go in and start looking at the data that's available. They'll start doing some exploration, they'll do some light prototyping, typically in a visual tool, because seeing is believing. They'll iterate that with the business users or the stakeholders to see if they're close and then when they're happy that they're on the right path, they'll push it back to the engineering team, right. To kind of rebuild it following all the standards and the processes. We then can take that model and sometimes break it down a little bit differently. So what I call big code or big design up front. Right. So bore the ocean. And so, you know, we kind of understand the problem and the domain, and then we get the engineers just to build a whole lot of robust stuff without actually proving that that data is going to solve our business problem. So we get lots of time and lots of delay as they do all that work. And then the analysts get it back, and they're like, well, that didn't actually do what we needed to answer this business question. And we go through iterations and loops on that. There's kind of a third model I see, which is chaos Reigns, which, you know, coming from a traditional background where I was taught and I still believe that data should be modeled. But this model came out of the big data wave. And it's this idea that actually you can write pieces of code that go from the collection of the data through to the end information product or dashboard, or reverse ETL into Salesforce, and that's bound as an object, and then you do it again, and you do it again, and they don't touch. And yes, you're not getting a lot of reuse, but what you're doing is you're making them all isolated and disposable. Right. And so therefore, your focus is on the speed you can do that process and the fact that you will never make anything reusable. But that's one of the, you know, that's your way of working. You've made a conscious decision for that. What do you see in the market at the moment in terms of the way the tools are encouraging that those ways of making toast. Is there a general trend coming out?

Yeah.

How do you see it? Lots of ad hocs, you know, lots of big modeling and coding up front or a prototype to production or something else.

So this gets a little bit back to, like, the cost of production has gone down so much that we probably have, like, fallen too far on one side of the spectrum. So the way that I would say it is like, yeah, there used to be a world where building a dashboard was this giant thing, Right. And kind of this initial thing, you go through the modeling process, you file tickets with it. A month later, somebody comes back with some, like, spotfire dashboard board that isn't exactly what you want. You file another ticket, and two months later, you finally Have a thing. And by this point, the project's already over and you don't care about it anymore. And, like, knowing you have to go through that process. Everybody is cautious about what they do. You know, like the, the people who are building the thing really want to make sure that they're building something that's good. The people are asking for it, don't ask for it unless they know they're going to need it for a while, because they won't have it for a while.

There is no turn this dashboard around for me tomorrow because that's just not how it's going to happen. You know, I can't. I can't go to slack and be like, build me this feature tomorrow. I know it takes a lot of time. And so dashboards are similar. We have really lowered the cost of building them. There are some tools that try to favor that. There's tools like Looker, which are sort of traditionally shaped and that it's still very much model up front, get dashboard out the other side. There are tools like Mode that are more built on the notion of, like, we need to do ad hoc analysis to answer questions that make building dashboards very cheap and easy. In all of those things, though, I think that the dynamic that comes out of it is you produce a lot of stuff. Data in general has a lot of exhaust. Like, there's a lot of things that happen that is kind of either one off stuff that doesn't need to be revisited, or it's something you needed for a minute, or it's something where, like, we're measuring this particular metric and it changes pretty quickly because the business is changing. The cheap production of this stuff just like flooded the world with junk.

And so I think that, like, it's less a dynamic. It is, to me, it is like, it is the third case that you are describing, but unintentionally, it's that we now can do all of this. So we just make a bunch of things that are kind of throwaway and don't necessarily think about what we do with them out the other side. In some ways, I think that's. That's. That is okay because those throwaway things are oftentimes the most important questions you would answer like, you don't need another dashboard. What you actually need is an answer, you don't need to go produce my dashboard. But I do think it's important for like, the next sort of step of the industry in some ways is thinking about how do we actually make sense of all of these things that we have produced in a way that were not meant to be permanent. Like, I have this. I wrote something a while ago about like even just the term ad hoc, where I think a lot of things you call ad hoc analysis, it's kind of whatever that. That's like become the phrase for you have dashboards and reports and you've got ad hoc analysis. The notion of ad hoc sounds very, it sounds very ephemeral. It sounds like a thing that's unimportant. It sounds like, oh, I will send you an ad hoc email. That is a thing I hammer out on my phone on the subway. That's not how I think we should think of it. We shouldn't think of these things as like throwaway. We should think of them. Sometimes they are, but sometimes they're very important. They're things we need to persist. They're things that matter. They were just meant for a particular question that we're not going to ask 100 times a year. And so in some ways also those are the most important questions. Like the questions you ask once and need to answer once are often the most important ones. Like if you have to ask it a hundred times, it's probably not that important that you get it right every time.

So I think it's like figuring that out a little bit. It's figuring out a little bit of the way that we actually. What is ad hoc mean? How do we actually take the big questions that we're answering and turn them into something that is more useful and more permanent? Not in the sense of like a manual dashboard, but in making them like a piece of knowledge that we can continue to perpetuate forward.

And that sort of. When we're starting with a data analytics team and we're trying to beginning their agile journey. I talk about ad hoc and my definition of ad hoc is making shit up. We don't do that. Even though we've been agile and we've been iterative, we still need some processes and practices. We need ways of working, we need to be professionals in what we do. So, yeah, I can see what you're saying in that sometimes ad hoc just means rough and, and dodgy. Sometimes it's more around let's do something iterative and quickly and see if it has value before we then invest more in it. Right, but we've all done the one where we get an ad hoc question and then, you know, we do a quick dirty piece of work and then the next thing is, oh, actually can you run that every week? Because you're now giving Me an answer to a question that I need. So interesting thing you said there about metrics and junk. We've all experienced the data lake data swap problem. Do you think we're about to hit the idea of model or metric junkyards with the tools we're giving analytics engineers to write lots of stuff quickly but not have a lot of rigor around the model and the reuse.

I think on the analytics engineers piece of this, it's starting to emerge. You, it's, it's harder to do it than it is on just like the, the question answering thing because it's a little bit, there's usually more process in place to produce it. So say that you want to say you get, you know, the ad hoc stuff that you're describing. Someone comes in and says like, hey, I have a presentation in 15 minutes and I need to know how many people, how many customers do we have in Latin America? And you're like, they don't really care about the number, they need a slide. And it's like they're going to round it to the nearest hundred. And so you could say 400, you could say 700 and they don't. You know, it's like it's all the same. It's going to be a slide that's up for 15 seconds. That's easy to create, easy to do something, it happens really fast, it's easy to throw away. Nobody checks it. Like now you have this kind of lingering thing that somebody may find in a year and be like, oh, there's so many customers we have in different parts of the world. It's like I was only counting Latin America. That doesn't actually say anything about other places, but the dashboard looks like it does. Therefore I'm going to take it as truth and like, oh my God, now nobody knows what's going on. That doesn't happen as much in the analytics engineering world because if you're like building a new data model or whatever, it usually goes through some sort of parody process. It's version control, all that kind of thing. However, that doesn't prevent us from adding a lot of cruft. Like version control does not prevent tech debt. You know, look at any product that's been around for a while and you will see even the most rigorously controlled and like developed systems with hundreds of people thinking about how to architect them well, will still produce a lot of tech debt. And most like looker models or DBT models or metric stores are not produced that rigorously. They're often kind of Thrown together initially in a fairly ad hoc way and then kind of layered on and layered on and layered on to become this kind of like sort of a giant stack of sedimentary rock that just looks, looks different from each layer. And you can, you can distinguish the point at which one person left the company and someone else joined because it all started to change a different color.

And so like, I do think there is a problem there. It's a problem that is still early because all of this stuff is still evolving. Like we're still figuring out what analytics engineers even do. And so we don't have like easy ways to govern that. But.

It feels like a increasing problem. Like there is probably a market. Three years ago there was a market for you to be a data consultant to come in and tell people how to like implement DBT and look. Today there's probably a market for you to be a data consultant to come in and tell people how to clean up their DBT and their LookML models. Like, I think we're probably getting to that stage of grief or whatever. And it doesn't mean those tools are bad or doesn't mean anything's done wrong. It's just like that's the next problem to figure out.

Yeah, I think maybe over here in New Zealand we were a little bit late to the rest of the world, but in terms of our startup, we're bootstrapping. So both my co founder and I still side hustle doing consulting gigs for large corporates. And right now the modern data platform has made a massive market for going into large corporates and helping them choose a tool from each of the category and how to integrate it. And so each one of those takes two to four weeks because they have a governance process on how they can select something, can go through the architectural forums and all that kind of stuff. So there's a whole market now for consultants to go in for three months just to pick DBT Snowflake and get it approved and cobbled together. So interesting. If we look at that hyper specialization though, and we're starting to see, and I've seen you talk about it in the awesome articles you write, there's a whole metrics layer, what we used to call a semantic layer, coming out as a subcategory in the market. So I'm again old enough to remember business objects universes, and I actually liked LookerML as a pattern for ways of defining semantic language that's reusable. So we're starting to see that category come out. Are we going to get hyper specialization where we're going to have a metrics engineer or a semantic engineer that works between the analytics engineer and the analyst who actually wants to use the data for good?

I don't think so. And this is.

Potentially a little bit of a way to, to solve the analytics engineer.

Analysts collision friction point thing.

Where so, so in the pre metrics layer world you've got people writing.

Doing any warehouse transformation. So you've got them writing like stuff in dbt you've got them writing with canal, okay, look at mail combines these two. But putting that aside for a sec, in that world the metrics become very much like the collision. Because I've got a table that's dimension customers. I need to figure out what like our recurring, you know, what is our monthly ARR from these customers. There is some computation that has to go into that that's not straightforward. You know, it's like do I include these customers or exclude these? Like what do I do with customers who churned? When do I actually credit the ARR? Is it when the customer signs? Is it when the customer like actually starts paying?

How do we deal with multi year deals? All that sort of stuff. So a bunch of logic that isn't necessarily implied in a table that says like you know, customer contract, start date, contract end date. Like there's a bunch of things there that you have to figure out.

And.

The metrics layer basically moves that conversation into what is to me the analytics engineering world. It makes metrics a governed thing in the same way that like data models are governed. And this is, this is traditional bi. Like this is what BI has done for decades and this is what looker did is granted they did it all like all of the modeling and the metrics modeling was done in looking out.

If it's done now in DBT in a metrics layer tool. Like okay, so you have DBT for like the kind of join logic and then you've got like a metrics layer for the, you know, how do I sum up dimension customers into ARR? But all that exists kind of before you actually get to the analysis. And I think that is a useful thing for like okay, analytics engineers are kind of defining the, the logical governance of your data. The analysts then pick it up and do stuff with it. And, and I think like there are some companies that would look at that and be like we don't need an analyst, we have marketing managers who can look at this and just like look at conversion rates and they will figure all this stuff out. And like okay, I can see why you might think that, but I think it's like not that hard of a sell in that world. Say, okay, the analyst is the person who needs to be able to say, I am making sense of these metrics. Analysis is not looking at a bunch of dashboards and then saying like, this is up, this is down, therefore we do this. It's much harder than that. And so the metrics are just yet another sort of entry point for them to ask questions. But oftentimes they will either have to combine them in weird ways that you don't actually want to put in a metric store because you don't actually have that metric, like cannot canonize in some way. Or they'll have a metric that they need to pull apart where it's like, oh, this is how we compute ARR. However, the question we have today is about ARR with companies that are above a certain amount, that are in overages, that do this or that. We need to actually compute ARR differently for those folks because it just is like that's what's called for in this analysis. Doesn't mean we're like redefining the metric or doesn't mean that we're introducing something new. It just means that the particular analysis doesn't. The ARR as it is written in the canonical way is not appropriate for the question we need to answer. And so like, in those cases, the analysts probably step back into what is sort of technically upstream of an analytics engineer. But who cares? Like, they should be able to access the raw data too. Like the analytics engineer sort of prepping the meal for them. But that doesn't mean that they sometimes don't have to go to the grocery store themselves.

Yeah. So again, it's for me, comes back to that idea of T skills. Right? Is that if the analytics engineer can define the metric in a place, everybody else can use it ARR, and we know it's a strong metric, then let them do it. But if they're busy and the analysts can define it using the same capability, then let them do it. If the analysts are going to define 15 versions of ARR, all I ever say is make sure there's a word in front of it. So high value ARR or gold customer ARR. Just make sure that you're using a different name that says it's not the core definition, it's a variation of that. But again, it can be the analytics engineer or it can be the analyst. They both should be able to do that for. So the right person can do it at the right time.

You're not going to define all of them. Like, I think that's, that's the other kind of key thing in this is analytics engineers are not going to define every table, they're not going to define every model, they're not going to define every metric. Not in the sense that they're not like there will be some that the analyst needs to find. It's that no matter how many metrics you govern, there will always be questions that call for some number that isn't one of those metrics. And so you're always going to be some, like, overlap there. But it's more the analytics engineers to me to find the canonical ones, the ones that show up on dashboards, on an office TV screen. And the analysts are looking at those, but they're also thinking, like, for this particular question, as you said, what is the adjective that goes before these metrics that I actually use to answer the question that I have that's in front of me right now?

Yeah. And for me, it's also about once we start democratizing something, we need to bring in some form of federated governance. As soon as we enable lots more people to do it, they will, they'll scratch their own itch, which is what we're aiming for. Then what we needed is some way of saying, okay, well how do we govern that without a large number of stupid data steward committees that will block everything from happening for three months? So it's that balance between empowering people to do the work and stopping ending up with a metrics junkyard that will tend to happen when we go through that cycle. So if we look at the future. Right. You know, there was years ago I used to talk about search BI and mobile bi, you know, things that were coming eventually and we still haven't got there for some of those. But one of the ones I see that's kind of came for a while and it stalled, but I think it will come back as this idea of natural language asking a question and getting a response. So we've seen Thoughtspot do it power. BI's got it. Google's got their Q and A service, the ability to say, how many customers have I got? What are my top 10 customers in natural language? And get a response. From my point of view, that's going to change the way the engineering or the analyst role works because we're not structuring the data anymore to be dims and facts to go onto a dashboard. Like, they may be the model we use, but we're not coming up with a Bunch of requirements. What we're doing is saying this data can be tagged or flagged as this concept or this descriptor. And then the system knows that when you say customer, it's looking at this piece of data. And when you say product and when you say order, it knows where the data is because you've classified it and it knows how to put it together.

To answer that question.

So are you seeing that natural language kind of capability coming out in the future? Or do you think it's just a dream, something like Siri where everybody's got it but nobody uses it? Or do you think it is going to come out in the market? And again, the way we work will change because the data structures we need will actually be slightly different. The way we work will need to be slightly different to empower that capability.

I have like pretty mixed feelings on this. So. Okay, so on. On one hand, to me, natural language stuff is good enough to be able to do it. Ish. I guess it does some impressive stuff. You can do some impressive, like natural language AIs have gotten pretty good, I'll give them that. Certainly it seems like translating a question into a query is not that hard. I mean, it's very hard. But it's like a problem that people have done a reasonably good job of solving. And certainly over the next five to ten years I expect people to do a way better job of solving it. And it seems like, okay, so that's something we can do. The place where I still have some skepticism on this though is the process of asking a business question is a strange one because it is usually someone asking for a very specific thing in very vague terms where they don't know what's vague about it. Like they will say show me the number of customers in Asia. And custom. Neither of those words, there's only two words in there that are relevant and neither of them actually make any sense. It's like customers. What does customer mean? Do you count people who are non profits? Is it all time customers? Is it past customers? Is it current customers? Like, is it customers that have churned? Is it customers that are. What is custom? Customer can mean a whole bunch of things. Is it customers who only make above a certain amount of money? Is it subscription customers? Like all that kind of stuff is like, who knows? There is some implied understanding of what that means. That means different things to different people. Like as an analyst, if you get a marketer asked you to show me the number of customers in Asia, you kind of know they're coming at this From a marketing perspective, they probably care about it in this way. And if you're confused, you could just ask them. But if you get a CSM who's asking you, they probably care about the active customers because they're trying to figure out a staffing problem, whereas a marketer is trying to figure out if it's worth putting money into Asia to support spend more money on ads. Plus, nobody knows what Asia mean. Like, what does Asia. Asia could be anything, anybody. Like, that's a huge range of things that could be. People may think it means one thing versus another. As an analyst, you have this context and I guess you could like, AIs could have that context. And in theory, like, if a person can do it, so an LP thing can do it. The other part of this I think is very, like, scary is you still have to put a human in the process. Like.

It'S the Tesla problem of like, great, this car can drive, right? Most of the time I still don't really want the car to drive by itself because if it's going to mess up, like somebody needs there to tell me that. And if I ask an AI, show me all the customers in Asia and it says, here's a list, there are 800, I have no way of really knowing if it did this the way that I think it did. Like, what did it, what did it interpret my question of customers to me and I guess I can ask it, but now it's like starting to get of like, all right, what does customer mean? It's like, well, you could have to tell me a bunch of things or show me some, like, technical definition that probably doesn't make a lot of sense. Like, it's, it's the sort of thing to me where there's a bunch of complexity to that. It makes it very hard to manage because, because the, there is no language that is appropriate for the person to actually ask the question. And there's not really a way to validate if you got the answer are right that you actually meant. And so I think like, when you're making decisions on that sort of stuff, like, you want to have more confidence in that. I don't know that that AIs will really get us there, maybe closer. Maybe there are things they can do, I don't know. But like, it, it's a, it's a thing that I don't have any really good thoughts on because it feels messy and hard. But also these like NLP AIs are very good. So I don't know, maybe somebody's Smarter than I am and figures it all out.

So for me, I think, you know, when you talk about language, right. I don't think right now we have a language that is ready and available to enable this to happen. The other thing I find interesting is we don't trust the machine. So we go, well, you can give me the number. I don't know how I'm going to trust it. But actually we trust humans, even though we know under the covers there's 16 different products and five people involved to take the data from the source system and actually get that answer. And yes, maybe we've got lineage, but actually the complexity of the work they're doing with that, with that code, we're not sure it's right. Right. We're really bad at writing tests. In the data world, we're obscenely bad. We're getting better, but compared to our software preference, we just get the data and we push it out there and we say it's right. So, yeah, I think again, it's interesting how we trust humans, but we don't trust the machines and we'll see if that changes. So while we're on thinking about the future, if I had a crystal ball and I said, look, what is the one thing I want to happen in the world of data? What I want is I want to grab a piece of data, I want to drop it into a bucket, I want the machine to model it for me and tell me what it means. I want to ask it a question and I want it to give me an answer that I trust. Do you think we're ever going to get to a world where AI and I'm using my fingers to quote the word AI will ever be able to do that? Or is data always going to be just too complex and the language we need and the way we want to ask the questions too varied to let the machine do that magic for us?

Yeah, I think we're a long way away from that because the questions are often very complicated.

And I don't think it's like impossible to get to a point where it is. You ask a machine for a metric and the machine tells you that, you know, like, sure, okay, give me a data point I think we can get to. Give me.

In. Talk about putting air quotes around something. Give me insight. I think we're a ways away from that because those questions are very complicated.

This is not a thing that, to my knowledge Anyway, that like AIs are particularly good at in any real reasonable way. Though I also am not that well versed in and what AIs do. There is a way to me that this potentially works if you think about the way that, okay, so take like a company like cc. Some are familiar with them. They're like one of these augmented analytics platforms. There's a lot of tools out there that are sort of broadly sort of shaped like this thought spot sort of markets themselves as this Power BI has some pieces of this. Google has some stuff like this. They tend to all work in a way that like looks a little bit academic. Where it is give it a data set and it give it a data set of a bunch of like the, you know, size and shape of flower petals and it'll spit out all these like sort of graphs of correlations where you can see, oh my God, look, the cars with four cylinders get better gas mileage than the cars with six. That is not at all how like business questions work. Like, you don't have a giant wide data set of a bunch of dimensions where you're looking for correlations between between like 50 different different dimensions and measures. It does in a few places. There are a few problems for which that is true. But overwhelmingly, like the business problems we have are things like, should we open an office in Japan? Like, what do you even do with that? Like, what is, what does a machine do? Probably what it does is it spits out like anything it can find that is related to Japan. It's like, okay, that's a starting point, but it's not really that helpful. It's these sort of wide, like you talk about this like T shape stuff machine like AI analysis to me is just like a very, very wide and not like an inch deep of here is a bunch of things that are all kind of correlated. Like, yeah, I don't know, go find a thread to pull on. The thing that could be interesting to me is if we get to a point where they help you ask, they basically think like an analyst, where it's like, oh, I found this thing. Here are the three questions that you might want to ask next. I'll go look up those. And like, rather than just sort of spreading out really wide, it's like here and then you kind of drill in and not like drilling and sort of like a drill down. BI sense, but drill in in the sense of like, now that I have learned this thing, what are other things that I want to learn that help me understand why that is? Or I see, like, should we have an office in Japan? Well, we have a bunch of leads in Japan. Well, now my question is they ever convert? Why do they not convert? Like those aren't drill down questions in the sense of like I want to drill into this dashboard and filter by this other thing. They're a totally new question, but a question is sort of the genesis is found in the previous one. And like if you enable that kind of flow, maybe, maybe you get to a point where it's sort of the science fictiony thing. Like wait, show me this. Oh, that's interesting. Let me go over here and show me this. Like ah, yeah, you could maybe get there but it's, it's a very different model to me than take a CSV and find insight in the CSV that is usually like it turns out that men are on average taller than women. You're like, okay, thank you for finding that correlation. That is much appreciated. What else do we do?

Yeah, I'm a great fan of the five whys. And also what I tend to find is the first question that Stakeholder asks you is not actually the question they want answered. It's just the first question they know to ask. You know, how many customers have we got? Okay, 42. Right? Where are they located?

Right.

You know, 10 in the U.S. okay, what are they buying? Right. So they have a series of questions they know they need to answer to get to the data. They need to make a decision. So yeah, I'm with you in terms of natural language capabilities that help them do that themselves without an analyst after being there, if they can trust it, that it's going to get us one step closer to democratization and self service.

There's another thing too that they have like a note on this somewhere, maybe a blog post one day.

Can AI say no? This is another thing that I think is a thing that holds this back is if I can ask the thing any question and it's like I asked the AI to jump and it tells me how high. There are times when it needs to tell me like don't do that, like don't ask that question. That's a bad question. Go this other way instead. And that's like one of the things that analysts can do is, is it's they can help guide the process. Not because they're just like, oh, you asked for this, here's your answer, here's your answer. They're like saying ah, maybe don't ask that. Like there's a better way of asking that. I understand what you're trying to get at and you need to see this instead. Or you're finding a bunch of spurious stuff don't ask that question.

And I don't know that, like, an AI has to be pretty smart to be able to not only be able to answer these questions, but also to understand the point at which, like, the question you are asking is structurally bad. It's a question that will answer. You will get an answer in a way that you don't want it to be, or it'll be a way that you don't think it should be. I don't know that that is a thing that we are that close to doing either. But maybe, I don't know. The AIs do crazy stuff steps.

Well, I think if somebody was designing that, really, instead of saying no, because nobody likes it when the machine says no, it should just say 42. And every time you got an answer of 42, then you'd know that you need to ask again. So, look, just time flies when you're having fun. And so I kind of want to close it out with one more question for you. And so, yeah, again, I'm old enough to remember when we move from mainframes to client server. And that technology step spawned a massive change in the data and analytics world. And over the last couple of years, we've seen that same acceleration and change based on cloud technologies. Given all the new tools, given all the new capabilities, given all the new ways of working, has this change over the last couple of years made an analyst's life easier, more fun, or has it made it worse for them? Has it relegated their role to more drossy work and less value work? What's your view? Analysts in a better space now, or have we done them a disservice?

Better space, but not universally so. I mean, the tools are better, they're just nicer to use, they're a lot more powerful. You don't have to sit there and watch a query spin for an hour nearly as often as you used to. There's a lot of stuff with, like databases where this wasn't even that long ago, but like 10 years ago, you'd run queries and like the cluster would go down and it was just part of life where you, you'd run the thing overnight because you needed to and you'd come back and something happened in the middle of the night and I got to wait till tomorrow to actually get your answer. It's very frustrating. It's like much faster to get stuff. So also, like, in that same time, you know, you got to do some analysis on stuff in Salesforce. Well, it's going to take you A week to get to that point. And you got to like, work with an engineer to actually do all this, like, kind of prep work to actually answer the question. You spend a lot more time just dealing with the system than you do with like, the stuff that is. That is like, quote unquote, fun. However, I don't think that, like, analysts have.

Gotten to the point where they're mostly doing the fun stuff. Part of this is potentially this, like the demand for that. Basically the explosion of the need for data creates an explosion for the need for dashboards. And now people are like chasing what is a much higher demand. And so they're not actually working on the fun stuff to the degree that they might want to. And so, you know, I think there is still stuff to be done and figured out there. The potential is certainly there for life to be better, but I don't know that, like, universally it is because we've just sort of shifted the types of problems that you work on. Rather than like building a week to build, taking a week to build one dashboard and going through the frustrations of that, it's now, well, I have to build 10 in a week. Yeah, I guess I'm like, more productive, but I don't know that I'm producing more value and I'm certainly not necessarily having more fun. So I think there's. It doesn't mean it can't be done, but it's just like it wasn't sort of like we just solved all the headaches of how people work with data and now magically, everything's just a great time.

Well, look, I think we'll close it out there. Look, thank you for coming on the show. On the show notes. I'll put in a link to your articles that you do every week. So I'd just like to thank you for writing those. They're well written, well thought out, well reasoned, based on fact and some. Some opinion, but opinion, you know, grounded in fact. So really enjoy reading them as I think the rest of the world do. So thank you for taking the time of writing such great content and making it freely available. It's. It's a thing of beauty for when people do that. So, yeah, thanks for coming on the show and we'll catch you later.

For sure. Thanks for having me and appreciate that. But yeah, this was a good time.

And that Data Magicians wants another agile data podcast. If you'd like to learn more on applying an agile way of working to your data and analytics, head over to agiledata IO.