AgileData #28 - analyst vs analytics engineer - Benn Stancil

Welcome to the Agile Data Podcast, where we talk about the merging of agile and data ways of working in a simply magical way. Welcome to the Agile Data Podcast. I'm Shane Gibson.

Howdy, and I'm Ben Sansell.

Hey Ben, thank you for coming on the show. It's great to have you. I think today we're going to talk about the world of analysts and analytics engineers. And how we can help them do good things with data in an agile way. But before we do that, how about introducing yourself for the people out there that are listening who haven't actually heard of you?

For sure, and thanks for having me. So like I said, I'm Ben. I am one of the founders of a company called Mode. We build products for analysts and analytics engineers. So it's a BI product basically built for analysts. And so, yeah, so I've been in the space now for about 10 years, originally as an analyst myself, and then as a founder of Mode, kind of been through a number of different roles as founders are kind of apt to do, bouncing around kind of between roles in marketing and product in our own internal data work. And now spend most of my time in either the internal work or in basically like the community, trying to understand the directions that people think the industry is going, how exactly that fits into what Mode's doing. Where it is that we can provide the most help in the future, where it is the market's evolving that, that, you know, we don't want to be a part of, things like that. Uh, and then having conversations like this to try to better understand that.

So if we look at the market at the moment, we've got that, that new specialization that's just come out, you know, the idea of an analytics engineer. And as you said, you know, you've got a product that's been serving analysts for a long time, from, from what I can tell, so well before these, this hyper-specialization came out. What's, what's your view on the difference between an analyst and an analytics engineer in terms of the way the market's describing it right now?

Yeah. So, I think there's, like, the commonly agreed-upon difference, which is basically, you know, analytics engineers are responsible for writing what amount to data transformations, that they are the ones who aren't necessarily getting data into a database, but once it's in a database, kind of applying logic to it to make something useful out of it. They get the sort of primordial soup of data, and then they have to turn it into something that can, in theory, be made into, turned into life. Analysts then are the people who are responsible for basically, like, making sense of that. And okay, how do we actually interpret it? How do we apply it to the business? What decisions do we make with it? How do we make recommendations to other folks? I think that, in theory, that line makes sense, and you have sort of the people who are the data engineers getting data into the system, you have analytics engineers who are kind of, like, maintaining the data in a nice clean way in the system, then you have analysts who are trying to draw the business conclusions from that data. In practice, the line gets pretty fuzzy around things like metrics definitions and those kinds of stuff where it's like, is that an analytics engineer? Is it an analyst? If it is an analytics engineer, then what does an analyst do? Because are they just, like, sort of handing those metrics off to a business stakeholder? So, I think that's where it gets kind of fuzzy. I don't actually know what other people's opinions are on this. Most people, I think, don't sort of fully say analysts are things that should go away, but it is sort of notable that there is a lot of squeeze there. And I think that's— there's a lot to potentially talk about in that, but I think that's a dangerous direction for the industry, kind of, is to overemphasize the value of analytics engineers at the expense of what it is that analysts bring to the table. Now, granted, I come from what you would probably call an analyst background, so maybe I'm biased here, but I think, like, that's the sort of boundaries that we're kind of drawing. And I think there will be a little bit of a pullback from what it is analytics engineers do as we start to realize, like, hey, we actually need a lot of these skill sets that analysts have that were kind of slowly eating up from the analytics engineer, you know, kind of bottom, if you will.

Yeah, I mean, I feel a bit sorry for the data engineers, but we can go back to that in a minute. So when I'm coaching teams, you know, because I spend a bit of my time when I'm lucky enough to coach data analytics teams, one of the things we try to encourage is T-skills, right? That people have special specialization typically in some skills. So, yeah, they might be strong transformation coders or they might be strong testers or strong facilitation or getting out requirements, but they tend to have other skills, right? So, a person that's more of a business analyst set of skills will tend to be quite good at writing documentation. So, we try and get T-Skill teams where, okay, when the person with the best specialization is busy doing a task, another team member can pick up another task that's similar and just get it done. They won't get it done as fast, but they can get it done, right? So we don't block. And the idea there is that actually we now have a team that can pick up the work to be done and get it done. And what I've seen the market do is move away from that. It's moved to hyper-specialization, it's moved to handoffs, right? So you described it, a data engineer goes and grabs data from somewhere and lands it, you know, an analytics engineer You write SQL to create a model. An analyst then comes in and tries to pick up that model and actually use it to answer the business question. So we have all this handoff. And for me, that's actually an anti-pattern for agile ways of working, right? That really hyper-specialization, that factorization of the process, and that handoff process is actually where a lot of the problems happen. And from my point of view, this whole hyper-specialization has been driven by the vendor market trying to come out with a niche, with a category, and then to own that. And so by coining a term and coining a role, you now have a persona that you target and you can focus on their specialization. So what's your view on that? Are you seeing that hyper-specialization, that teams now handing off rather than a group of people working together and doing the skills, the things that they have skills to do?

We don't see that that much, but I think that dynamic is there in a way that is tricky. So, okay, so I'd make a couple of points. One is I very much agree with you that vendors drive a lot of this because they're looking for their niche. And the data market now is a very, very crowded market. There are tons and tons of products in the space. And there still is a little bit of like a dance being done by all of those vendors. And Mode is one of these vendors. So, like, hey, sort of, this is not a thing that I am not guilty of. A dance done by those vendors to figure out, like, the places that they fit where they are, if not the category creator, sort of the clear best thing in this particular use case. There are way too many companies for there to be categories. Like, it would be ridiculous for these companies to also, like, I am my own specialized category of, like, SQL-driven A/B testing done on top of Snowflake for e-commerce companies. Though somebody's probably tried. But as a result of that, I think they do look for, like, their places where they can kind of build fences around what they do. Part of that comes in, like, defining roles, defining these different things. You see a lot of, like, kind of the thought leadership type of content that's all angled at this sort of direction. Analytics engineering, I think, is something that has transcended that to some degree. I mean, I think, you know, the folks at DBT probably did the most to promote that particular role. I'm sure that is obviously clearly motivated by what it is that DBT sells. However, the uptake in the industry, I think, has been such that that was less of them inventing something that wasn't there and then more just capitalizing on a direction that actually made sense for folks. And they, I'm sure, push it, but it's a reasonable push probably to make. And some of that's real. A lot of that's real in their case. That said, I think that there are, on the margins, it creates some weird tension where, in the analytics engineering thing, okay, there are now people who identify this way and want to protect this space. And so, you have these weird edges where it's like, who's responsible for what? And I don't think it's so much like a resistance to collaboration, but it's more of a little bit of, like, being territorial and saying, "No, this is the thing that I want my job to be because I need to make sure I carve out enough of a role for what I do." And people want to have influence in the business and things like that. You know, people want to not have some tiny, narrow little niche to work on. They want to solve, like, bigger problems. And I am not sure that, like, the analytics engineering role is quite yet defined well enough to prevent the people who are in it from annexing a little bit more than is probably right. We used to have this dynamic of data engineers and analysts, or data scientists, whatever you wanna call them, like, sort of consumption people. Data engineers do a bunch of stuff. The place they collided is these pipelines. There's that whole Stitch Fix, I think it was Stitch Fix, the blog, data scientists shouldn't write pipelines, or data engineers shouldn't know pipelines. So you have this sort of friction point there, but for the most part, people stay in their other lanes. We now introduce analytics engineers in that middle point, crowds out data engineers, it crowds out some analysts. Data pipeline writer is not a particularly compelling job. Nobody wants the job title of your job is to be like the manager of data pipelines. And so I think analytics engineers start to push outward to the point where they are pushing into what it is that analysts do. And my concern is basically find ourselves in a position where that role becomes big enough that the analyst role starts to deteriorate and we start to hire for people who are good at maintaining pipelines and modeling data and sort of injecting business logic into messy data, as opposed to people who have the primary skill is solving a problem. And so, I don't know that I care that much about how this shakes out so long as we still hire a bunch of people who, like, their primary skill set is, I think critically about business problems, I think about applying data to them, I am not— I don't care if I'm good at technical stuff, I am good at looking at a problem being like, okay, how do I use data to help answer this question? And while analytics engineers can do that, it's kind of the secondary skill. And so, to me, the thing that goes wrong here, potentially, this isn't how it's gone, but it could go wrong, is if we say, "Okay, we're replacing data engineers and data scientists with data engineers and analytics engineers," and nobody has the primary skill of making sense of business problems with data. We punt all that over to business stakeholders who understand their domain but aren't really data experts. And now you have this gap of, well, how do we actually interpret this data to make a business decision in a good and sound way?

Yeah, so look, let's break it down to looking at it from two personas, right? So I'm gonna start off with data engineer first, and then I'll kind of go off into the analyst, and we'll kind of look at it from those two points of view. So, I'm old enough to remember ETL developer, back in the days where we had tools that had to do the transform in memory, couldn't do it in the database. So it was extract, transform, and load versus extract, load, and transform. But effectively, an ETL developer was responsible for collecting the data, transforming and combining and conforming it, and then making it available for somebody to write a report, an analyst or somebody else. And then I saw that role kind of get rebranded, once the data scientists turned up, it got rebranded as a data engineer, came out of some of the cool startup companies. But they were still solving that problem, right? Data had to be collected, data had to be made fit for purpose, data had to be given to the people that wanted to use it for analysis. And then we saw the analytics engineer come in. So, okay, I'm gonna take that transformation role. And I agree, I agree, the ability to do that transformation writing SQL has shown us that actually we have a lot of people who are literate in SQL now, who can do that role, right? So we've democratized or made it, more people can do that work. And I think that's a great piece of value that we've got in the data domain. But the poor old data engineer now, what they're left with, they're left with grab the data from Salesforce and suck it in and land it into, you know, an S3 bucket. Before the analytics engineer gets to do the cool stuff. And actually, when I look at people in those roles, what I'm seeing them start to do is going, well, that's, you know, like you said, that's boring pipeline work, right? I don't wanna be stuck doing that. So, you know, we typically look over the fence at our software brethren and they're going, hey, DevOps, that's cool. Let's become DataOps engineers, right? We've got more problems to solve because technical people like to solve problems. And so as a DataOps engineer, hell, now I've got lots of technologies. I've gotta get a metric store and I've gotta get, you know, and I've got to get a catalog, and I've got to integrate those things because nobody does it for me, that's a cool technical problem to solve. So, they're, again, moving from this hyper-specialization of move data from left to right because it's boring and going out to use their skills to solve lots of problems. So, that's what I'm seeing, but what are you seeing in the data engineering space?

I think that makes sense. And I think there are two effects or two things that are different. In the data engineering world. One is the analytics engineering piece of, yeah, you're not writing transformations, you're not— As a data engineer, you are mostly divorced now from the business in that it used to be you had to understand, okay, what's the point of this data that we're pulling in? Great, we're not only, like, getting some sort of streaming data from this thing, it's like, we need to understand what it's gonna be used for because we are needing to write it in a particular way so that it can be used for that kind of use case. Now, and this certainly doesn't apply to every case and stuff like that, but, like, you can probably be a successful data engineer without having a clue what your business is doing. Like, you can get a thing of, like, I need to get data from here to here. How people deal with it is on the other side, and, like, I don't touch that. And sure, you probably are a little better if you know what's going on, and certainly, like, in practice, you actually would, but I don't think you really need to. Like, you're just, like, writing raw data from one place to another. So, I think that is one way in which it's different. And so, I agree with you, it pushes things back down to, like, looking for other problems. The second thing is, basically, like, the vendor ELT, the Fivetran and Stitch and stuff of the world have made it so a lot of those pipelines, and even things like what Snowflake can do and things like that, some, like, AWS products, have made it so those pipelines are also not things that you need a data engineer to do. You need a specialized problem to hire a data engineer. A startup of 20 people likely doesn't need that because all of the data you need to source is gonna be coming from either warehouses or from SaaS apps that you can go buy a tool off the shelf and that'll be the one that can write the thing into your warehouse for you. And maintaining a database is now a thing you just pay Snowflake to do, you pay BigQuery or whatever. All of those kinds of operational tasks, have been sort of abstracted away by vendors too. So, it does sort of leave a data engineer like, "What do you do?" The things that we typically see are one of two directions. One is this, like, data ops concept, which is kind of like DevOps-y type of stuff. It's like figuring out how to, you know, keep the systems up, maintain sort of the legibility across a bunch of different tools, basically make the thing work in a way that isn't gonna be designed for, like, okay, great, you have Fivetran, and you have dbt, and you've got Snowflake, and you've got Mode, and you've got Census, and you've got Hightouch or whatever, and all these things, like, those things need to actually communicate with each other in some way. This is a problem, I think, in general with, like, new modern data tools is you have a bunch of disparate tools that can't talk to each other. And so, it's like, okay, how do you actually create an experience out of that? Like, what is the experience of using this platform? And data engineers, I think, are kind of well-suited to think about that and the ops of it. The other problem is it seems like they are becoming more specialized in that there are some companies that have true data engineering problems and that's where you have to go to do this stuff. Your average 30-person company does not need a data engineer or even like a 100-person company does not really need a data engineer. Netflix sure does. Uber definitely does. The companies that are doing things at real scale, doing things where, like, timelines, latency really matters, doing things where they're trying to solve problems in unique ways that aren't just, like, connect Salesforce to a database and write queries on it, do really need those folks. And so, I think there's probably, like, some upmarket draw for data engineers where it's like, okay, I need to go work at one of these companies that has true data problems rather than a company that's just trying to munge a bunch of stuff together they can use some SaaS vendors to do.

Yeah, I agree. I think the data engineer comes into their own when there's a scaling problem. So I had Sean Maguire on the podcast a little while ago. I'd worked with Sean many years ago in New Zealand, and he's now at Dataiku. And he said, you know, one of the things he said was he went to a conference, which was a data science conference, and there was a presentation by somebody from Uber, Netflix, you know, one of the Airbnb, one of those. And he asked them the question of, well, you know, as a data scientist, how do you get access to your data? And, you know, the person looked at him strangely as if it's the weirdest question ever. And he said, well, I just select the data engineering team and 15 minutes later they send me a link and I use it. And so, you know, a lot of us aren't used to working at organizations of that scale where actually you can have a team of engineers that just make the problem go away. I think the other one is, as you've identified, is this idea of, companies that use SaaS products, yeah, that, you know, they're using Slack, they're using Salesforce, they're using HubSpot, and there are now solutions to democratize getting access to that data because it's a repeatable pattern. But the large companies, they are building their own platforms, right? They're not using off-the-shelf software as a service. And so therefore, again, we need an engineer, a data engineer, to solve the technical problem of how do you make that data available. So if we take the lens and we flip it to the analyst. So, in my experience, you know, again, I'm old enough to remember when the analysts were responsible for working with the stakeholders to understand the business problem. So, what is our problem? They were responsible and skilled to explore the data to see how the data might give us an answer of where the problem lies and potentially what we could do to solve the problem. They would typically create some way of visualizing or presenting a view of the problem using data and potentially a view of that problem have been solved. And heaven forbid, the really good ones would actually be responsible for helping roll out the change in the organization to make the problem go away, right? To close that loop and actually see the benefit. Right now, what I think I'm seeing is, Again, this hyper-specialization means that these analysts are getting relegated to dashboard designers. What are you seeing in that market?

There is some of that, though I actually put a little bit of that, like, I think that is in some ways the responsibility of the analyst to not do that. I think most data teams would say yes, like, they don't want to be dashboard designers. Their job is to help make the business decisions. They're there to to support, like, the strategic initiatives of the company, all that sort of stuff. A lot more people say that, I think, than are successful in pulling it off. Like, it's hard to do. And I think it's hard to do because dashboards are now relatively easy to create, and everybody wants them. And there's, like, a sort of increase in demand because you can do it more. All these people have access to a bunch of data. Like, say you're a support team. It used to be where it's like, okay, you use Zendesk, you look at the Zendesk things, like, that's kind of the data that you get. And the analysts are focused primarily on kind of the big company problems and things like that. Now it's like everybody has access to data. Everybody knows they have access to it. Support team wants access to, like, they want the same things everybody else wants, and they should be able to have them. And that creates a lot more demand for, we need to build dashboards on kind of stuff. I think as a data team, it is difficult to get away from that. It's easy to think, "Well, we need to build sort of the foundation, and then we'll finally get to these bigger questions." And it's a mirage that you never reach because the more dashboards you build, the more people have questions, the more things they need to poke at, the more, like, "Well, what about this? What about this?" that comes up. And so, I think at some point, it is on the analysts themselves to say, like, this is how far we will go and no further. Beyond that, like, we're going to answer these more important questions. Like, we are— yes, you don't have a dashboard that is exactly this problem, but so what? You don't need it. What's more valuable for my time is to work on these questions. So there's, like, some discipline there of saying, like, we'll only build so many. You know, a company with 1,000 metrics is a company with no metrics, basically. But, you know, like, I don't know how much this dynamic, like, the new data engineering, ETL, analytics engineering world affects this, other than it just creates so much additional data, basically, that while building an individual dashboard is, in theory, easier, you have this giant proliferation of things that you could do. You can sort of busy yourself forever with that work, more so than you could have in the past. Maybe there's something else to it, but it feels to me like this is a This is a job of the analysts to recognize where their value is and to draw the lines in the sand and say, "That's what I'm going to do," rather than the business needs changing or something like that. I think it's just like the job of the analyst is to stay the same. Even though the tools have evolved, fundamentally, their job is no different, and people should just focus on that.

I think when I'm working with a new team, one of the things I get them focused on is what I call how to make toast. Based on a really great video from TEDx. And it's this idea of what is your process, right? Who does which bit of work and when do they do it? And so we see a bunch of patterns normally that a team will come up with. So there's the prototype production pattern, I call it, right? Where they'll go, okay, we've got a problem. The analyst will go in and start looking at the data that's available. They'll start doing some exploration. They'll do some light prototyping, typically in a visual tool. Because seeing is believing. They'll iterate that with the business users or the stakeholders to see if they're close. And then when they're happy that they're on the right path, they'll push it back to the engineering team, right, to kind of rebuild it following all the standards and the processes. We then can take that model and sometimes break it down a little bit differently. So what I call big code or big design up front, right? So bore the ocean. And so, you know, We kind of understand the problem and the domain, and then we get the engineers just to build a whole lot of robust stuff without actually proving that that data is going to solve our business problem. So we get lots of time and lots of delay as they do all that work. And then the analysts get it back and they're like, well, that didn't actually do what we needed to answer this business question. And we go through iterations and loops on that. There's kind of a third model I see, which is chaos reigns, which, you know, coming from a traditional background where I was taught and I still believe that data should be modeled. But this model came out of the big data wave. And it's this idea that actually you can write pieces of code that go from the collection of the data through to the end information product or dashboard or reverse ETL into Salesforce. And that's bound as an object. And then you do it again and you do it again and they don't touch. And yes, you're not getting a lot of reuse, but what you're doing is you're making them all isolated and disposable. And so, therefore, your focus is on the speed you can do that process and the fact that you will never make anything reusable, but that's one of the, you know, that's your way of working. You've made a conscious decision for that. What do you see in the market at the moment in terms of the way the tools are encouraging those, you know, ways of making toast? Is there a general trend coming out? How do you see it? Lots of ad hoc, you know, lots of big modeling and coding up front or a prototype to production or something else?

So, this gets a little bit back to the cost of production has gone down so much that we probably have fallen too far on one side of the spectrum. So, the way that I would say it is, yeah, there used to be a world where building a dashboard was this giant thing, right? And kind of this initial thing, you go through the modeling process, you file tickets with IT, a month later somebody comes back with some, like, Spotfire dashboard. Board that isn't exactly what you want. You file another ticket, and 2 months later, you finally have a thing. And by this point, the project's already over, and you don't care about it anymore. And, like, knowing you have to go through that process, everybody is cautious about what they do, you know. Like, the, the people who are building the thing really want to make sure that they're building something that's good. Uh, the people who are asking for it don't ask for it unless they know they're gonna need it for a while, because they won't have it for a while. Um, there is no turn this dashboard around for me tomorrow, because that's just not how it's going to happen, you know. I I can't go to Slack and be like, "Build me this feature tomorrow." I know it takes a lot of time. And so, dashboards are similar. We have really lowered the cost of building them. There are some tools that try to favor that. There are tools like Looker, which are sort of traditionally shaped, and that it's still very much model up front, get dashboard out the other side. There are tools like Mode that are more built on the notion of, like, we need to do ad hoc analysis to answer questions that make building dashboards very cheap and easy. In all of those things, though, I think that the dynamic that comes out of it is you produce a lot of stuff. Data in general has a lot of exhaust. Like, there are a lot of things that happen that are kind of either one-off stuff that doesn't need to be revisited, or it's something you needed for a minute, or it's something where, like, we're measuring this particular metric and it changes pretty quickly because the business is changing. The cheap production of this stuff just, like, flooded the world with junk. And so, I think that, like, it's less a dynamic. To me, it is like, it is the third case that you are describing, but unintentionally. It's that we now can do all of this, so we just make a bunch of things that are kind of throwaway and don't necessarily think about what we do with them out the other side. In some ways, I think that is okay because those throwaway things are oftentimes the most important questions you wanna answer. Like, you don't need another dashboard. What you actually need is an answer. You don't need to go produce my dashboard. Uh, but I do think it's important for like the next sort of step of the industry in some ways is thinking about how do we actually make sense of all of these things that we have produced in a way that were not meant to be, uh, permanent. Like I, I have this— I wrote something a while ago about like even just the term ad hoc, where I think a lot of these you call ad hoc analysis. It's kind of whatever that— that's like become the phrase for, okay, you have dashboards and reports and you've got ad hoc analysis. The notion of ad hoc sounds very ephemeral. It sounds like a thing that's not important. It sounds like, "Oh, I will send you an ad hoc email that is a thing I hammer out on my phone on the subway." That's not how I think we should think of it. We shouldn't think of these things as throwaway. We should think of them— sometimes they are, but sometimes they're very important. They're things we need to persist. They're things that matter. They were just meant for a particular question that we're not going to ask 100 times a year. Sometimes also those are the most important questions. The questions you ask once and need to answer once are often the most important ones. If you have to ask it 100 times, it's probably not that important that you get it right every time. So I think it's figuring that out a little bit. It's figuring out a little bit of the way that we actually— what does ad hoc mean? How do we actually take the big questions that we're answering and turn them into something that is more useful and more permanent? Not in the sense of making a dashboard, but in making them a piece of knowledge that we can continue to perpetuate forward and that sort of thing.

When we're starting with a data analytics team and we're trying to, you know, beginning their agile journey, I talk about ad hoc and my definition of ad hoc is making shit up, right? We don't do that, right? Even though we've been agile and we've been iterative, we still need some processes and practices. We need ways of working. We need to be professionals in what we do. So, yeah, I can see what you're saying in that sometimes ad hoc just means, you know, rough and dodgy. Sometimes it's more around, let's do something iterative and quickly and see if it has value before we then invest more in it, right? But we've all done the one where we get an ad hoc question and then, you know, we do a quick dirty piece of work and then the next thing is, oh, actually, can you run that every week because you're now giving me an answer to a question that I need. So, Interesting thing you said there about metrics and junk. We've all experienced the data lake, data swamp problem. Do you think we're about to hit the idea of model or metric junkyards with the tools we're giving analytics engineers to write lots of stuff quickly, but not have a lot of rigor around the model and the reuse?

I think on the analytics engineers piece, of this, it's starting to emerge. It's harder to do it than it is on just, like, the question-answering thing because it's a little bit— there's usually more process in place to produce it. So, say that you wanna— say you're getting the ad hoc stuff that you're describing. Someone comes to you and says, like, "Hey, I have a presentation in 15 minutes and I need to know how many people, how many customers do we have in Latin America?" And you're like, they don't really care about the number. They need a slide. It's like they're going to round it to the nearest 100. And so you could say 400, you could say 700, and they don't, you know, it's like it's all the same. Um, it's gonna be a slide that's up for 15 seconds. That's easy to create, easy to do something, and it happens really fast. It's easy to throw away. Nobody checks it. Like, now you have this kind of lingering thing that somebody may find in a year and be like, oh, there's so many customers we have in different parts of the world. It's like, I was only counting Latin America. That doesn't actually say anything about other places, but the dashboard looks like it does, therefore I'm going to truth and, like, oh my God, now nobody knows what's going on. That doesn't happen as much in the analytics engineering world because if you're, like, building a new data model or whatever, it usually goes through some sort of peer review process, it's version control, all that kind of thing. However, that doesn't prevent us from adding a lot of cruft. Version control does not prevent tech debt. Look at any product that's been around for a while and you will see even the most rigorously controlled and developed systems with hundreds of people thinking about how to architect them well will still produce a lot of tech debt. And most, like, Looker models or dbt models or metric stores are not produced that rigorously. They're often kind of thrown together initially in a fairly ad hoc way and then kind of layered on and layered on and layered on to become this kind of, like, sort of a giant stack of sedimentary rock that just looks different from each layer. You can distinguish the point at which one person left the company and someone else joined because it all starts to change a different color. And so, I do think there is a problem there. It's a problem that is still early because all of this stuff is still evolving. We're still figuring out what analytics engineers even do. And so, we don't have easy ways to govern that. But it feels like an increasing problem. Like, there is probably a market. 3 years ago, there was a market for you to be a data consultant to come in and tell people how to, like, implement DBT and look ML. Today, there's probably a market for you to be a data consultant to come in and tell people how to clean up their DBT and their LookML models. I think we're probably getting to that stage of grief or whatever. And it doesn't mean those tools are bad or it doesn't mean anything's done wrong. It's just like, that's the next problem to figure out.

Yeah, I think maybe over here in New Zealand, we're a little bit late to the rest of the world. But in terms of our startup, we're bootstrapping. Both my co-founder and I still side hustle doing consulting gigs for large corporates. And right now, you know, the modern data platform has made a massive market for going into large corporates and helping them choose a tool from each of the categories and how to integrate it. And so, you know, each one of those takes 2 to 4 weeks because they have a governance process on how they can select something, can go through the architectural forums and all that kind of stuff. So there's a whole market now for consultants to go on for 3 months just to pick DBT, Snowflake, and get it approved and cobbled together. So interesting. If we look at that hyperspecialization though, and we're starting to see, and I've seen you talk about it in the awesome articles you write, there's a whole metrics layer, what we used to call a semantic layer, coming out as a subcategory in the market. So I'm, again, old enough to remember Business Objects universes, and I actually liked Looker ML as a pattern for ways of defining semantic language that's reusable. So we're starting to see that category come out. Are we gonna get hyper-specialization where we're gonna have a metrics engineer or a semantic engineer that works between the analytics engineer and the analyst who actually wants to use the data for good?

I don't think so. And this is, potentially a little bit of a way to solve the analytics engineer-analyst collision friction point thing. So, in the pre-metrics layer world, you've got people writing, doing in-warehouse transformations. So, you've got them writing, like, stuff in dbt, you've got them writing with Camel. Okay. LookML combines these two, but putting that aside for a sec. In that world, the metrics become very much like the collision because I've got a table that's dimension customers. I need to figure out what, like, our recurring, you know, what is our monthly ARR from these customers. There is some computation that has to go into that. That's not straightforward. You know, it's like, do I include these customers or exclude these? Like, what do I do with customers who churned? when do I actually credit the ARR? Is it when the customer signs? Is it when the customer, like, actually starts paying? How do we deal with multi-year deals? All that sort of stuff. So, there's a bunch of logic that isn't necessarily implied in a table that says, like, you know, customer, contract start date, contract end date. Like, there's a bunch of things there that you have to figure out. And the metrics layer basically moves that conversation into what is, to me, the analytics engineering world. It makes metrics a governed thing in the same way that, like, data models are governed. And this is traditional BI. This is what BI has done for decades. And this is what Looker did. Granted, they did it all, like, all of the modeling and the metrics modeling was done in LookML. If it's done now in DBT in a metrics layer tool, like, okay, so you have DBT for, like, the kind of join logic, and then you've got like a metrics layer for the, you know, how do I sum up dimension customers into ARR? But all that exists kind of before you actually get to the analysis. And I think that is a useful thing for like, okay, analytics engineers are kind of defining the logical governance of your data. The analysts then pick it up and do stuff with it. And I think like, there are some companies that would look at that and be like, we don't need analysts, we have marketing managers who can look at this and just like look at conversion rates and they will figure all this stuff out. And, like, okay, I can see why you might think that, but I think it's, like, not that hard of a sell in that world. Say, okay, the analyst is the person who needs to be able to say, I am making sense of these metrics. Like, analysis is not looking at a bunch of dashboards and then saying, like, well, this is up, this is down, therefore we do this. Like, it's much harder than that. And so, the metrics are just yet another sort of entry point for them to ask questions. But oftentimes, they will either have to, like, combine them in weird ways that you don't actually wanna put in a metrics store because you don't actually have that metric canonize in some way, or they'll have a metric that they need to pull apart where it's like, oh, this is how we compute ARR. However, the question we have today is about ARR with companies that are above a certain amount, that are in overages, that do this or that. We need to actually compute ARR differently for those folks because it just is like, that's what's called for in this analysis. Doesn't mean we're redefining the metric, or doesn't mean that we're introducing something new. It just means that the particular analysis doesn't— the ARR as it is written in the canonical way is not appropriate for the question we need to answer. And so, in those cases, the analyst probably steps back into what is sort of technically upstream of an analytics engineer, but who cares? They should be able to access the raw data too. The analytics engineer is sort of prepping the meal for them, but that doesn't mean that they sometimes don't have to go to the grocery store themselves.

Again, for me, it comes back to that idea of T-skills, right? Is that if the analytics engineer can define the metric in a place everybody else can use it, ARR, and we know it's a strong metric, then let them do it, right? But if they're busy and the analyst can define it using the same capability, then let them do it. If the analysts are going to define 15 versions of ARR, all I ever say is make sure there's a word in front of it. So high-value ARR or gold-customer ARR, just make sure that you're using a different name that says it's not the core definition, it's a variation of that. But again, it can be the analytics engineer or it can be the analyst. They both should be able to do that so the right person can do it at the right time.

You're not going to define all of them. I think that's the other kind of key thing in this is analytics engineers are not going to define every table. They're not going to define every model. They're not going to define every metric. Not in the sense that they're not like, there will be some that the analyst needs to define. It's that no matter how many metrics you govern, there will always be questions that call for some number that isn't one of those metrics. You're always gonna be some overlap there, but it's more the analytics engineers to me to find the canonical ones, the ones that show up on dashboards on an office TV screen. And the analysts are looking at those, but they're also thinking, for this particular question, as you said, what is the adjective that goes before these metrics that I actually use to answer the question that I have that's in front of me right now?

Yeah, and for me, it's also about once we start democratizing something, We need to bring in some form of federated governance, right? So as soon as we enable lots more people to do it, they will, right? They'll scratch their own itch, which is what we're aiming for. Then what we need is some way of saying, okay, well, how do we govern that without, you know, a large number of stupid data steward committees that will block everything from happening for 3 months? So it's that balance between empowering people to do the work and stopping ending up with a metrics junkyard. That will tend to happen when we go through that cycle. So if we look at the future, right, you know, there was years ago I used to talk about search BI and mobile BI, you know, things that were coming eventually and we still haven't got there for some of those. But one of the ones I see, it's kind of came for a while and it stalled, but I think it will come back as this idea of natural language asking a question and getting a response. So we've seen ThoughtSpot do it, you know, Power BI's got it, Google's got their Q&A service. The ability to say, how many customers have I got? What are my top 10 customers in natural language and get a response. And from my point of view, that's going to change the way the engineering or the analyst role works because we're not structuring the data anymore to be DIMMs and facts to go onto a dashboard. Like they may, be the model we use, but we're not coming up with a bunch of requirements. What we're doing is saying this data can be tagged or flagged as this concept or this descriptor. And then the system knows that when you say customer, yeah, it's looking at this piece of data, right? And when you say product and when you say order, it knows where the data is because you've classified it and it knows how to put it together to answer that question. So are you seeing that natural language kind of capability coming out in the future, or do you think it's just a dream? You know, something like Siri where everybody's got it, but nobody uses it? Or do you think it is going to come out in the market? And again, the way we work will change because the data structures we need will actually be slightly different. The way we work will need to be slightly different to empower that capability.

I have pretty mixed feelings on this. So, okay, on one hand, to me, natural language stuff is good enough to be able to do it-ish, I guess. It does some impressive stuff. You can do some impressive— natural language AIs have gotten pretty good. I'll give them that. Certainly, it seems like translating a question into a query is not that hard. I don't think it's very hard, but it's a problem that people have done a reasonably good job of solving. And certainly over the next 5 to 10 years, I would expect people to do a way better job of solving it. And it seems like, okay, so that's something we can do. The place where I, like, still have some skepticism on this, though, is the process of asking, like, a business question is a strange one because it is usually someone asking for a very specific thing in very vague terms where they don't know what's vague about it. Like, they will say, "Show me the number of customers in Asia." And neither of those words— there's only two words in there that are relevant, and neither of them actually make any sense. It's like, customers, what does customer mean? Do you count people who are nonprofits? Is it all-time customers? Is it past customers? Is it current customers? Like, is it customers that have churned? Is it customers that are— what is customer? Customer can mean a whole bunch of things. Is it customers who only make above a certain amount of money? Is it subscription customers? Like, all that kind of stuff is like, who knows? There is some implied understanding of what that means. That means different things to different people. Like, as an analyst, if you get a marketer asks you to show me the number of customers in Asia, you like kind of know they're coming at this from a marketing perspective and they probably care about it in this way. And if you're confused, you could just ask them. But if you get a CSM who's asking you, like, they probably care about the active customers because they're trying to figure out a staffing problem, whereas a marketer is trying to figure out if it's like worth putting money into Asia to spend more money on ads. Plus nobody knows what Asia means. Like, what does Asia— Asia could be anything that anybody— like, there's a huge range of things that could be. Um, people may think it means one thing versus another. As an analyst, you have this context, and I guess you could— like, AIs could have that context. And, and in theory, like, if a person can do it, so, so an NLP thing can do it. The other part of this I think is very, like, scary is you still have to put a human in the process. Like, it's, it's, it's the Tesla problem of like, great, this car can drive right most of the time. I still don't really want the car to drive by itself because if it's going to mess up, like, somebody needs there to tell me that. And if I ask an AI, show me all the customers in Asia, and it says, here's a list, there are 800, I have no way of really knowing if it did this the way that I think it did. Appropriate? Like, what did it interpret my question of customers to mean? And I guess I can ask it, but now it's, like, starting to get of, like, all right, what does customer mean? It's like, well, you could have to tell me a bunch of things. You're gonna show me some, like, technical definition that probably doesn't make a whole lot of sense. Like, it's the sort of thing to me where there's a bunch of complexity to that. It makes it very hard to manage because there is no language that is appropriate for the person to actually ask the question, and there's not there's not really a way to validate if you've got the answer right that you actually meant. And so, I think when you're making decisions on that sort of stuff, you want to have more confidence in that. I don't know that AIs will really get us there. Maybe closer, maybe there are things they can do. I don't know. But it's a thing that I don't have any really good thoughts on because it feels messy and hard. But also, these NLP AIs are very good.

So, I don't know, maybe somebody's smarter than I am and figures it So for me, I think you know when you talk about language, right? I don't think right now we have a language that is ready and available to enable this to happen. The other thing I find interesting is we don't trust the machine. So we go, well, you can give me the number, I don't know how I'm going to trust it. But actually we trust humans even though we know under the covers there's 16 different products and 5 people involved to take to take the data from the source system and actually get that answer. And yes, maybe we've got lineage, but actually, the complexity of the work they're doing with that code, we're not sure it's right, right? We're really bad at writing tests in the data world, you know, we're obscenely bad. We're getting better, but, you know, compared to our software brethren, we just get the data and we push it out there and we say it's right. So, yeah, I think, again, it's interesting how we trust humans, but we don't trust the machines. And we'll see if that changes. So while we're on thinking about the future, if I had a crystal ball and I said, look, what is the one thing I want to happen in the world of data? What I want is I want to grab a piece of data, I want to drop it into a bucket, I want the machine to model it for me and tell me what it means. I want to ask it a question and I want it to give me an answer that I trust. Do you think we're ever going to get to a world where AI, and I'm using my fingers to quote the word AI, will ever be able to do that, or is data always going to be just too complex and the language we need and the way we want to ask the questions too varied to let the machine do that magic for us?

Yeah, I think we're a long way away from that because the questions are often very complicated. Um, and I don't think it's like impossible to get to a point where it is you ask a machine for a metric and the machine tells you that, you know, like, sure, okay, give me a data point. I think we can get to give me— and we're talking about Purnell coaching on something, uh, give me insight. I think we're a ways away from that because those questions are very complicated. Uh, this is not a thing that to my knowledge anyway, that like AIs are particularly good at in any real reasonable way. Um, though I also am not that well versed in what AIs do. There is a way to me that this potentially works. If you think about the way that— okay, so, so take, take like a company like Sisu. Um, so I'm not familiar with them. They're like one of these augmented analytics platforms. Um, there's a lot of tools out there that are sort of broadly sort of shaped like this. ThoughtSpot sort of markets themselves as this. Power BI has some pieces of this. Google has some stuff like this. Is, they tend to all work in a way that, like, looks a little bit academic, where it is, give it a dataset, and it— give it a dataset of a bunch of, like, the, you know, size and shape of flower petals, and it'll spit out all these, like, sort of graphs of correlations where you can see, oh my God, look, the cars with 4 cylinders get better gas mileage than the cars with 6. That is not at all how, like, business questions work. Like, you don't have a giant, wide dataset of a bunch of dimensions where you're looking for correlations between like 50 different, different dimensions and measures. It does in a few places. There are a few problems for which that is true, but overwhelmingly, like, the business problems we have are things like, should we open an office in Japan? Like, what do you even do with that? Like, what is— what does a machine do? Probably what it does is it spits out like anything it can find that is related to Japan. It's like, okay, that's a starting point, but it's not really that helpful. It's these sort of wide— like, he's talking about this, like, T-shaped stuff. Machine— like, AI analysis to me is just like a very, very wide and not like an inch deep of, here is a bunch of things that are all kind of correlated, like, yeah, I don't know, go find a thread to pull on. The thing that could be interesting to me is if we get to a point where they help you ask— they basically think like an analyst where it's like, oh, I found this thing. Here are the 3 questions that you might want to ask next. I'll go look at those. And like, rather than just sort of spreading out really wide, it's like here. And then you kind of drill in and not like drill in and sort of like a drill-down BI sense, but drill in in the sense of like, now that I have learned this thing, what are other things that I want to learn that help me understand why that is? Or I see like, should we have an office in Japan? Well, we have a bunch of leads in Japan. Well, now my question is, do they ever convert? Why do they not convert? Those aren't drill-down questions in the sense of, like, I want to drill into this dashboard and filter by this other thing. They're a totally new question, but a question that sort of the genesis is found in the previous one. If you enable that kind of flow, maybe, maybe you get to a point where it's sort of the science fictiony thing, like, wait, show me this. Oh, that's interesting. Let me go over here and show me this. Like, ah, yeah, you could maybe get there. There. But it's, it's a very different model to me than take a CSV and find insight in the CSV. That is usually like, it turns out that men are on average taller than women. You're like, okay, thank you for finding that correlation, that is much appreciated. What else do we do?

Yeah, I'm a great fan of the, um, 5 Whys. And, um, also what I tend to find is the first question that a stakeholder asks is not actually the question they want answered. It's just the first question they know to ask. You know, how many customers have we got? Okay, 42. Right, where are they located? Ah, right, you know, 10 in the US. Okay, what are they buying? Right, so they have a series of questions they know they need to answer to get to the data they need to make a decision. So yeah, I'm with you in terms of natural language capabilities that help them do that themselves without an analyst having to be there if they can trust it. It's going to get us one step closer to democratization and self-service.

And there's another thing too that this is— they have like a note on this somewhere, like maybe a blog post one day. Uh, can AI say no? Like, this is another thing that I think is, is a thing that holds this back, is if I can ask the thing any question and it's like, you know, I asked the AI to jump and it tells me how high There are times when it needs to tell me, like, don't do that. Like, don't ask that question. That's a bad question. Go this other way instead. And that's like one of the things that analysts can do is they can help guide the process, not because they're just like, oh, you asked for this, here's your answer, here's your answer. They're like saying, ah, maybe don't ask that. Like, there's a better way of asking that. I understand what you're trying to get at, and you need to see this instead. Or you're finding a bunch of spurious stuff, don't ask that question. And I don't know that, like, an AI has to be pretty smart to be able to not only be able to answer these questions, but also to understand the point at which, like, the question you are asking is structurally bad. It's a question that will answer— you will get an answer in a way that you don't want it to be, or it'll be a way that you don't think it should be. I don't know that that is a thing that we are that close to doing either, but maybe. I don't know.

AIs do crazy stuff these Well, I think if somebody was designing that really, instead of saying no, because nobody likes it when the machine says no, it should just say 42. And every time you got an answer of 42, then you'd know that you need to ask again. So look, just time flies when you're having fun. And so I kind of want to close it out with one more question for you. And so, yeah, again, I'm old enough to remember when we moved from mainframes to client-server and that technology step. Spawned a massive change in the data and analytics world. And over the last couple of years, we've seen that same acceleration and change based on cloud technologies. Given all the new tools, given all the new capabilities, given all the new ways of working, has this change over the last couple of years made an analyst's life easier, more fun, or has it made it worse for them? Has it relegated their role to more drossy work? And less value work. What's your view? Are analysts in a better space now, or have we done them a disservice?

Better space, but not universally so. I mean, the tools are better. They're just nicer to use. They're a lot more powerful. You don't have to sit there and watch a query spin for an hour nearly as often as you used to. There's a lot of stuff with, like, databases where— this wasn't even that long ago, but, like, 10 years ago, you'd run queries and, like, the cluster would go down and it was just part of life where you'd run the thing overnight because you needed to and you'd come back and something happened in the middle of the night and now you gotta wait till tomorrow to actually get your answer. It's very frustrating. It's like much faster to get stuff. So also like in that same time, you know, you gotta do some analysis on stuff in Salesforce. Well, it's gonna take you a week to get to that point and you gotta like work with an engineer to actually do all this like kind of prep work to actually answer the question. You spend a lot more time just dealing with the system than you do with like the stuff that is, that is like quote unquote fun. However, I don't think that analysts have gotten to the point where they're mostly doing the fun stuff. Part of this is potentially the demand for data, basically, the explosion of the need for data creates an explosion for the need for dashboards. And now people are chasing what is a much higher demand. And so they're not actually working on the fun stuff to the degree that they might want to.

And so, I think there is still stuff to be done and figured out there.

The potential is certainly there for life to be better, but I don't know that universally it is because we've just sort of shifted the types of problems that you work on. Rather than, like, taking a week to build one dashboard and going through the frustrations of that, it's now, "Well, I have to build 10 in a week. Yeah, I guess I'm more productive, but I don't know that I'm producing more value, and I'm certainly not necessarily having more fun." So, I think there's It doesn't mean it can't be done, but it wasn't sort of like we just solved all the headaches of how people work with data and now magically everything's just a great time.

Well, look, I think we'll close it out there. Look, thank you for coming on the show. On the show notes, I'll put in a link to your articles that you do every week. So I'd just like to thank you for writing those. They're well-written, well-thought-out, well-reasoned. Based on fact and some opinion, but opinion grounded in fact. So really enjoy reading them as I think the rest of the world do. So thank you for taking the time of writing such great content and making it freely available. It's a thing of beauty when people do that. So yeah, thanks for coming on the show and we'll catch you later.

For sure. Thanks for having me and appreciate that. But yeah, this was a good time.

Datamagicians was another Agile Data podcast. If you'd like to learn more on applying an agile way of working to your data and analytics, head over to agiledata.io.