#262 Self-Service Business Intelligence with Sameer Al-Sakran, CEO at Metabase

Hi, I'm Maggie, VP of Curriculum at datacamp, and I know firsthand how overwhelming it can be to keep up with the rapid changes in data and AI. But staying ahead doesn't have to be hard. That's why our curriculum team has curated the best learning content to help you and your team stay ahead, covering everything from data and AI literacy for busy executives to cutting edge generative AI for developers, and everything in between. Whether you're upskilling yourself or your team, we make learning practical, engaging and flexible. It just works. Power your data and AI transformation journey on datacamp.com.

And it's your choice as a data practitioner or as someone who's running a data team. Do you want to have those questions come back to you as ad hoc requests, or do you want to factor that in to how you conceive of the shape of analytics you deliver? Your tool should let your users ask questions 2 through 20 without having to harass a human. So that's Self Service Analytics.

Welcome to Data Framed. This is Ritchie. The typical workflow for solving a problem with analytics is that a business person asks a question to a data analyst. They run some numbers and send an answer back. This then generates more questions and the cycle repeats a few times. Since cross team communication is typically slow, that provides a big drag on the amount of data driven decisions you can. The solution to this is to have the business person answer the question themselves. This is Self Service Analytics. It promises huge productivity gains, but as with many wonderful ideas, it takes a lot of effort to set up. That means that Self Service analytics has long been an unreachable pipe dream. But with tooling and training getting easier, I want to say that it's now within reach and how you might go about getting there. Our guest is Sameer Alsakran, the CEO at Metabase where he's building a low code self service analytics company. Sameer has a background in both data science and data engineering, so he's got a practitioner's perspective as well as executive insight. Previously he was CTO at Expo and Blackjet and he was also the founder of Simple, Hadoop and Adopilot. So let's ask Sameer how to achieve the self service dream.

Hi Sameer, welcome to the show.

Thank you, thank you. Pleasure to be here.

Excellent. So Self Service analytics has long went out as a way to scale business intelligence, but what does that actually mean in practice?

I think it means different things to different people and a lot of very smart people think it's completely non existent and a lot of other People that maybe are not as smart like myself, but have opinions, think that it is a different frame to have on how normal humans should consume the product that analysts, data scientists and data engineers produce. So I think it's, it's a perspective shift that I think is deeper than it is like a vendor like buzz or brand. I think the core idea is people tend to not have single questions. So you have something that is the sales performance of a product by quarter. Now most normal people don't look at that and say, cool, I'm done. There isn't really a moment where you're like, this was a beautiful dashboard, thank you so much for having built it. You deserve a raise. Typically it's like, oh, interesting what happened in this region? Or does this look the same across all sales reps? Or is this sales thing even working? Should we like pivot the company? And that typically those questions, which I'll just say questions 2 through 20 get asked and it's your choice as a data practitioner or as someone who's running a data team, do you want to have those questions come back to you as ad hoc requests or do you want to factor that in to how you conceive of the shape of analytics you deliver? So I have a, I have a horse in the race. The horse is basically like your tool should let your users ask questions 2 through 20 without having to harass a human. So that's self service analytics.

That is very true that there are very few business problems or business questions that can be answered which is a single question. You actually, you need that sort of level of iteration. So go on, talk me through. As it got easier to be able to do that where you're asking 20 questions in a row and it's a normal person doing this without some technical input.

Absolutely. I mean I think if you zoom out some like let's say from the 70s onwards there's just been the stair step of, you know, the forward march of progress has brought us much easier to use tools. If you were doing this in like the 1970s, you're writing, I don't know, Algol or COBOL or something horrific, you're using some crazy like proto driver, you're using a pre SQL API to pull all this data out, you know, the 80s brought SQL. Then you got spreadsheets. At some point you got like tableau which for all that were a competitor like did some amazing things to really reduce the complexity of asking questions. And I think that the average skill level required to productively ask any question in data Space has just ticked down decade over decade for the last 50 years or probably more. Honestly, like if you go back further, I just for a whole list of reasons, I look at the world as pre SQL, post SQL. So like my window ends 10 years before SQL was introduced, give or take. But yeah, I think that today people have much more data, much more context, much more understanding of what's going on inside of an organization than they did 10 years ago. And some of that is the proliferation of just reporting as a thing in every tool. And some of it is that the tools we're using both to collect data, to shape it into a way that is intelligible, but and then to pull it out have improved generation with generation. And if you fast forward 6, 18, 24 months, I do think the proliferation of natural language interfaces to data is going to increase and that increasingly you can just ask the damn computer like, hey, like what happened in February? And you'll probably get a crazy answer today, but in 24 months you'll probably start getting a viable physician answer that the average human can just like roll with and ask questions three through 20 at that point.

Okay, so here like the big change is really generative AI then. And I do agree with your point that things are getting a lot.

Oh, okay.

Not generative AI.

Go on. I mean, I'm way more excited about tool using agents than I'm not generative AI personally. So I think it's still LLMs. Like we're still discovering all the weird and wacky things you can do with a large language model. I still think that the problem of hallucinations and like random crap coming out of the other side of the generative process is very real. I do think that most of the current players in the space are too enamored with demo mode as opposed to actual real life question answer mode. But the writing's very, very clearly spray painted on the wall and we are definitely going to reach a point in quarters, if not year, probably not years of like, hey, I just want to talk to the computer and have it tell me things. So I think that we are in the process of at minimum a shift similar to the interaction of the mouse where you had a bunch of software that was the pre mouse area to type things and memorize keyboard shortcuts and keep a little set of notes about like your favorite set over here and so you memorize them. So a world where like you just got to click on things. And so I think similar to how there's transition from I have to memorize A bunch of commands to I can just click on things that happened. We're going to have from click on things to click on things and like talk in natural language or talk in natural language to the computer. I think that one's baked in. But if I can toss a grenade in the punch bowl, I think this whole thing of let's generate SQL based on natural language is not the path to go down.

Interesting. Okay, and this is because of the hallucination problem or are there other issues there?

I just think there's a subtlety to what most people want that is not strictly speaking human derived, but is nonetheless linguistically ambiguous. And to make that less of a torturous statement, I'll use the example of like, you have a business, the business makes money. There are a lot of terms to describe how businesses make money. You've got revenue, you've got like recognized revenue. You have bookings, you have earnings, you have ebitda, you have, you've got like all kinds of things I'm too checked out to fully understand in the accounting world. But like all these things have meanings and you can mix and match them. And I think that there is insufficient coverage in the training corpus to really disambiguate all this stuff and to then turn that into, I have an arbitrary schema. Let me figure out what magical joints do and then like what revenue recognition schedule to enforce on top of that. Like that just isn't going to happen for a while. I suspect if you draw the line out, it'll happen at some point. I don't want to give predictions in like number of months or years for that. But like we're not there. I do think that, hey, here are some numbers that we know how, how to describe. We know that these numbers have certain semantics and certain meanings and certain just like contexts that the human in this company cares about. And dear computer, can you pretty please massage these numbers and display them to me in a certain way. That's very tractable and you can make that work today. So generate some SQL based on net revenue retention isn't happening right now and it's comical every time you try especially arbitrary schema. But hey, here's a bunch of metrics I defined. Here's semantic layer based on the semantic layer. Dear computer, please tell me what happened in February of this year that made revenue go down. And you can probably cobble together something very, very compelling. So it's not that like large language models are pointless. It's just like it's not going to.

Be that easy okay, so this idea of a semantic layer where you've got some sort of business meaning baked into your code, this seems very important. Then the maybe more hand coded SQL at a low level and then you're using the generative AI, the natural language interface on the semantic layer, is that correct? Actually we probably need to back up and just explain what a semantic layer is as well.

Yeah, so like, I mean humans have a different mental model of what's happening in their business or their organization or their massive multiplayer game than the data model that lives on disk. And so in your mental model there's certain relationships, there's certain concepts that are like the primary concepts. So for most people in a business setting, retention is a capitalized word. It's a very specific thing. There are different kinds of retention, those kinds are important and do different work, different jobs and that the definition of what retention means is non trivial. So there is a generalized, hey, it means that the number of people that stuck around. But if you start having price tiers, you start having reactivations, you start having people that pause the subscription and come back, you start having people that upgrade or downgrade, there's all these things that impact the actual number and that has to be defined somewhere. And so the semantic layer effectively is where is that defined? And so is that defined in code and SQL in pre processing, is it defined in your dashboard in code where you mark up the data afterwards? But at some point like the revenue recognition schedule has to be applied or the logic for retention has to be applied, or what activity means has to be defined? Like is it active user somewhere logged in? Are they someone that looked at a photo or are they someone that like saw that full video? Does it include people that are like drive bys that are not logged in? Do you want to combine your logged in and you're not logged in session? Somehow like all that stuff has to be woven in somewhere. And I would say that's not strictly speaking just the semantic layer. It could also be, you know, pre semantic layer transformations as well. But it's that collapsing of your whatever correct or right data schema you have for storage and transforming that into something that more closely maps to the end user's cognitive model of what they're looking at is just like for this conversation call the semantics asterisk layer.

Yeah, all those points are definitely very well definite pain points at data camps that arguing over well, how is this defined exactly? And it always gets very complicated because sometimes it involves product details, sometimes it involves like different commercial teams. And then You've got, obviously the data people need to be involved in that as well. And going back to your original point, I say you define these like normal people wanting to be able to work with data. How does this semantic layer stuff get defined then? When you've got so many different teams involved, who needs to do it?

I mean, who does? It is a pretty gnarly and pretty politically charged conversation. So practically speaking, it'll be some mixture of data engineering, finance and or business operations. It'll probably involve the analyst team or some places it'll be the data scientists. But functionally there are people that implement this and people that figure it out. Currently, typically the center of gravity is along with the people that know how to turn those decisions and that information into something that can be executed. So data engineers that are doing this in dbt, you got some tools or some organizations, it's the analysts that are kind of just rolling this up. In some places it's in the actual schema at rest and in other places the data warehousing team takes care of it. But fundamentally those decisions come from somewhere. So there's some accountant somewhere in the organization that read something that the IRS puts out, says oh, we need to recognize SaaS revenue this way and services are in this way. And so that decision has to flow through to the people that implement it. And then in whatever platform you have, you implement it. The thing that I'll point out is that that translation is error prone, tedious and the source of a lot of heartburn. And so anything that you can do to reduce the skill level required to capture those decisions and to put them into whatever execution like format you need pays off organizationally. So if you're forcing this to all be done in super complicated Scala modeling somewhere in the chain, then you're forcing someone that speaks Scala to talk to someone that speaks accounting and you're relying on that transmission path to be error free and plot twist, that's not going to happen. So the degree to which you can give your accountants tools to capture this and getting it as close to the accountants or the finance people as humanly possible, or the product manager that is running retention, or the customer service rep that's dealing with, or the customer service manager is dealing with like what user satisfaction means. And so the ability to federate or decentralize the capturing of this information is to like your overall interest. And I do think that there is a little bit of protectiveness or defensiveness or just desire to be part of the data priesthood where it's like oh no, like the data priesthood must decide how like this model happens and like we are the, the blessed keepers of the DBT repository and like only we can do this. And the reality is, in my experience, people in the line of business are much more sophisticated in understanding these concepts. So the average marketer has a much tighter understanding of a lot of marketing ops numbers than the average data scientist or analyst or data engineer assigned to market. And so being able to put more of a load onto the people that have the competitive model as possible is beneficial. And dumbing down the tools that you're doing all this in is very beneficial. And so letting again, creating a tool that a normal person that understands their space deeply can type in the things or say the things they need is in that best world for this. I think in reality this compromise is made. So SQL is better than Python, is better than Scala, and something that's sort of gooey and easy to use is better than SQL.

Yeah, I love the idea of the data priesthood. I think some people just want to wear robes to work. But yeah, definitely you've got this big gap between if you're doing some data engineering and you're doing some accounting or marketing and then, yeah, how people talk to each other. So I like the idea of bringing people closer together and making a wider variety of people being able to use these things. So maybe we need some motivation here. Have you seen any success stories for where this has worked really well? And you have had people who aren't that technical being able to perform analytics?

We have a large customer that has managed to do this pretty well. I think what it looks like for them is I want to say they have a Data team of 5ish people supporting a 2000 person company. And a lot of what that data team does is they're creating data sets and starting points for people to use themselves. And so the lens they view themselves through is not, we are here to create things that people will then use. We're here to facilitate people asking their own questions. And ideally, if people are not able to, we'll try to figure out how to make them able to. And so the perspective that is held is that analytics is not a service desk. It's not a place you go to to say, hey Bob, can you please tell me X, Y and Z? But rather it's hey Bob, I'm trying to figure out this number and I can't do it. Can you like tweak the model such that I can? And so I think that is one of the primary shifts I do think also, and so this is my own personal opinion, but establishing some sort of, let's say, an institutional no that is liberally applied towards bespoke analytics requests and not letting the data or analytics team become a place where. Sorry, I'm trying to figure out who I want to annoy and how badly I want to annoy them. Can I go into a brief digression and I promise it'll come back to or describe it?

Sure.

All right. So there is this common corporate pathology where if I'm in a meeting and I don't know what's going on and there's a decision has to be made and we've got to make this that. One of the socially acceptable outs for me, not knowing what I'm talking about, is to demand more data. And it's a pattern that you see over and over and over and over again. It has a lot of names. It could be like we need to be responsible and look through all the relevant information. We gotta be data driven. We gotta do X, Y or Z. But the fundamental thing is I want to deflect my responsibility, either individually or collectively from making this decision or figuring this out. And I want to now put another burden on someone else, which is, hey dear, so and so, please get me more information. And it's often done in a very vague way. So it's often some version of like can you do a retention analysis for me? Or can you like look into blah blah and the vaguer it is and the more work that is formed out and the more some poor analyst team has this like scurry around at like 3am on a Sunday, like really the better it gets me out of my pickle. And so again, like going back with that digression and coming back in, I think the ability for the analytics department to say no to stuff like that and to say no to the random barrage of ad hoc queries and to force the department to be enabling as opposed to just mashing out dashboards, analyses, charts on behalf of individual questions is what lets you scale that team and critically foster self service in the organization. So an organization where anyone with power always demands a bespoke analysis and everyone else is left to fend for themselves isn't really doing self service. What they're doing is they're starving people information unless they have power. And for the people that don't have power, it's like, hey, go do this thing whenever you want. I think the ability to say this is how we're getting data into your hands if you want to hire an analyst on your team and you want to make that up the thing, that's great, feel free to do that. But fundamentally we are decentralizing and distributing the load of asking questions, and the decentralized team is there to facilitate, enable and iterate on the data sets until this is possible. Digression number two, I'll bring it back. Which is, it's actually a lot of work to create a data set that can be self serviced and it's not work that happens magically. Like you've actually got to go in there, make a schema shape, make a semantic model, show it to people, realize all the ways in which it sucks, come to grips with the fact that you're not as smart as you thought you were, that you're going to have to take the feedback of your users, weave that in to create something simpler. You have to let go of things that you think are quote unquote correct and try to map this to the end user's cognitive model. And all this is iterative. It takes work, it takes some humility. Takes a lot of skill. Actually takes a lot more skill to dial in a model that accurately represents the user's coveted model than it does to just like take whatever crap you have on your SQL tables or your like parquet files or whatever, wherever you've got going on these days, and then just be like, I don't know, that's the data model. And I guess I should, let's like turn this from like an integer into a text field or something, I don't know. And then call that a model. Like there's actually a lot more work to it. So like do the work maybe is like the motivating thing, which is like do the work to let yourself be lazier in the future. So it's all about front loading the work you do and then foisting this all off on your constituents to do themselves. All right?

And then hopefully one day life is glorious for everyone because you've done all the hard work up front. So I have to say I don't think this was your intention, but I'm now thinking it's a great career skill if you want to put off making a decision. You're like, oh yeah, we need to do some more analysis on this and you get to delay it. I'm curious, how often do you use that technique yourself?

I try to catch myself when I'm doing things that future me would want to slap the present me for. So there is a certain.

Yeah, that seems Like a good rule of thumb. It's like if you're going to be disappointed in yourself later on, then yeah, probably don't do it. But. Sorry, I interrupted.

Oh no, I just say like there's. So maybe like this also segues into a different thing which I do want to cover in this context because I think I would like this message to spread a little further, which is every time I hear someone say they're data driven, I kind of assume they don't know what they're doing. Ouch. Very few people that I know who are like data practitioners, ML researchers, good operations people. Like data driven is just one of those cringy words that is really the symbol that you don't have proper nouns for. You do it. That you are treating data as this magical thing and that you're arguing the space of magic as opposed to concrete tangible things. One of the things I've said over the years is data is a four letter word and it's really a symptom of ignorance. And real people have real words for stuff they care about and understand. So you have users, you have customers, you have visitors, you have contracts, you have videos, you have like subscribers, you have advertisers, you have creative pre rolls. Like these are all things that have proper nouns that have semantic meaning and they don't live in the world of data. They live in the world of like actual real concepts. Like data. Is this just aggregate noun for a bunch of stuff I don't understand. And data driven specifically is usually some version of like well all the data tell me. And there's this like really just to get super dorky for a second, what you're really saying is I'm not going to be aware of the model that I'm using to evaluate this data. I'm going to pretend that my naive messy sloppy model is somehow capturing reality. And I'm going to then defend my interpretation of a handful of sparse data points on the basis of this hyper simplistic model and pretend that the data is telling us what to do and it dramatically under represents how much interpretation of these data points is happening and lets you pretend that your sloppy model is actually like just what the data is. And you know, the functional reality is that all these semantic things we're talking about have specific meaning. There's a model that define that meaning in that context and that the awareness of what that model is lets you make good decisions. So again, to bring it all back in, to not have this be a one hour rant about me going off which, you know, we can do some other day. I do think it's important to be cognizant of how you're evaluating data, what it is you're looking for data to answer to. Fess up to the a priori mental model you have for what the data says. And then when you're having conversations about making decisions to do that with full awareness of your simplifications and what it is, how it is you're interpreting it. And this, for me, mostly shows up in practice when people are discussing, like, findings that are drawn on an aggregated view of the underlying system that basically drops all the interesting texture and information in it. And this is a very, very common pathology for junior analysts and junior data science to do, where you just like, sum a bunch of stuff, you look at a couple dimensions, you say, like, oh, we should just change the conversion funnel because Obviously there's a 20% drop here and 20% is bad and there's a 2% drop everywhere else. So clearly this step is a bad step. And at some point you're like, but dude, that's where we ask for a credit card. Of course there's going to be a drop. There's certain things where the physics of the system or the actual reality, the things that have the tangible reality and what it is you're measuring, it's not just a bunch of numbers. There's a thing happening there. And if you don't understand that thing, you look at this funnel and you say, there's a kink here that's really strange, we should fix that kink. And it's like, thanks. I would never have thought that asking someone for a credit card might cause people to drop off. So going back to humility and just understanding the system and fessing up to the state of your knowledge about the system you're studying and not just treating as a bunch of numbers that you smash together and put on a chart and then interpret the chart as if it's just this abstract, idealized set of time series.

So I definitely agree that if you're not relating your data analysis back to the what's really happening in the business, then you're going to come with nonsense, you're going to get bad ideas. But you mentioned that data is often just a synth. Well, I don't really know what's going on. But isn't that the case that a lot of the time in the business you actually don't know how to proceed and therefore you need to use data in order to get some insight?

I Don't think so, because I don't think data is what informs you, what's going on. I think data is what you've put into your mental model to then interpret. And I think there's again, like I mentioned, sleight of hand because I think that's what's actually happening, which is you're assuming, for example, and just like groundless in reality. I do like the conversion funnel conversation a little bit because it's generally applicable to many businesses. There's lots of things modeled as this. I do think the awareness of the segments of users going through is critical. So I think that treating a conversion funnel as just a monolithic blob is okay from some perspective. But if you're looking at that and you're just saying this stage has a drop off to this stage and I'm going to interpret that, I think if you don't understand what that stage is doing, what's actually happening in the different segments of your user base for that, are you flushing people that should be going through the funnel properly, or are you losing people that should be successfully going through the funnel, or are you disqualifying people that should not? Like that's a different conversation. And so I think just looking at like that 70 people, 70% of people here down to 50% of people here, that that's bad is simplistic and kind of naive. And I do think that this is where understanding the physical system that your numbers are capturing, there is an underlying mathematical model that is in your head that you're interpreting these numbers through. And your awareness of that model is critically important and to understand what's happening. It's not necessarily. You keep slicing data until it gives up and you find some sub segment with some metric that looks different. You're like, well, it seems like we have lower conversion in eastern Moldova than we do in the mainland of the United States. It's like, cool, but what's happening and why? And I think this is where domain knowledge, I shudder to say, but qualitative research understand the world that you're living in, not just like the domain. All that needs to shape the model that you're using to interpret these numbers through. And I think that there is a core skill in specifically data scientists, more so than other segments like analysts and data scientists, more so the data engineers, where understanding the system that you're measuring is what makes your work viable and interesting. So there's a certain like junior analyst disease where you run some numbers, you show something's correlated, you know, you show up to the person running that business unit. Like I discovered there's a correlation between asking for credit cards and suit user drop offs, therefore we should not ask for credit cards. And it's like really? Like, come on. And so I'm not saying don't look at data that's exactly not. I'm actually saying the exact opposite. But it's understand the perspective you're bringing onto the data that's up to simplicity or the fidelity of that model that you have in your head. And in my own attempts to make everything jargony, which is not true, but like, just for the sake of this conversation, I would call it the model driven as opposed to data driven. And it's the awareness of how it is you're interpreting data and treating that model that's in your head as something that's malleable and something that's explicitly stated as opposed to something that's implicitly hidden behind the interpretations you make of these random data points in time series and histograms and whatnot.

Okay, yeah, I'm definitely with you there. That you need to make sure that you're not just acting blindly on the numbers look like this. And it needs to match back to some kind of business model that either you have in your head or written down somewhere. All right, so I want to go back to a thing you said earlier about. So you gave an example company with five data scientists supporting 2,000 people, and you said that one of the big changes was that the central data team that were just preparing the data to make sure it's high quality and everyone else working on data was embedded in business functions.

That sounds very cool, but not that they're embedded. So it's not that there was embedded analysts, it's that people that would in other companies be serviced by analysts were doing this themselves. So the key thing is not that you're embedding analysts in all these teams. Sorry, that was a throwaway line of me making fun of like the big scary VP who's demanding things. It's more that there was an emphasis on creating data sets that were interpretable and intelligible and using tools that were simple enough for the end users in that function to use themselves. Like Metabase. All right, wonderful.

Yeah, I like that idea. So everyone else in the company is sort of implicitly analysts because they can use the tools themselves. How do you get to that state? It sounds like it's the dream, but maybe quite tricky to get to.

So where do you begin? Data. Data shared. So the data that is put in front of normal humans has to be interpretable and usable by those normal humans. So no matter what the correct way to store data on a data warehouse is, you want to present a form of that data to an end user that matches their mental model. So excessive normalization is a really bad idea. Mixing and matching foreign keys, embedded arrays and join tables is a bad idea. So if you're exposing through tables in your data model to end users, you're just doing it wrong. It's pre aggregating or shaping the data such that the things that end users really care about come out quickly as opposed to requiring a self join and a bunch of crazy filtering. So it's really just like pre chewing the data for your end users and not stopping at this is the correct way to represent events. We have an event log. Do whatever you want. The answer to that is like I can't do stuff with event logs. Can you pretty please at least sessionize this or at least pre aggregate a few things or just make this closer to what I think of when I think about product analytics. I think that there's a fair amount of work that is again, high skill, difficult and intrinsically error prone and hence iterative in nature that this data team does and hypothetical data team at your company need to do to get something in front of people they can use. And there's a whole corpus of knowledge and set of tools around product design and how we build products that are usable. There's a lot of things that we do casually today that in a different generation would have required training in this specific tool that today are just like, I know how to click on things. I'm like, I see a bunch of prompts and things make sense. And, and like I'm not using weird crazy keyboard shortcuts in AutoCAD, I'm just using Figma. And Figma just kind of lets me do things very, very easy. And so I think that all the things we learned on the UX product design side of the world can be applied in metaphorical sense to data set design and that we should be treating the design of data sets in the same way that we treat the design of user experiences for end users. Because you're designing data and the UX of that data is what's in the columns, what is being, what are the connections? Are things pre aggregated? Are they sessionized? Are you pre computing certain metrics on certain things? Are you forcing users to engage with something very different than their mental model? And basically to what degree the thing that Hits them is just whatever the right format is on a SQL in a reg database, but versus is it actually something they're able to engage with? And yeah, that's a lot of work and you're going to get it wrong and you're really designing for someone whose brain is not like yours. So you're going to have to take a shot at it, show it to them, see what they think, watch them use it, watch them fumble it, take that, go back, redesign it, put in front of them again. And you either need to do this in a user testing sort of way where you just like show in front of people and ask them to do things and if they can't do them, you ask them like why they cannot, or you'd have to instrument everything and make sure you deeply understand which columns are being used, what custom navigations people are using, Are they having to tap an analyse on the shoulder and get a certain join or a certain multi stage query done for them to then reuse and edit? But there's a lot of looking at what your users are doing, understanding whether they are able to use the data set that you designed for them or whether they're just looking at this being like I don't know what to do here, can you please get me retention numbers for the last quarter? And again, that's a lot of work, but it is front loaded work that then lets your constituency or your stakeholders or the line of business folks be responsible for queries 2 through 20 and not you.

Okay, so this sounds like we're getting quite close to the idea of data products where data is packaged up in a particular way and it's got associated documentation, it's designed to be used by other teams without your sort of intervention. Do you want to talk me through what needs to happen to go from like data to data product?

Then I would maybe remove documentation. Max. I think documentation like in product design is a crutch. So you know the statement that you don't know how to use this, therefore I'm going to write some words for you that will inform you of how you should use it, is kind of a crutch for bad design. And so I think the mechanics are some form of sketch out, just like the entity diagram or just like what the system that you're studying in a company is, what the primary nouns are, what numbers matter to them, what are the aggregate metrics, what are like the per entity metrics. You just kind of like get that out on a whiteboard or on notepad or something with the person who's in business, you like rotate that in your head a bunch of times. You try to understand kind of like how that maps to the underlying stuff on disk and you understand what questions they ask, you understand what navigations are required, you understand what were the common filters, how they think about those filters. Are there things where you have in your mental model a foreign key where it's actually nesting or composition, Are there lists of things that are being represented? Foreign keys? But there's a fundamentally different set of Legos in that cognitive space than in SQL. And so you somehow have to map the separation concepts and connections and et cetera in the SQL ish data world and reshape it towards something that more closely matches that entity diagram. So you're going to do a bunch of pre joins, you're going to like pre compute a few things, you're going to have enriched users. There's all these things that you do that ideally get you closer and closer and closer to that naive bar napkin diagram of how you use this. Think about the world and as you get closer, you show it to them. You basically ask them to using whatever tool you're using, say, hey, can you figure out how many people signed up in eastern Moldova last week? You watch them do it and if they get it wrong, you just take notes and you go back and you try to reshape what you put in front of them. And you should probably emotionally budget for three to 10 iterations. So if you think you're going to get it right the first time, you're delusional and you really should just like fess up to it. It's going to take you a couple shots. If you're really close to the kind of person that's doing the querying, you'll get there in three if you're very different than the person doing the query. So if you're talking about like you're a data engineer and you're getting a data set for performance evaluation for a backend team, that's probably three iterations. If you're a data engineer and you're trying to do something for call center reps, it's going to take you 10 and just emotionally budget and like time budget for that and just go through the process and don't give up when you put it in front of them and they can't do things. I think the most common reason people don't believe in self service analytics is they went through this process with the expectation that they're just going to have a user conversation. They're going to get the requirements, they're going to like write the right data model for those requirements, they're going to hit submit and publish and that magically the end users are going to know how to use this. And when they don't, they'd be like, oh, it turns out you have to be part of the priesthood, otherwise you can't really do analytics. Aha, I'm special. And it's like, no, dude, you have nine more cycles, go through them.

That sounds amazing. But also it feels like it's very difficult to budget for 9, 10 iterations in practice because someone's going to say, well, why does it take 10 attempts to do this right? Shouldn't you do more planning up front? And so how do you go about building in that culture of iteration with multiple teams working together? Certainly I think the cross team coming together over and over again, that's generally organizationally tricky.

I think that there's two different parts of it. So one of those is there's a difference between phoning in the iterations and having the iterations, the improvements. So I think if you're able to say, hey, we're going to start small, we're going to get some stuff in front of you that takes care of questions two through five and we're just going to get something, we're going to do our very, very best to get you something that answers 20% of your questions and that you then build on that incrementally. So if you're trying to do the whole thing, then you're kind of doomed. And again, I think that there's a lot of things we've learned in software engineering that should be applied to data engineering, but they're not really. I think people are coming around to version control and testing and observability decades after the software engineering fields are holded. I think there's things around iteration and de risking and product discovery that again has been part of the playbooks for at least 10ish, maybe 15ish years that the data engineering community needs to copy over into their world and again copy with modifications. You're not going to do it exactly line for line, but in the same way that you wouldn't build some new social network in one shot and then when it didn't work, you'd be like, well, you know, we don't really have time to do more than one iteration. At some point you have to get the organization to understand that waterfall software development is really, really tricky and is likely to flame out and Delivering analytics in a responsive, iterative, adaptive way is actually a good thing. And that the organization's better served by us delivering 10% than another 10%, then another 10%, then another 10% in weeks 2, 4, 6, 10, and 14, rather than taking a year and a half to deliver all of this in one big mega project and then probably fail. So I think again, in software in general, mega projects don't, don't turn out very well. And a lot of the ways we found to de riskless is to ship things in smaller increments. And I do think, again, that is a lesson that for the people who are running data infrastructure and data teams, they should internalize. And in some ways it's the job of the director of data or the VP of data or the manager that's responsible for it to contextualize this with a broader organization and to show them a that they're better off. So, like, they're going to get this like, train of goodies every quarter and that just more and more will show up and to wean them off of the delusion that you're going to get it all right in one shot. Because again, like, software doesn't work that way. And I do think that all the data stuff we're doing is largely software. It's not exactly software development, but like, it's really, really close in many core ways.

Absolutely. And that's really interesting that there are a lot of skills from outside the world of data that are actually really useful for, for data professionals. So you mentioned the idea of like switching to an agile methodology from software engineering. You mentioned things like user experience skills and design skills. Are there any other skills that you think data professionals ought to have from outside the field?

I think those were like the primary ones. I can dig deeper, but I just reinforce the emotional budgeting, which is very few software engineers today come up thinking they're going to get it all right the first time. Whereas comparatively, I feel like I talk to a lot of analysts that like, are going to spend eight weeks on a dashboard and deliver it and then hope for the best. So incrementalism is probably the biggest one. And it is kind of a soft skill because the practice of chopping up a bigger set of dreams into a set of milestones is a skill and it's difficult. And so the ability to do that planning, whether you are an engineer manager or junior analyst or data scientist or someone changing careers like that is a thing. And in some ways it is more predictive of your success than a lot of the hard skills are so you're great at SQL, but you suck at decomposing the problem. And you keep biting off more than you can chew, like, guess what, you're not going to deliver things. So that ability to deliver and that ability to look at a big problem, understand where the seams are and understand how to match the seams, where it's easy for you to code or easy for you to write, or easy for you to generate or like design with the ones that correspond to what the people you're giving this to actually need from you. Those themes are never matched perfectly. So it's very, very rare that the engineering or the data science milestones correspond to how users perceive value. And that the art of knowing where to slice is something that I honestly think is a very significant determinant of how successful you'll be. And so getting practice at that, because you're not going to do right the first time. So that's one of the things that look for opportunities to practice, look for ways to work towards it, and critically be less defensive about why your initial set of milestones is correct or is right, and instead think about from their perspective what they're getting at milestones 1, 2, 3, 4 and 5, and use that kind of sensibilities of what they're getting from you as a way to better serve them. So in many companies, data science or analytics or data engineering, you're successful through other people. You're not like building this grand edifice that will be shown on Architectural Digest. You're helping people with jobs do that better. You're helping people make better decisions. And the degree of success you achieve is largely measured by do they make better calls? It's not, do you make them feel stupid and do you get to feel smart about it? It's are you actually able to get things in their hands that make them better versions of themselves? And I do think that that viewpoint will go along A specific viewpoint of is my plan for how to ship this giving people incremental wins and does it make them do better at each step is a key thing to selfishly learn because it will make you look better in tune.

Excellent. So I agree that coming up with milestones within a project, very difficult. It seems like this is part of a larger suite of skills around project management. So do you think data professionals also need some project management skills or is that something that should be outsourced to, like a dedicated data project manager?

I think that I would push back a fair amount on that statement and I'd say That problem decomposition is a key determinant of success for pretty much all roles. And if you're relying on someone else to do problem decomposition for you, you're basically relegating yourself to be executor and that you're effectively putting a really, really severe ceiling on how large and how impactful a project you can run yourself. So yes, there will be project and product managers that do this at a different level, but even within like task, if it takes more than one day to implement, you're doing problem decomposition whether you want to or not. Now you could be doing problem decomposition that you're going to sit there in your editor and just like, you know, hit run every while, every once in a while and see if it works or you're being intentional about it. So maybe the statement is be intentional, be aware of like what the consequences of your decomposition is. And the main thing, I'm just saying is that like that skill of decomposing a larger, messier problem into a set of steps is a key thing that everyone should be learning. And I actually would say it is one of the largest determinants of the success of analysts and data engineers that I've seen in my orbit, which is if you're able to do that, you're able to tackle bigger and bigger projects at a much higher rate of success. If you're relying on someone to give you a bite sized pre shoot project, there's where they've figured all this out for you. Then you're kind of relegating yourself to B league.

All right, so before we wrap up, what are you most excited about in the world of analytics?

I'm excited about analytics kind of going away and I think that there's a couple forks in the timelines that we're in. I do think that historically analytics has been defined by what people sell. So you buy a business intelligence tool, or you buy a dashboarding tool, or you buy a GIS system, or you buy a data science platform and there's all this really dumb, self limiting things that the industry has done that are really just there so that you can be in the right quadrant on the Gartner reports. And I think that a lot of that is increasingly being silly. And I do think that there's a recutting of what the domain of a given data product, team, whatever is. And to state more concretely, I don't know why so many analytics systems are read only. And I think we're doing some stuff that makes that weaves in, writes and transforms modifications. I Think, like, this missing stuff our competitors are doing. There's other players in the space that are mixing and matching internal tools, reporting and charting. I think, like, Airtable and Retool are both adjacent to us and doing things that blur the lines. Supabase does things that blur the lines between data exploration and, you know, RM and modeling. And I think that I'm very, very unexcited about the future of bi. I'm very excited about the future of, like, I've got a database with stuff in it, and I've got humans that want to, like, do stuff to that information and they want to view it, they want to massage it, they want to use it, they want to integrate it. Downstream tools. And I think that the specific container that a lot of people building tools are kind of forcing us into and not respecting the natural desires of the person staring at those pixels, but instead being like, oh, no, it's out of scope. It's not something that we're a dashboard tool. We don't put buttons in there. And so I do think that a lot of that is slowly being shaken loose. And so I'm excited about the next step stage there. And I do think one of the things that will really, really drive us home is just the world of LLMs and generative AI, where it's just like, this thing can do so much more than just repackage up a time series and vomit it out as a chart. Are you really going to stop there? And why? What is that boundary that you're hitting that you care so much about? It's like, take that data, throw it to another agent, have that agent do some stuff for you, have it all be packaged up and sent over to third service. Have that service and do some magical things for you. Do that whole thing in one playground and don't stay in your lane quite so much. So I'm excited about people building tools not staying in their lanes.

All right, so really going beyond just, I'm a consumer of data, so I can actually go around, play around with it and have, I guess, taken action on the data, even though I'm not that technical. That sounds like a bright future. Yeah, I'm looking forward to it. All right, excellent. Thank you so much for your time, Samir.

Likewise. Been a pleasure. Talk to you all soon.