The End of the Modern Data Stack (w/ Benn Stancil, Mode)

Welcome to the Analytics Engineering Podcast, featuring conversations with practitioners inventing the future of Analytics Engineering. Hello, welcome to season five of the Analytics Engineering Podcast. Taking a bit of a break from last season, but I'm excited and even inspired to dive back in. You probably note that this time around, it's just me. Julia sadly signed off as my co host last season, and while I'm bummed to lose her as a partner, I promise to do my best to forge ahead. In episode one, I wanted to spend some time talking about my most recent post on the Analytics Engineering Roundup. I titled it Is the Modern Data Stack Still a Useful Idea? The post has created some waves around the ecosystem. Lots of the feedback I've gotten has been really positive. As long as you don't go to Hacker News. On this episode, I wanted to talk to someone whom I could count on to disagree with me at least a little. Ben Stancil probably needs very little introduction. As the co founder of Mode, one of the OGs of the modern data stack movement, Ben has been watching this ecosystem evolve for the past decade. Over the course of an hour, we get into the history of the modern data stack, how the term slowly lost its original meaning, how it came to mean something else, and where Ben thinks we collectively fell down as an ecosystem. Honestly, the part of the conversation I found most fascinating was when Ben said that we should all aspire for data to become more like it. Not exactly why I got into the field. If your knee jerk reaction is to disagree with that perspective, I'm right there with you. But here, I'm out. He has some fascinating parallels that he draws. All right, without further ado, let's get into it. Ben, welcome to the Analytics Engineering Podcast. It's been a little while.

Good to be here. Thanks for having me back.

Was it season one? I mean, it's been like, I think three years or something.

What season are you all on at this point?

We're on season five. I mean. Oh, wow.

Okay. Yeah, I think it was one or two.

There's a couple of Netflix shows that I feel like they put out multiple seasons a year, which seems to break the laws of physics in the universe. Like working moms. I don't know if you ever watched that, but it came out when my wife and I had our first kids and then they released like seven seasons in like three years. You're like, that's not. That's not how that works. That's.

So I had a roommate who used to watch the Bachelor and the amount of, like, content that the Bachelor produces, it's like a two hour episode every week. There are multiple seasons a year. There are these like side episodes throughout the week of like other talk show. It was just an astronomical ecosystem.

Yeah. For folks who don't know you, which, you know, maybe few and far between, but give a quick intro to yourself and then I want to ask you about the, the big news that's happened since we last spoke publicly, but the acquisition that you've been going through over the last year. Sure.

So I'm Ben, I am one of the founders of a company called Mode. Mode was a sort of BI tool focused on data teams, as will become relevant, I suppose for this conversation, an early modern data stack vendor. So we started it back in 2013 based on some basically like internal tooling and stuff that we had built for ourselves to enable data teams to basically write SQL, share results, those sorts of things really quickly. So over the years it became kind of a fully fledged BI tool. Um, and then the last six months Mode was acquired by Thoughtspot, which is a, another BI tool, a larger BI tool that is focused on kind of AI powered bi. And so now the last six months have been largely, you know, trying to figure out ways to make these two companies work together and bring the products together in a way that makes sense.

For, for both sets of customers and you personally. Your title is now Grumpy guy who writes a newsletter, like effectively.

I mean at Mode I was, for a long time I was the Chief analytics officer, which is like some vague made up title that you can create when you start a company. And you're like, this is, you know, vague enough that you could do whatever. At one point, towards the end of it, I became the cto, but not like a technical CTO writing code like the sort of CTO that's sort of a sales engineer of sorts that shows up with customers and he's like, here's what we're going and here's our direction and vision, all that kind of stuff. And then inside of Thoughtspot I am officially the field cto, which is what you want to make it like. It is essentially the same thing as what I was doing with Mode, minus sort of the founder responsibilities I would say that come along with, you have to do when you're actually kind of the person. So still a lot of like customer facing stuff. Being a grumpy guy on the Internet generally trying to, trying to help translate what people say in the market into product roadmap and ideas and things like that, or trying to translate product roadmap and ideas into things that are useful and interesting to say to them.

You frequently are kind of translating the data ecosystem to itself with your newsletter. Do the folks at ThoughtSpot use you to translate the ecosystem externally to how to talk to customers or how to build product or any of that? Are you the ecosystem whisperer?

Sort of. Though I would say so. So ThoughtSpot's background historically was selling to enterprises. They have been very successful selling to major banks and insurance companies and healthcare companies and things like that. And in some sense, despite being shaped very much like a modern data stack vendor today, they've historically been someone on the periphery of that ecosystem. Part of that is because the customers they sold to, part of that is because they were not a cloud first product in their early years. It was an appliance that you just install.

In the post that I wrote, I talked about Tableau, but the thoughts about it was in the back of my mind too when I was talking about like all pre cloud data companies had to go through this harrowing journey of becoming cloud data companies. And I know that that's been a challenging journey and I think that you folks have come well out on the other side of it, but everybody has some scars from that process.

Yeah, yeah. And we as mo like we avoided that. Basically they were on the kind of final stages of that when the acquisition closed. So, so when we started talking to them, part of what they were interested in is like that has been their last couple of years was largely migrating everything to the cloud and becoming a cloud first company. They essentially had completed that journey basically around the time we started talking to them and they were starting to think about, you know, okay, what's next? And, and Melody was part of that story. So yeah, so that's that. So in some sense, I think like there is the, the sort of ecosystem whisperer, it goes in both directions after this acquisition. This was one of the reasons it was interesting to us was for them, they are an enterprise company that very well, like speaks that language very well. They know what those customers want, they know what people like Wells Fargo want. But when they're selling to digital natives or whatever you want to call them, that's more of a new thing for them. And so historically Mode has always lived in that ecosystem. And so there's a lot of, okay, what is it that that makes these people tick? What is it that we can sell to here? Whereas for us it was the opposite. And I think that's true for a lot of, a lot of again, kind of the classic modern Data stack folks where it was primarily A, S and B, Silicon Valley kind of mid market focused enterprise and, and so there's a lot of like figuring out of how to sell to you know, the. Well far because of the world it's a different notion. So there's some like whispering back and forth there I would say between both sides.

I want to get into some stuff around the modern day stack where we're at, whether that's a useful term. You and I both. I think we're doing the like Alexander Hamilton, Thomas Jefferson thing where we're like arguing that we just need Greek pseudonyms but. Last question before we do that. When you describe ThoughtSpot, you said AI powered BI, right? I think that's the phrase you use. Yeah, there's a lot of AI powered blank right now and there's varying degrees of reality behind that phrase. In many ways ThoughtSpot has more credibility than most companies to say that because that was kind of the idea from the beginning. I would just be interested in. You can only say what you can say publicly, but I'd be interested in what you have seen. And as a professional curmudgeon, are you excited about AI plus bi?

This is one of the reasons again that we were attracted to this deal was, was to some extent that that like this was post. We started talking to them post chat GPT post like AI being a huge thing and sort of realizing that, that this could be this giant wave of stuff that sort of resets a lot of, a lot of how we do things. And my view then and view now was that if there is a company that is positioned to take advantage of that, that already exists. Thought Spot is in as good a spot as anybody. And so the reason I think for that and the way I see it is like AI as it exists today doesn't actually solve that many analytical problems for people. I think you've talked about this before where we've had this thing now for this thing being, you know, chatgpt effectively but like all of the kind of the things that have been inspired by that for a year and a half ish now and like there's not that much that we're doing. Like it's, it's kind of a toy in this space.

I felt a little bad. I wrote that in a, in a newsletter and I was just like oh my God, this is like the most regressive thing that you could possibly say in a tech newsletter. Like we don't use AI that much.

But I think like it's, it's and honestly, you could say that with a lot of functions. I actually don't think that there aren't that many big businesses built on this. That it's like there are a handful of things that were buzzy for a while. Things like Jasper got big quickly. I don't know if that's sustainable, but it, it isn't. It hasn't yet like really materialized into something, something huge for any real function at all. I don't think data is one of those two. It hasn't been affected that much by either. But part of the reason for that I think and my, my view on this is basically we all got enamored with like chat, GBT and chatbots and things like that. But in reality, to make these things work, to make like AI actually something that, that really makes a difference in how we do our jobs, there's a whole lot of work that has to be done underneath it. That in the BI case, you know, I know y' all folks on Yalls team did kind of like BI plus the semantic or AI plus a semantic model versus AI without it, and found that, okay, the semantic model makes a huge difference. That sort of thing, I think is what you have to do to make this work. Where it's, it's not just like take a chatbot and plug it into a tool and magically you have some solution.

It's.

You have to build an enormous amount of sort of like infrastructure around that to make it work. That the easy examples are like, if I ask, you know, how many shirts did I sell in California last week? What in the world does that question mean? There's a bunch of things you have to sort of teach it so it can answer that. But also like if you're asking that question, you want a visualization, you got to build a bunch of visualization technology to make that work. You got to build a bunch of ways to distribute that. Like that. That's not, that's a small piece of the problem. And so ThoughtSpot has been building for a long time all of those things around it where they built the infrastructure. So when you ask a question in natural language, how can you possibly answer it? They built like visualization tools, modes, built visualization tools. There's a lot of the things I think that make an AI powered BI tool possible that you have to build with or without ChatGPT or whatever.

Because like probably you want to start by asking an English language question and then you want to get dumped out probably into a more traditional BI type experience where you probably don't want to edit your chart by speaking English. Like, please make this series purple. You probably want to like have it be landed in some kind of experience where you can iterate on it more normally.

You want this sort of, My view is you want this sort of mix of like, occasionally it's nice to have, let me just tell this thing what to do. But you also like want the interfaces that make it easy to interact with. Like, it doesn't make sense. It wouldn't make sense for, for DBT to build an analytics engineering interface that is like a chatbot instead of writing SQL for DBT models. It's like that, that's a disastrous way to create models. Is a chatbot like, no, I want to write the code.

I think it's like a good way to start. And then, yeah, you have to then say, okay, like 20% of this is actually garbage and I need to fix it.

Yeah. And so you end up having to build the kind of core tool anyway and then figure out ways to integrate AI into it in some like, sensible pattern. And so thoughtsbot has been thinking about that the AI that they were using to do it was not nearly as capable as what it has become, but in a lot of ways that's like, okay, that's an easier thing to plug in. We're great. Instead of having a thing that tries to translate this human question using sort of the state of the art models from three years ago. Now it can use GPT4, so that just makes it better. But all the underlying infrastructure is stuff that they have been thinking about and trying to refine for a long time. And so I think like that, that is my general, my optimism about it is that yes, if it, if it reinvents bi, it's not going to turn BI into a chatbot. It's going to just be like BI augmented. And that requires a lot of the pieces that they've been thinking about for a while.

There was an interesting post out yesterday by this VC called NFX that it was about competitive differentiation in AI and essentially that like AI is not itself a competitive differentiation because AI is like water. It's like computation. You just, the fact that you can compute things is, is not differentiating and neither is plugging into the OpenAI API. And they said one of the most interesting questions that they ask when they are evaluating AI powered companies is is there a business here? If you take the AI away? And I think that's so interesting. And it plugs in nicely to what you're, you're saying, which is like you have to build the whole thing and then AI can be an accelerator on top of part of the experience. But if the entire thing is AI that might not actually be a business that has competitive advantage baked in.

Yeah, the, the way that I've thought about this and I, I say this Having just pitched ThoughtBot as like a AI powered BI tool is AI will be successful when we stop talking about it. When it's no longer this is a CRM with AI it's like it's just a CRM that works that you don't even need to think that the AI is there that nobody pitches probably to some of the modern day side conversation. Nobody's like it's a CRM. But in the cloud you're like so what? Like great, they're all in the cloud. I don't care like what does it do for me day to day that makes things better. And I think that AI now we're selling the same thing plus AI you're kind of missing like the actual tangible value there. And when you start selling just the, the sort of magic that it provides without it sort of A.I. sure, whatever. I wouldn't care how it happened, it just happens. I think that's when, when it starts to valuable one.

One of my favorite features that we, we have I saw a demo of recently internally. I don't know how soon this is going to ship but it at some point it will make its way into the wild. Drew, my co founder hacked together an LLM powered formula builder and from some perspectives I mean it seems like a very simple use case but I mean how many times have you been in a SQL terminal and you were just like shit, I have to go like look up something on Google, click on a stack overflow post to like remember the regex formula on like how to parse some string and in fact it works brilliantly. You just say like I want to get the domain name from an email address and it just like gives you the formula. That's the kind of thing that doesn't mean that DBT cloud is power. I mean yeah, like we will incorporate AI into the experience but it doesn't fundamentally change the job to be done of the product that we are selling.

Yeah and this is, this is the version of so so Mode released kind of its AI powered features I guess in the last, over the last few months. And, and the way we're approaching it is it kind of was inspired basically by design where mode to tool you write SQL queries rather than be like here Let me tell you in English, the SQL query to write. We want you to be able to basically give us an outline, tell us like, all right, I want to do this, do this. You can write sort of pseudocode, you can write SQL and say, okay, fill this in here, join these things like express the thing that you want to create and kind of the shorthand that you want to create it in. And then we'll try to fill in the ugly details so you can be like, ah, this is a function that, yeah, parses the domain out of the email address. Like, just put that in the thing and we'll figure out what to do with it. And so it's not about just like chat with your data. It's still, it's. It's sketch, sketch using whatever thing makes sense to you. And then we'll try to like make it like, do the tedious stuff that you don't have to do. And I think that sort of thing to me is it's those sorts of experiences that I think ultimately are the ones that, you know, whether or not boat is that or not, we'll find out. But it's that kind of thing where it just sort of integrates more naturally into what you're doing. Instead of saying like, replace interface X with new Chatbot, it's like, yeah, I don't actually want that. I kind of want to do what I was doing, but like, without the tedium. Yeah.

Okay, let's switch gears. Let's talk about whether or not we should continue saying the modern data stack. Is that a useful idea? I wrote a post on this last weekend and it's one of the more widely read posts that I've written in a little while. And I think some, sometimes folks are surprised that I would write this because the term certainly has in the past had real marketing value for us as a company. So like, why, why do you try to kill something that's working? But the thing that got me to write this in the first place is that, you know, I have this memory of, you know, you, you've been doing this for longer than I have, that in 2015, 2016, there were this set of tools that were really natively designed for cloud data environment that took as a given the elasticity of, of storage and compute and the power of SQL in a way that got you to redefine the products and then redefine the workflow as associated with that. And that was exciting having having like worked at Data for whatever ten plus years before that. I was like, oh my God, this job that had gotten kind of frustrating and stale actually felt really, really exciting again because I could not do all the dumb stuff that I had been forced to do previously. Anyway, the term was descriptive at that point in time. Like there was actually a difference between a quote unquote modern data stack company and a note our data stack company. And, and I feel like at some point along the way it lost that descriptive power and instead became something else. And I, I tried to like describe that, that process in my post, but you were kind enough to send me a pre read of the post that you're going to be sharing and, and I think you did a better job of summarizing my argument than I could have. So, so please tell me what I think on this topic.

I have never used chat GPT to write one of these posts and I like plugged yours in and was like summarize this and it was just garbage. It was not a good summary. That is not what the general arc of your narrative to me is, right where there were a handful of companies early that were doing things differently. I think your points are mostly right too that it was primarily cloud and like this kind of focus on SQL when SQL wasn't popular yet, it was sort of a, yeah, everybody, I think.

People forget that people who came to this in the last couple of years, like no professional wanted to put SQL on their resume in 2015. They were like I must use Spark or Python or R or something.

Yeah, I mean we, so we started moat in 2013 when we were out having our first like fundraising conversations with VCs which was have been, you know, late 2013, early 2014, SQL was like what in the world are you doing? You get a lot of skeptical looks about that now. That's kind of the standard. So like I think there was, it was mostly that and it was mostly cloud. And to some extent I think, and you've written this before, like hey, these cloud databases, cloud like redshift essentially at the time, but, but like cloud data warehouses are a thing that suddenly enable a bunch of stuff. And there are other things I think like, like the, the ELT kind of philosophy became a big part of the early stages and stuff like that. But yeah, to me generally there was a period where the modern data stack and honestly probably before the term really emerged, there was a group of companies that all sort of saw things in the same way and then we didn't really have a way to describe it, but it was like these are, this is what this sort of is, is this Collection of similar tools, similar philosophies. I think there was some like community to it. Not in like the traditional sense, but in the sense that most people kind of knew each other. They all kind of came from similar lineages, honestly.

Yeah, but you kind of had to, because I'd be curious how close was your co selling relationship with like a Stitch or a Fivetran or. Or a Snowflake or any of these companies back in like 2016, 17, like to a certain extent, like you couldn't use Mode or any of the other BI tools that. Well, unless you had already implemented some of these other products.

We weren't co selling that much with them, partly because the companies were too small to really like run reasonable co selling motions. But I think we. There was a lot of sort of soft co selling where we leaned a lot on five Trans Stitch companies like that, y' all and Fishtail, honestly, Fishtown more than DBT at that point. We leaned a lot on y' all as like validation of the general principles that we were trying to sell. It's like, look, there's these other companies that are doing this too. We all kind of have these similar beliefs.

Like it's like social proof.

Yeah. This isn't just us coming up with some crazy idea. It's like there's, there's something real here. And so I think there was a period where that was what it was. And, and honestly, in some ways, like the branding of it as the modern Data stack was the point at which in your post you, you know, I think you, you're right, you referred to it. Basically it became like a meme. And I think that's essentially what it. Like it became popular, it got buzzy. A bunch of people wanted to put money into it, a bunch of people wanted to start companies around it. And like that, I think took the core pieces of what it was originally, which was a hand like vtl, it's databases, it's bi, it's transformation. It's kind of these like big tent pole things. And people started looking for all sorts of interest, like all sorts of ways to basically plug into narrow and narrower swim lanes where, you know, there's like observability and like data dictionaries and reverse ETL and like a half dozen other categories, I mean more than half dozen, like dozens really of other categories of kind of like niche problems that if.

You go to modern Data stack that XYZ, I think it's like 29 categories, something like that. Yeah.

And like, I don't maybe these Things are big. I don't know, it's, it's still time for these things to be big. Maybe some of these things become, become super important. I don't know. But like they were chasing an unproven problem in my view. They weren't chasing. I have some confidence in whether or not Mode or Thought spot or whatever succeeds. It will be decided by whether or not like we are able to build a successful. We aren't also just like trying to prove that the market for BI exists.

Right. The market for BI has been there before us and it will be there after us. And you don't need to bet on that category.

Yeah, you may not win it. It's competitive. There's a lot of other things in there, but like, yeah, it's there. Whereas I think for a lot of these other companies, they're sort of trying to prove that this is actually a thing that we need at all with varying degrees of success. I think, you know, some, some will, some won't. I don't know, we'll find out. So yes, I do think like the term, it stopped encompassing anything. I think it still encompassed something meaningful. It was just like it was a phase. The way I've described this before is to me the modern data stack was like a term for data companies that launched on product hunt.

It's kind of a joke, but like.

It, it to me it like fits the era of when people cared about product hunt. It fits the sort of philosophy of like these were very sort of bottoms up kind of consumerization of IT type of companies. And that's basically, I think it's a useful term to describe that era Now I don't think it's a useful term to describe any sort of coherent philosophy. And so saying a modern day stack company like I, I can, I can picture my head what that means. It isn't what it would have meant the collection of tools in 2013 or 2014, but it is like a useful descriptor. It's not necessarily a good descriptor in the sense like, I don't know if you want to be labeled that anymore, but I think it is a useful descriptor in that sense.

You know, sometimes you write to convince yourself of a point and the thing that really convinced me of my point as I was writing this was if you buy the kind of technical definition that I had proposed which is post cloud and SQL oriented and then you should be able to go from not modern data stack to modern data stack and if you kind of like, you know, change the underlying technology and you are now, you know, kind of based on those principles. And I think that, you know, I use the example of Tableau. Like, tableau was not very aligned with like either of these principles. Yeah. Previously. And now I think that you can make a really good case that, that they fit the technical description. But if you say, like, is Tableau a modern data stack company? The answer is like, that doesn't really compute. Like, they weren't really a part of that era, that market trend. And, and so if it's no longer being used as the technical descriptor, but it's used as like the description of a vibe or a Did you launch on product? Then it's just like, okay, I don't really care that much about this term.

Yeah, I mean, like, to me, the more interesting one is Power bi.

Okay, sure. Yeah.

Because. Because Power BI launched post all this stuff. It's actually pretty new. It's all, it's newer than Mode, for instance. But I don't think most people would say that it's a modern data. It's like just because it came from Redmond.

I don't know. Like. Yeah, yeah.

There's not really a clear answer for that.

I don't think anybody. The PM at Microsoft wasn't, wasn't like putting up their product hunt post.

Yeah. Microsoft is well past caring about product hunt. Can you imagine? But I think it is a useful descriptor. I don't think it's useful descriptor for, for if you're a CIO evaluating something like, is this modern day? Like, no, I think it's a useful historical term.

Sure.

In the same way, like, big data is a useful, like, I think that now kind of is like, it's an era. We kind of know what that means.

Gosh.

But like, no, CIO or whatever is like, hey, we need to go buy some big data tools. You're just like, what in the world are you talking about? Like, okay, we kind of know what that means, but like, it's not a thing in that sense. And so I think it's kind of the same thing.

What's the one company that you most associate with the big data trend?

Cloudera.

Yep. Okay, I'm on cloudera.com right now. Do you think that cloudera.com says big data at all?

I mean, I kind of would think Cloudera would say like 404, but the.

Answer is it does not. There's no mention of the term big data on cloudera.com.

I would be more interested if Cloudera says modern data stack.

Wait, wait, I can. Oh, Jesus, Ben. It does say modernize your data stack, which is not exactly the same thing, but it is pretty. It's pretty close. I love that. And I, I never thought about that before Big Data. There was a reason that we used that term in whatever it was, 2013 or 14 or something. Like all of a sudden we had. Well, I mean, companies like Google and Facebook had had this for a while, but enterprises were. Were starting to think about processing kind of clickstream and behavioral stuff. And they were like, oh, shit, this is like a lot of data. And they needed a way to talk about, like, what do we do with all this data? But then it became a whole thing, and then it was not useful anymore. And so there's these, like, trends of the ways that we talk about things that feel very antiquated in respect and in retrospect.

And it became, I think it, it even more so than the Modern data stack became a meme. I mean, it became a meme that was like Main street where it was.

Like HBS did articles about big data. Yeah.

To me, the kind of the. The peak of this was that New York Times, I think, was New York Magazine, but like, that Target article. Do you remember this article?

This is the one about pregnancy. Predicting pregnancy.

Yeah, exactly. Exactly. Where it was like, oh, my God, Target knows that your daughter's pregnant before you do. And that was like, big data. And I was like, I don't know, like that's. That's all it meant was some sort of crazy Minority Report thing is going to be like, understanding what happens in the world before you do? Like, okay, I mean, I don't know. It's not a useful term now. It's not really. It wasn't really descriptive of anything at that point, but it was. I don't know, it was a historical marker.

Yeah. Okay, so I think that both of us are kind of aligned on this, which makes for a boring podcast. Let's talk about the stuff that I think that we have some level of. Of disagreement about. I'll say that, as is generally my approach, I'm an optimist and I can't help but seeing the glass half full. And so certainly I agree with you that there's been a lot of kind of marketing use of this term. There's been a lot of company creation and, and time spent and energy, whatever, and a lot of this will eventually come to nothing as happens when these kinds of things happen. But I tend to see that as a natural part of the process and that there are good things that come out of that a little bit. Like how there was a ton of fiber laid as a part of the dot com boom. And like yeah, was it pets.com or webvan or I don't even remember some of the like most famous examples. Like they didn't, many, many of them, most of them didn't make it as companies but in the process they created something of value, something of lasting value. I see this through the, the lens of Carlotta Perez's cycles of technical disruption framework and I see that the bubble phase as a natural part of that. The good things happen out of it. But I think you were focused on why don't you tell me why I should be less optimistic about the phase that we just went through so well.

So what I did.com boom is probably another big data slash modern data stack era thing where like.com not super descriptive of anything specific but like we kind of know what that means. So my question on this I don't have a. I'm not confident in my answer. My question here is like yes, I agree with you that, that we've done a lot of things, we've explored a lot of stuff, figured out some stuff that would be really valuable and make people's lives easier and things like that. And yes, there's a lot of stuff in your view of this like this kind of analytics workflows, stuff like that that is better. No doubt. I think my question is the broader ambition of what the modern data stack aim to enable of like these sort of data driven companies kind of for lack of a better term and, and our ability to all make way smarter decisions and stuff like that positions like what data does and, and what the modern data stack and the tools that would be a part of that do as as this like transformative and strategic effort that is a necessary thing for every company to have. And I think that there's a real question in my mind is whether or not that's true. That yes, I think that it will make the people who do analytics and reporting and those sorts of things better. It will make some companies who do like real data work who are building you know, prediction models for what I should watch next on Netflix better but, but the way I'd frame it is like suppose that we went through some crazy and this may have happened and we just aren't aware of it. Some crazy like boom and HR software. There's the modern HR stack where they all talk about like how HR is going to be the strategic asset that you have to have at a business. And there's a ton of companies that get built around, like, making HR central to what you're doing and it's a strategic thing and all that kind of stuff. At the end of the day, it may be like HR is kind of just like a mechanical function that companies need, but it's not actually strategically that differentiating. And I think, like, my question really is like, yes, I think we can certainly make the somewhat mechanical workmanlike stuff that we do better, but the modern data stack was never quite. I mean, in the early days, I think it was about that. In the later stages, it became this much bigger ambition. And I don't know if that ambition can actually be realized because I don't know if it's actually like, that valuable of a thing to even do. And so, yes, I think, like, there will be places where it becomes super useful for the stuff that we know we have to do. But, like, I just don't know that I. In 10 years, will we be like data teams? Yes, clearly. Strategic assets, or will it just be. It's. It. It's hr. It's, it's a, it's a support function that you have to have. It's legal, but, like, nobody's investing huge amounts of money in the technology for their legal teams unless you are a very, very specialized company that needs something very specific in that field.

Gosh, I wish we had started this conversation 20 minutes ago. We should have killed all that time of a thought spot. Forget all that. Let's. Now it's because you just opened about 12 cans of worms.

Do you have meetings after this? I can, I can go a bit long if you want to. I don't know if you can, but you have a busier job than I do now.

This is, I think, sometimes where I have a myo. Myopia about this because I always come at this from the perspective of the practitioner. And I want to get you to talk about shelling points a little bit. And, you know, what is it that we're all trying to achieve here? And is there a cohesive vision? And I have a pretty clear idea of, like, what good looks like from the perspective of, like, what should the data practitioner's process look like and. And who is a data practitioner and how that will make all of those data practitioners just, like, unbelievably more productive? And. But then, okay, great, we can talk about that. But you went a step further. You said, like, tell me how this differentiates the business. And that's fair because sure, any technology ecosystem only exists as far as it is providing value to the customers that it sells to. And maybe there's two different versions of that. Maybe one of them is this thing will provide you a dramatic competitive advantage that will help you win in the marketplace. And maybe the other one is this is a thing that you have to do, but everyone's doing it. So it's not really like your legal team is probably not going to be what makes your company compete and win. I mean, okay, maybe Oracle's did, but like in most cases that was not. And that's not how that plays out. So, so maybe the question is like, is data in category A or category B?

Yeah, I think the, the modern data, the first version of the modern data stack, the first, the phase where it was replace old stuff with SQL cloud, that kind of thing. Yeah, I think was focused on the. Let's just make the practitioner's jobs way better. Great. Let's make the stuff that you were describing in the beginning of like, this is painful and I don't like it, I'm not excited about it, into something that's like fast and fun and creative and like you can, you can work in like the flow states. I think that you've written about the later phase, to me, it became bigger, it became like. And partly because to prop up the amount of money and the amount of excitement around it, it became like, no, this is like a hugely important thing for the business. This isn't just for the practitioners, this is for the company. This is for like this, this value that you could unlock and the strategic asset that would be contained within it. I don't think we've proven that that exists. And so I think like the modern data stack in its later stages as this like big thing, this movement required the belief that this was for something bigger than just making practitioners lives easier. And so I think like we didn't prove it. We didn't prove it in time because the market turned and now like that money's not there. And so like, maybe we eventually do, I don't know. But like in some ways I think the ecosystem had its moment and it, it didn't deliver it. Now it may come back. And I, I think there's like when it's less noisy and there's more just like people kind of putting their heads down and trying to do stuff. I think that may be the time when it really emerges, but which is.

Usually when it does emerge. Yeah, yeah, it is that phase of the cycle and it may still.

But I think, I think there's, there's questions to me as to. Yeah, basically your point of like something to improve things for the practitioners. Absolutely. I think that has existed something that has these like much more profound impacts on the business. I don't know. And the former one can be supported by a handful of companies that make a lot of money and there's a lot of things in like, I'm sure there's lots of legal tools that make a lot of money and that's valuable. It can't support some massive dozens of billion dollar businesses ecosystem that requires the latter. And I don't know if we have the latter.

Yeah. Even if there's a there there, that doesn't necessarily mean that there are dozens of billions of dollars of companies that, that might not be the right market structure. All of this is a good thought exercise. And, and I think that you're, you're very carefully not saying you don't believe in that future. You're saying we haven't proven it yet. And that's, that's true. Certainly we haven't proven it yet. I think it's, it's my job in this context to make sure to say that I do. Like we, we continue to be one of the more optimistic companies out there. Like I, I think that. Okay, so here's a question for you. How many people in your standard 10,000 person enterprise, mid sized enterprise do you believe are data practitioners in some way? What percentage of the workforce data practitioners.

In like a way that, that people would identify such. Not like these people use data like professional data practitioners that could show up at DBT's conference and would be like.

Probably today would not be people that would like show up at coalesce but using data is an inextricable part of their day to day jobs.

I have a 10,000 person company, I would say that 75% are involved in somewhat like use Excel. Yeah, I would say that in terms of like data practitioners that they would probably self identify as like that. I would guess 500.

Yeah. Okay. I'm basically on the same page with you. My belief about the future of the space comes from, I really believe that we have a playbook for transforming the lives of those first 500 that you're talking about. I think we know, we know a big chunk of what their workflow needs to look like and it, it involves cloud data platforms and as much as possible automated data movement and it involves using mature workflows to govern how the data gets transformed and there's probably some whatever tooling on top of that to make sure everything's working right and whatever. And so. So I think that we have kind of a map for what that starts to look like, and that on some level, you're right, those are jobs that need to get done, but those probably are not the humans that totally transform the business. My belief is that those other, whatever, 7,000 people that use data all the time, the optimistic scenario is that their jobs change. And I don't think that their jobs mostly have changed yet. I think that they still exist in a very similar world that they existed in, I don't know, at the beginning of my career, 20, 22 years ago. And that's the opportunity, and I feel like it inevitably will. I think you have to be a pretty big pessimist to say, like, well, we're never going to make any progress on people that are just emailing each other Excel spreadsheets.

I never thought of it this way, and I can't decide if this is a thing that is a good thing or a bad thing. I think it's going to sound like a bad thing, but it's actually a very good thing. And I may be sort of coming to your side on this a little bit, though, in a way that I would not have expected. So we have.

We.

Like, one of the things that is. That has been sort of a narrative here for a long time, and y' all have been. Have been. One of the key proponents of this is like, data teams is engineering teams. Right? Let's follow some, like, engineering best practices. And I think in some ways, I think that is. That is the way that people get to this bigger step is like, look, engineering teams are clearly very important. They clearly can have this huge impact. If data teams can have that kind of impact, like, oh, my God, what could we do?

Be.

I think that's. That's fair. I'm not sure that we've delivered, but I think that's like a reasonable. That's a way for us to really sort of pull a very big lever. As you're talking about, like, how many practitioners, you know, how many data practitioners are there? This, like 7500 versus 500. The actual thing that seems like, is it. Where. How transformative is it. How many people are, like, IT users? Like, literally everybody. Yeah, but how many people are IT professionals? I don't know. Not that many.

Right.

And like, it is an enormous market now, obviously, it's like a very vague one. It doesn't really know what it means, but, like, there Is some kind of. Is it a strategic asset? Like actually probably yes.

Yeah, that's kind of a boring one to talk about now. 20 years ago it was fun to talk about, but.

But even now it still kind of is.

Yeah.

100 Covid teams that had like, or companies that had like good IT teams that could respond to Covid very quickly probably were doing way better than ones that really struggled with what to do here.

And I think that companies with great IT departments are going to take advantage of AI way faster.

Yeah, there.

And there's.

There's tons of things there that like actually a really good it and like if you don't have. You have lousy it. Everybody's frustrated all the time. It's like a thing probably people quit jobs over is like this thing is a pain. But there's something very boring about saying like data should be like it. But I think like that's actually maybe the, the closer version of where this goes. Where it's like the effect of what data teams do isn't in this so clearly tangible. Like, oh my God, I go to my thing and I automatically ask a bot and it tells me all my answers. And like I'm just. I am just like Tom Cruise and Minority Report answering all these questions. It just like creates this kind of underlying.

It's like the fabric of how your job gets done.

Greece is sort of organizational wheels. Yeah, it's a very, very widespread, very, very pervasive and very, very powerful thing. But not in like a sexy way. That actually is a way to. The modern data stack has not been attempting to build it. The modern data stack has been attempting. And you know, maybe to your point, like now when it's less noisy, this is the phase in which it gets deployed. I think as you said in the other your post, but like that may be what deployment looks like is not. Oh, we got to build all these other modern data stack tools that are sort of very in your face. It's more of like we're starting to just deploy these things underneath the. Underneath the surface in a way that a lot of people around the business become data users without necessarily even recognizing it as such.

Yeah. Okay, great. I had this really weird topic for a COLAS talk back in 2020 and I talked about the different architectures of central nervous systems, or just nervous systems sometimes, not always the central nervous system and how different animals, as a result of the architecture of their nervous system, process information. And to me that's what the exciting future looks like is the information architecture of a large Organization is still kind of archaic because we haven't built all of the connective tissue so that it can actually flow through systems that understand what's happening. When you email a spreadsheet to somebody else, yes, that is all captured in technology, but there's no way to reason about like what is actually going on there. And so to me the big opportunity is to connect all these dots and, and you use the word sexy and it probably isn't that sexy. It's creating a new architecture for how information flows from person to person or maybe from person to bot throughout a company that has, that's able to deliver the data and the context and the trust signals and all of the stuff that historically has prevented us from kind of doing the important stuff, which is making decisions with it, acting on it with like alacrity and confidence.

Yeah, I think I would agree with that. And there is something to me in there that is a very different. And maybe this is, this is you know, sort of the modern day sec, but a very different vibe and approach to that than how this has historically been thought of where I think like we don't have like a clear experience of what this thing is supposed to look like. I think it's very hard to describe how our lives are supposed to be made better by this. And I think like the story to me that would be interesting is if I think about this in terms of the IT analogy I can kind of start to tell myself like how has it made my day to day life better? And it's like well you can log on to this thing and you have one place to go to log into all of your apps and you're automatically synced to stuff and when somebody leaves the company like the whole thing gets changed really quickly and there's easy for communication for stuff like this.

And like I never thought I would love single sign on but man, I don't want to go back to. Actually the experience now of logging into multiple different things is, is really good.

There's just some like streamliness there that's not. There is probably to me a version of that. I don't know exactly what this is. There's probably a version of that for what data does that's like in the sort of data enabled business or whatever, you know, what does that actually look like? How are, how are the day to day lives of like these kind of 7,000 data practitioners that wouldn't label themselves as such. How are their lives like different? And I don't, I don't know that we've really articulated that story. That to me is not a way that I've thought about this around. Like what is the. The it extrapolation of this? Instead of the kind of hand wavy. We will have sort of magic question answering bots in our pocket that, you know, every curiosity will be solved with a chatbot kind of thing.

Okay, that's giving me a lot of food for thought and probably items for my upcoming writing queue. A lot more to think through there. We've kind of been talking about this, but typically we've always ended episodes with asking, what do you hope is true of the data industry in 10 years? But given the past year, the way the last year has gone, I feel like 10 years is like hopelessly long. I want to shorten this to like five years. Like what, what do you hope is true about the data industry in five years?

I think this gets it. Some of what we were talking about was like, it has a clear direction for where it's going that there is this experiential vision of like what this is going to create. Because I think that's a lot of the scattered bluster of the last few years have been because we've lost sight of that or we've never really had it again. I think we had it for how it's going to improve workflows for practitioners. We never quite got there for how it's going to sort of have the, the broader impact that we wanted it to. And I think if we can, if we can see that that's not going to come from like somebody dictating it from on high is going to come from like everybody sort of, oh, we realize where this future is headed and we can kind of like start to build in that direction. I think that's when like the real progress ultimately gets made on and I don't know, like, maybe we get there, maybe we don't. Maybe we never have that vision.

Maybe we do.

But I hope that that's. That's where now this is.

This is. You're supposed to be hopeful here, right?

This is. I hope that we do. And the optimus in me says we will.

Thanks, Ben. This has been a lot of fun. Looking forward to doing it again soon.

Awesome. Thanks for having me.

The Analytics Engineering podcast is sponsored by DBT Labs. I'm your host, Tristan Handy. Email us@podcastbtlabs.com with comments and guest suggestions. Our producers are Jeff Fox and Dan Poppe. If you enjoyed the show, drop us a review or share with a friend. Thanks for listening.