The Bundling vs Unbundling Debate w/ Tristan, Benn Stancil and David Jayatillake

Welcome to the Analytics Engineering Podcast featuring conversations with practitioners inventing the future of analytics engineering. Thank you for joining us on a special episode of the Analytics Engineering Podcast. This is a rapid response style format. There's a conversation that is happening all over data, Twitter and Data Substack that is is spiraling out of control. And we have a couple folks here. We're going to see if we can make any sense of it. That is a very open question. We'll see where we get. But I'm joined today by Ben from Mode, who is has become a jack of all trades there, but I think most famously is known to us through his Friday Night Fights substack. Thanks for joining us Ben.

Thanks for having me, Jason.

And David, who's been on the podcast before as well, who is a co founder at Evora.

Thanks for having me Tristan. Yeah, glad to be here.

So bundling and unbundling. There's been more posts on this topic than I think we can actually try to summarize here. But I think the place I want to start with, I want to find this tweet. Sarah Krasnick writes, how do you define the modern data stack? And if you're using one? And Ananth, who is the author of the Data Engineering Weekly newsletter, writes, modern data stack is a set of vendor tools that solve niche data problems, lineage, orchestration, quality, with the side effect of creating a disjointed data workflow that makes data folks lives more complicated. Shots fired. Maybe that's the best place to dive in. What do you see as kind of the core of this conversation? David, let's start with you.

Partly it's, it's, I think it's historical. I think people remember a time when those data stacks were more consolidated, whether that was like Teradata, Oracle, Microsoft, and they think, oh, that was better in a way because even though it can click with what we needed it to do, it was all in one place and had much fewer tools. But having said that, like it didn't work very well. It wasn't scalable. Now people, people could say, well what if you had a cloud version of that stack that did work well and that could do pretty much everything you wanted it to do. I'm not sure that having having it so tightly joins and having only one company looking after the whole stack, whether it would meet everyone's needs, it would probably meet many people's needs. But if you think about how things like web development and software engineering has gone, they have a very complicated stack of many different tools that join together to deliver the best solution for any given organization. And I could see us going that way as well.

Ben, what do you think? Frame the conversation from your perspective.

My view is that there was originally a, as David said, basically a breaking apart of a few big tools into kind of the polls that the major pieces got pulled out. And so, okay, so there was clearly like this kind of unbundling, I suppose, of like the monolithic bi or monolithic warehouses. The space got hot, basically, and it attracted a whole bunch of venture money.

And so I was going to say, I think the venture. You cannot separate the story from like the venture money flowing into it.

Yeah, like it blew up and now there's tons of companies looking for places to like, insert themselves. And so, like, I think I agree with the idea that, sure, broken apart may be better than one giant thing, but I think we've gone probably way too far, mostly because there's just a bunch of money chasing problems. And I think that's going to create it like a very difficult experience. I don't think that's a bad thing. I think Martin Casado has talked about this. I think with you, Tristan, about, like, this is how spaces evolve. This is what happens. Like this, this chaos is part of sort of the natural process of, of technology improving. But I think it's going to be like a real. For now, we have to sort of figure out how to deal with that mess. I don't think just because, like we have individual pieces of technology that are good, that means the overall experience is good. So I think there will be some way to figure that out. I think, like the economics of the market right now are not going to sustain 300 modern data stack companies that are all trying to charge 50k for their products. But I think how that actually shakes out is. Is a difficult thing. The way I would put, like the unbundling, bundling bit is it's a little bit of a semantic argument kind of for the sake of like coming up with something to say. In reality, it's just like we do data differently. We're kind of reconstituting the thing that what we had before is just being like rethought and pieces are going to be unbundled, pieces are going to be bundled. It's just sort of redrawing the lines. I think that's roughly where we are.

I think we should also consider the cloud aspect to this as well as the venture money. I think part of the problem is that the cloud vendors all joined in and, oh, let's just host postgres as a data warehouse and make it scale or build something new. Right. Some of them did do that as well, but they didn't build the rest of the stack. They just built those things. And so then other people have gone and built different parts of the rest of the stack.

So I do think that there's a certain amount of like, noisiness to this debate where it's something that people seem to want to fight about and so more people continue to join the fray. One of the things that I want to see if I can convince the two of you of, and maybe this will be a very boring podcast if we all just agree, but I don't think think that in software engineering land, in 2000s, when you were seeing the rise of web frameworks, I don't think that there were arguments about bundling and unbundling. I don't think that people were like, oh, the Rails ecosystem is trying to do too much, because maybe there's multiple reasons for that. I think that part of it is that the Rails ecosystem is this completely open set. It's, it's a, it's a framework and a series of functionalities that like, plug together and you can choose to use parts of it, you can choose not to use parts of it. But like, the beautiful thing about it is that if you want to, you can use them all together. So I frequently find myself in a lot of these debates thinking that it's not actually one or the other, it is, how do you get both? And the reason I think that we, the core part of like, why we find ourselves in this one or the other mindset is that Rails was not a commercial entity. It had no opinion. You like, use it however you want to use it. It's just there for you to use however you want. But all of the folks participating in this conversation tend to have a viewpoint and they want it to be either bundled or unbundled and for the ecosystem to find its way there. I don't know. I think about this from a composability standpoint and creating experiences for users as opposed to trying to win a technology war.

I think I agree with that and I think that that is one of the ways in which the like analogy with engineering doesn't work where I'm not an engineer. So I don't know, certainly don't know what was happening in like the Ruby chat rooms in, in 2010 and like rails was coming big. But we're trying to build products, we're not trying to build technology. Like yes, technology is part of that, but at the end of the day this is a big product. And it's more akin to me of like, why isn't there a CRM stat? There's not. There's just Salesforce built a better CRM than everybody else and attacked on a bunch of stuff and now everybody buys a Salesforce. This is a bigger thing than Salesforce. It's a bigger thing than the CRM. But I think it's closer to that experience than it is web development frameworks where it was just technology. It was like people were trying to figure out how to build all that sort of stuff.

So let me pose an answer to that and then David, I want to see if you feel like my answer is bs. I think that the analogy to Salesforce is instructive insofar as what we are trying to build here is. Well, at least I think most of this conversation is around data infrastructure and not data products or data applications. Whatever you want to call like the consumption layer. And I would not in any way suggest that. I think that the right answer in the user experience layer is 100% absolutely. That it's going to be like commercial products. Great. But I think that we've time and time again seen that infrastructure is best built by open source and open standards. And you tend to find a winner take all solution at a layer of the stack and everybody uses it and it's open and you just freaking build on top of it from that point forwards. So I think that we're kind of trying to build commercial solutions in infrastructure in a way that like historically infrastructure doesn't really get built. Okay, David, what do you think?

Yeah, I would agree with that to a level. Like you see things like Terraform out there, which are one of a kind, more or less tools for infrastructure that everyone uses. And there's a, there's like one popular like open source backend per stack usually like Django for Python.

Yep, yep.

And node for JavaScript. But there's. And I think it's really interesting that it links back to how like things are composable and like Ben, Ben's point on, you know, can everyone just pay like 50k for every little piece of the stack? And I think no, but you know, with the advent of lots of these things being open source, with the advent of lots of these things like adopting a product led growth model where they're not costing 50k, they're costing like 10k or less suddenly. And if they do compose well with you know, data roads of Apache Arrow. Right. Then could. Could it actually be workable much like how you can pick and choose from like the Rails ecosystem.

This is. I'm gonna. Hopefully you don't have any engineers that listen to this thing. I'm gonna completely butchered analogy here. So this made me think of actually Rails but in a different way. So Rails as I understand it is like a model view controller framework. So basically you fill it up into three different pieces. You have like the model which is kind of the data. It's the structure of you know, the kind of underlying. This is what is true. You have the View which is the front end presentation and then you have controller which basically translates between the two and it's like this is how you present how you manipulate this underlying data intuitively presented to.

To the View.

To me it actually feels like you could divide the stack in the same way and each one actually has sort of a different paradigm of what the product looks like. The View is a commercial product that you buy that you, you know, you put in your credit card, you log into whatever product and you go look at it and that's like. It's just commercial product. The model to me feels like the actual infrastructure. It's like warehousing, it's things like that. In those cases it's like sort of open source but you don't care because you're actually just paying for compute and storage like you're paying for infrastructure in a commoditized way. The controller is the interesting layer and that to me, I mean DBT very squarely sits in that. So like Tristan, you have much better protection on this than I do. That one to me feels like where the open source frameworks actually make the most sense. That while the kind of model part of it. Okay, yeah, you have some open like it's all postgres or something under the hood, but not really. The View is. I mean the controller feels like the piece where it's like. That's where the open source frameworks really make a huge difference because it's translating between. How do we basically make the data layer modeling part of the stack Compostable with. Or composable rather with the.

Whatever you heard it here. It's compostable.

Yeah. With the View layer like it's. It's that translation to me that feels like there's particular value in the, the open source side of this.

Yeah. Does your brain just think in metaphors or did you have that one prepared beforehand?

Strictly thinking analogies.

That was, that was good.

People in meetings do not appreciate it, Just say what you mean.

I like this, and I certainly agree that there's a lot to pull from that metaphor. I think that one of the things that we have to answer here is the question about homogeneity versus heterogeneity of actual computation that we want to perform. And so actually, my co founder, Drew, wrote an issue of the Roundup recently. I focused on HTAP databases, so hybrid transactional and analytical processing databases. And I think that in a similar way, we have to have an opinion here on whether there will be a single way to run, like orchestrate computation. So the DBT approach is very highly structured. All models look kind of the same. You can do a similar set of things with them. Right now, DBT does not support arbitrary Python execution, Scala execution, whatever. And so there's lots of things that are not suitable for DBT today. And I think that that's why the bigger DAG still has a tremendous amount of value, because you can't squish everything into DBT's model of the world. So you have this heterogeneous processing dag. So the question is, as databricks develops, they're more data warehouse SQL functionality, as Snowflake develops their more Pythonic ML type functionality. You know, BigQuery also rolled out an integration with serverless Spark, which, you know, I think is. It's got some work to do from a UI perspective. But as these things like start to combine forces, is it possible that the heterogeneity that data engineers have previously had to kind of solve via stitching everything together in airflow, is it possible, possible that we could, whether it's DBT or not, honestly, I'm not trying to convince everyone here that DBT should do everything, but is it possible to imagine a way that all these things kind of start to look more similar than different?

Yeah, I think so. I think this goes back to the point like what Ben was talking about, the model that in that model, view controller relationship. And I think the model for data is more complicated because inside that web backend you're just talking about relational database that's abstracted or NoSQL database. Whereas for what we have to do in data, we have to build these really complex layers of like, entities and metrics and semantics that then are usable. And I think that's where the model is deeper in data than it is in software engineering in many instances. But yeah, can more happen inside that, that model? Yes, I think so. And even if it's just kind of to the left and to the right of the dag, if you. You know, where things could happen before and things could happen afterwards, but it still fits within it, that's still just. That alone is hugely powerful compared to what we have today. And, you know, you've got tools like Foul entering that space, enabling that space, that are quite exciting.

So, okay, I have, like, three points on this, and one's a question. One's a question for you, Tristan. The first question is. Well, it's not really a question. I guess the first thing is databricks, to me, feels like the, like, biggest $40 billion mistake.

Oh, ouch. Wow.

Where? Not like, they did great. They're obviously great. They're all gonna be rich. But it feels like they try to market this thing as, like, this complicated piece of technology that nobody quite gets. And the way that I always think of databricks is it's a tool that someone smarter than me uses. And like, everybody I know seems to have that same thing where it's like, I don't know how to use databricks, but people who are smarter than me certainly do. They seem like it's powerful. And Snowflake basically seem to come along and say, well, we built data bricks, but it looks like a SQL database. Everybody can use that. The question that I have on that is, like, in this compute thing, is there anything stopping these companies from essentially just saying, hey, we store all of your data in one. Basically an S3. We put a bunch of different engines on top of snowflakes, and the initial engine is just like a SQL thing that looks like postgres. Great. We should have a Python engine. Like, rather than this kind of. Oh, it's a spark integration. It does all these sorts of things that's like this kind of confusing, monolithic enterprise piece of software. To me. What I really want is just like, all my data lives in one place. I can connect to it with different compute engines that speak different languages. And right now, that's kind of what, like, databricks does. It's kind of what Spark does, but in this way that feels very hard to get your head around. And so I think, like, we actually could just solve this by saying, hey, actually. And I suspect Snowflake will do this, where it's like, we should have a way to connect to it. With Python, it just looks like Python, but underneath it, all of your data is there. And a nice, accessible. The question I have for you, Tristan, and this is like, the wild idea, how far apart can you pull compute and storage? Like, can those be different companies. And is there anything that actually long term stops DBT for being a compute layer that sits on top of S3 and cuts out Snowflake out of this entire.

Oh gosh, you opened all the cans of worms all at the same time, which is something that you're famous for doing. Let me pick up the thread on the and and so the answer to like, can DBT do X? I think that our preference is to do as as little as possible when it comes to compute and storage. As little as possible. Limit zero and I spoke at the subsurface conference recently and was on a panel with Ryan Blue from Iceberg and Iceberg. And I'm. This is towards the edges of my deeply technical knowledge. But Iceberg is a table format. It's not a file format. Parquet is a file format. But Iceberg is a way to organize a series of parquet or other files in cloud storage and have a unified like metadata e way to figure out what file to go to when you run a certain query. And so that, that kind of feels like a big part of what a database does. But the interesting thing is it actually doesn't have a SQL or other endpoint to connect to to do processing. It doesn't have the compute layer, it's just the table layer. And so Dremio, the host of subsurface is, loves this because they are a compute layer that doesn't have a. They don't have a strong opinion about how you should be storing data. And you know, we've talked about this stuff as data lakes historically. But the interesting thing about the historical data lake paradigm is that it's just been a compute layer paired with a shit ton of parquet files. And okay, maybe you've like got some other. Maybe you've got like hive table format in there. Something there's. But I think that we're actually getting better at this table level, which I think is the right abstraction to pair the storage and compute with. And you've also got Snowflake is supporting Iceberg. I think others are supporting Iceberg. So you could start to imagine, Ben, you were just saying, like, I've got all my data in this one platform and I can access it via like multiple different engines. And I think that that is like a. That is certainly an approach. And my guess is that all the data platforms will want to move towards that world. But I think that there's this other approach of like, all my data is stored in Iceberg tables and I can have two contracts, I can have a snowflake contract and a databricks contract and actually each one of them is a little bit better at like the thing that they are best at. And they're both reading and writing the same set of files now. Okay, there's like some, some like. I kind of like rounded off some corners there and I, I realize that, but I think that there is a version of the world that looks kind of like that.

So you're saying this, we need to unbundle cables, which is really going down to a level of this that's like, oh my God, we're going to have like files in one place and like the Mac to the files in another. Is a, is a new level of unbundling of this whole stuff.

You weren't ready for that.

I am not prepared for that. That kind of makes sense. I think there's like some clever stuff in there that seems kind of cool, but like that's. I don't have the mental fortitude for that one.

So. This really reminds me of a conversation I had, it must be nearly a couple of years ago with the Firebolt guys when they were really, really early and they had this new F3 file format which underpins part of their platform. I was saying to them, are you going to make this format open? Are you going to allow people to edit their F3 files that sit underneath eyeball? Because that's something that Snowflake doesn't allow you to do, that people would like to be able to do. And that's, I think that further decoupling of storage and compute. Because currently, yes, this compute and storage is decoupled in a service like Snowflake, but you don't get to touch the storage. It's, it's theirs to manage. So. And you can't even access it without going through their compute. And that's, and I think that's, it's still quite tightly coupled in that, in that sense. And I, you see things out there. There's some really cool open source tech coming like DuckDB, which can just sit, which is that complete. It's just compute. You just apply it to some files and you can, you know, you can query those files and that, that's, I think an interesting vision of the future, admittedly at a small scale.

I think that the fact that the platforms all have their own file formats is a feature, not a bug. But it does make this vision that we're moving towards, not that we're moving towards, but like we're discussing right now, more complicated. I was just hanging out with Arjun, the CEO of Materialise, and he was talking about how you can't really separate the on disk representation of the data in a system from like the performance characteristics of the system itself. Like the file format is built to facilitate a particular mode of computation. And so I think that what you will actually have is there's still going to be many copies of many data sets floating around. But hopefully it's not data engineers who are like making all of these copies that, you know, hopefully in this version of the world, data engineers are ingestion tools and data engineers are responsible for sourcing the single clean set of data. And then whatever parts of that big data set that you want to load into each of these specific platforms to do particular types of processing, they will end up onboarding the data, that specific subset of data into their own file formats so that they can materialize. Does streaming. Like you can't do streaming in the way that they do on parquet files. Like, it just doesn't, doesn't work.

And I think there's something to that. That is, that is the reason to me that none of this, if this, if this doesn't work, that's the reason why.

Yep, yep.

But it's not, it's not just like the file format. Yeah, like the file format matters to the experience of actually the compute you put on top of it. And I think that that translates throughout the entire thing where at some point it's like, this may technically make more sense. It may be something that, yeah, you can underline, edit the underlying things. There's all these benefits. But most people don't want to build their own car. They just want to go to a dealership and buy a car and like, they'll take the worst radio in the thing because that way the car is like ready to drive off the lot the moment you buy it. And I think, like, there's going to be. If this is the direction we go, there's a ton of opportunity for somebody to come along and just reverse the whole thing and be like, here it is in a box. All this stuff they don't really care about. And like that's sort of the pendulum of tech. But yeah, I think, like, that's the sort of thing that really, to me, highlights like, oh, all of these experiences get tied together and it's the little edges, like the little scenes in between them that make a huge difference in how they actually are used at the end of the road. And I think somebody will just basically say, hey, that flexibility, you don't need it what you actually need is something that when you use it, you're like, this makes me happy to use it. Yeah.

And certainly on sometime horizon, I think it's likely that we will see movement in that direction. Your post recently on App Stores and specifically Snowflake's acquisition of Streamlit, I think is what that thread makes me think of. There's no more compelling reason to see integration than a very smooth experience for buyers to buy more stuff. And if, if Snowflake creates a way, I see the iOS app store as a little bit technology, but mostly a payments platform that everybody's just like, gotta save credit card in. And like you like double tap the button on the side of the phone, it does face id, and then you're just like, I think I just paid money, I'm not even sure. And so if the ecosystem is able to move in that direction, where we don't all have to implement Zora to do our own subscription billing, we just let Snowflake and Databricks and the rest of them do that for us and trust that because we're buying into these systems like the friction is dramatically lower. And I'm not actually suggesting that this is what I want. I'm only suggesting that like, there is compelling gravity to this, this vision of the future.

And I think there is definitely an attraction to be able to have those app stores that you can just plug and play things into a safe endpoint and not worry about integration as much. But the thing is like, if you look at Snowflake and you look at Salesforce, there's a big barrier to entry in terms of the cost of having those platforms. With each of them, you'll be spending hundreds of thousands of dollars really to just have access. And imagine if, like, if you go back to that model view controller analogy, if that controller was the access for like apps to, to access the data to therefore be part of the App Store, and that controller was more or less free for anyone to enter. And then the way the App Store made money was purely through the fees for, for the vendors being in the App Store. You know, that's, I feel like that's a bigger proposition than sales like Salesforce or Snowflakes app stores.

So, so I have, I have two questions on this and it's question for you mostly, David, because I think you're, you're. Avora, to me is a product that sort of fits into this bucket. Yeah, one, I think I'll do your point on that. On the thing, I think the reason, one of the reasons this is compelling for Snowflake is because they don't actually just have to charge the app store fee. Like, it's not just a fee. It's as if an iPhone also was like a carrier and made money. Every time people are using the WI fi, right?

Yeah.

You're ringing the cash register the whole time. You could like give every bit of the app call straight back to the, to the provider. It's just like now you're just driving more compute. Like so it's. In some ways I think there's the economics there make more sense with Snowflake. The question I would have though is, does the. Do the economics make sense for you? Because like, and I don't, I have no idea what the software world was like. I'm sure there is a huge difference in the way like software developers think about the economics of the industry. In a world where you buy software for 50, $60 a pop and you get a CD in a box versus what it is now. Like the expectation these apps are free are like $2. I don't know what the most expensive app I've ever bought on a phone is, but it's probably $10 and like the cheapest software I ever bought when I was like downloading stuff with a CD, which was 10 years before, all of that was probably like 30. And so to me that's like the question about like the data side of this is, does it make sense for folks like Vora to sell their product for a third, a fifth of what they currently sell for it if the convenience is there, where people could just be like, yep, okay, I'll pop this in and buy that to me is kind of the more interesting question in all this is like, do we, do we sort of change the whole dynamic of what these software products look like? Where enterprise products start to look more like app development, where they can be built with small teams, they can be built. Tristan, you've made this point before. They don't have to be all be venture funded. Like people can just build them with a handful of people and make pretty good money in doing that.

So I think it is possible because when you've gone for that product growth approach and you're quite lean, you don't have to have a huge amount of money to fund building those apps. Like a product like Avora is already a collection of data apps. Like it's two or three data apps already. So if that's plugged into one of those app ecosystems and rather than having tens of customers, you've got Thousands of customers. Yes. At maybe even one tenth of what you charge. Maybe as an enterprise customer, what you might charge an enterprise customer. The economics still can make sense and the friction for the customer through just, you know, oh, I want to connect this Avora anomaly detection app to the, to the App Store and then my data flows to it and I get to use it. That lack of friction is amazing. You know, there's some huge value in that.

I'm excited about this future for the different ways that you'll build companies. Folks may have followed our journey over the years, but we really believed at the outset that, well, once we started taking DBT seriously as a product, we were hoping to build it towards a model that looked a lot like Basecamp the company. So a couple dozen people and a lot of credit card swipes and it turns out that that's just like not how data works because the enterprise data space is really where a lot of the total dollars are in aggregate and enterprise just because the data is so highly sensitive that you have to plug into to do this stuff you have to go through legal and sales and all of these things. So we were naive and we updated our priors. But in a world where these types of guarantees in the enterprise are coming from the platform or the App Store instead of coming from the individual vendors, we could have built a totally different company. I mean there's a non trivial number of conversations that happen at DBT Labs today that are about a set of things that are not actually about the experience of an analytics engineer. They are like, when are we going to launch a multi tenant control plane in the eu? Which is like, it's incredibly important. There's a huge number of human beings who really want that from us, but it has literally nothing to do with the fundamental innovation. And there's many, many, many companies who are solving these same problems over and over again.

Yeah, the cost of sales in the enterprise is so high, it's like such a, such a high thing to be able to sell into. And I, I think like we've seen some of that again. I mean I have a little bit of experience in this, but it seems like we've seen that change a lot in tools that are some, like some verticals that has changed a lot that you can now sell to the enterprise without going through that process. And I kind of wonder if like data can get there. And that's basically to me what this is. It's like can data get to the point where the consumerization of data it like is that a thing that can actually happen. And I don't know, maybe, but it certainly seems like we're. We're inching closer to it.

And that's where I think the analogy with Salesforce comes in. Because Salesforce is enterprise software and people buy it for that. But then they also use Salesforce App Exchange to buy relatively low cost, you know, additions to the software or just apps to do something a bit different, and they feel safe doing so because all of the security control, Dax, all that's all handled, you know, even know, know the vendor, that's all covered by Salesforce, they don't have to do it.

Maybe none of us actually know the answer to this question. Maybe I'll have to Google it afterwards. But if you sell, sell an app in Salesforce App Exchange, do you have to keep all the compute and storage local to Salesforce infrastructure? Or can you, like, pipe the data out to your own servers, do stuff with it and pipe it back? Because to me, that's like, I think that we will, we should anticipate that if this model is to be successful, there will be real constraints around the way that you have to build products. Like, you will have to create these products that are completely contained to the platform, because the minute that the data can escape the platform, then they can't really provide any guarantees around it.

I think it's about storage as well. So if these apps store, take the data and then store them somewhere else, immediately adds complexity. Unless you know for sure that the data isn't sensitive and then that adds another level of complexity. Whereas if they maybe have their own compute and it's just processing and then sends an output back, that's different. I think that's a different level of security and control. But again, also, if what you do is on the marketplace, you're hosting your software and you're not providing the infrastructure, then that's even easier. Right? You haven't moved data anywhere. The data stayed in the app store.

Fortunately, Snowflake has ways to help store your data for you so they can at least solve that problem.

Yeah, and, you know, I think this.

Would see you getting to a place where, like this. Maybe it's taking this analogy too far. But, but the way that like iPhones and Androids have, like, which permissions do you give these apps and stuff like that. Like, does it have access to this or that? Like, I could see you having something similar. It's like this app only stores things locally. This app stores stuff on their side. This app will do this or that. And you Kind of like make the choice that way where there's, you know, these kind of parameters around what you give it permission to do and not.

Yeah, okay. We're talking a lot about infrastructure and it's probably my fault. I tend to think about things from the bottom up. But like the, the thing that's fundamentally motivating this conversation is that I think that all of us are interested in more humans being able to participate more meaningfully in data driven workflows, like doing more of their work in a data informed way. And so if this conversation is interesting to me, it's because I think there's really a very significant potential for lots more experiences to exist. And I think that in order to create the simplest data product experience today, it is a tremendously high and sell it to a customer. It is a tremendously high bar. And so the only ones that get created are the ones with the biggest possible applicability. They tend to be like very horizontal products. Like, Ben, you're famous for using the example of, of Yelp as a data product. And like there is not really a data product that looks like Yelp today that is widely used. And I think it's because the economics of it don't really work. So I'm excited for this entire vision because like, I think it enables more people to do more interesting stuff.

Yeah. And I, I think there's, I think there's a lot in that and there's a lot of appetite for it. Like I, the, the Yelp. The two points I'll make about that is a lot of people in the data world will, when you bring this sort of stuff up, they're like, the problem is in technology, the problem is you need to be analytical thinkers and things like that. Like making use of data isn't a technical skill. It's a, like a skill set that people need, which I think is true. But I think a lot of people like can get to that point pretty quickly if they are enabled in other ways. And I think the Yelp piece is part of that where like Yelp makes us pretty data driven and we kind of don't even notice. The other thing is a dynamic that I think I talked about this in the co op place to go with the dynamic that we see in mode, which is that mode becomes people. When people look at mode, they're like, this tool doesn't have that much self serve stuff because I have to write SQL. Actually what ends up happening is SQL becomes fairly self served. People like learn how to write SQL because it's not that inaccessible. And so in a lot of ways, one of Mode's best self serve interfaces is a SQL editor that you don't have to like get database credentials to access. And I think that same thing can apply where it's like actually giving people experiences that feel like they're opening the door to them, even if they are sort of more technical or more challenging, they may initially think can get a lot of people to places that we don't necessarily think they can. Tristan, you talk a lot about like the people operating at the top of their license. Yeah, I think like that license could be much higher than we initially sort of assumed. The problem is really just like these things are inaccessible and there's a lot of stuff that gets in their way to be able to do it. And these sort of apps to me are like a possibility of that, where it's not just like let's dumb data down to sort of super point and click stuff. It's like, actually let's figure out ways to open it up so that people can like start to understand a little bit more, can actually play with it more. Not by saying let's just give them a bunch of drag and drop tools, but by saying like, no, we'll kind of give you a constrained environment where you're actually kind of learning as you go instead of. Instead of just kind of operating in this very sort of padded, you know, garden rep list.

David, this is your space to be optimistic. We're mostly a bunch of curmudgeons here. Do you want to join us in our optimistic streak?

Yeah, I do. See, even with something like Mode. Yeah, absolutely. I love the Mode SQL Editor. It's great. But then you've also got things in there like the new Visual Explorer, which I think is really, really cool. But that in a way is. That's like an app on its own. It could be an app on its own that data gets plugged into. That's the sort of thing and you kind of end up with. There could be people who just also sell these kind of like runtimes that can be piped together with like Arrow or some other data format. So Hex is kind of like a system like that. It's a collection of runtimes that pipe together. And so if you have like the App Store as the entry point for those runtimes like that, it kind of works.

Okay, so coming to the close of this, this has been a fun conversation. I think that we've kind of said that the core debate of bundling versus unbundling is like maybe what we need to be doing is having a more nuanced conversation, one that doesn't start by drawing a traditional airflow dag and carving it up into these balkanized components. But is there something there? Do we have concluding thoughts around bundling and unbundling?

I do think there will be some bundling. Right. There are some things which are being done and are very similar disciplines. Like if you think about ETL and reverse etl, it's possible for those things to become bundled because the discipline is very similar. It's just directionally different. Observability discovery could be bundled. Right, those sorts of things. Because again, the discipline is so similar. It's about metadata management and lineage and things like that. And I think that's where maybe the use cases have become so fine grain that. Did they need to be that split? Sure, there could be some bundling, but then I think there's that. But that's almost like a horizontal bundling that could happen. And then like the vertical bundling, you know, I think that could be more, you know, that kind of marketplace thing, you know. But I think anything and anything and everything will happen at the same time. You know, there will be full, full bundled solutions. Like gcp, for example, could have a full bundled solution and then there could be a completely unbundle and there will be a completely un. As there is today solution as well.

Yeah, I don't know. We just detonated a bomb in the whole thing and now it's like we're picking up the pieces and figuring out what goes where. And one way that, I guess the way that I would put this and to kind of put my money where my mouth is moat sits on top of a lot of this. And so we have to make some bets on how this actually shakes out. And largely we're not going to make any bets. Right now my view is it doesn't make sense to be like, oh, what's going to happen is we're going to bundle airflow with compute tasks, or we're going to bundle data dictionaries with observability, or we're going to unbundle ETL and this other. Like, to me, there doesn't make any sense for us to build a roadmap on top of something that is in such flux. There are some places where it's like, yeah, you can kind of point these things look like they may be coalescing together or whatever. But I think it is still largely the places where I think there is enough mass that they don't split apart is like data pipelines, ELT warehouses and storage though apparently maybe that's coming down the road. But I think for a long time, like that'll. That'll stay and a transformation governance layer beyond that. How those things split apart, I don't know. But certainly it like at Mode, we're not going to make big bets on that's shaping out in one particular way or the other because I think like, too much is in the dynamic is too chaotic for us to make a whole lot of sense of it or to like predict one way or the other, which is maybe a boring answer.

But no, I think, I think it's useful. I will say that even though we are DBT sitting at the center of a lot of this, I don't want to come down on one side or the other of this question either. I think that later today as we record this, that later in the day I am going to attend a product demo where some folks internally have hacked together early experimental snowpark support and we've got DBT running Python models and participating in the DAG and all of this stuff. And so. Okay, great. I'm not trying to say that this is about to ship tomorrow, but I'm very curious about that experience and how it will feel and if it will be as magical of an experience as DBT on SQL has been for so many people. Because I really think that a lot of what we're talking about in the bundling and unbundling conversation is really things that you need a little bit less of. If your graph is all contained in a single structured format, I think you don't need a separate lineage tool. You don't. Your observability challenges are a little different than they would have been otherwise. So I'm not strongly opinionated that like, that's where everything is going. But I think it's something I'm very curious about and it's. It's possible that could be a capital V, capital G, capital T, very good thing for a lot of people if that's the way that things end up evolving.

I think if you look at like, things that top up from the consumption layer, you see tools like Moat and like Hex, where you can kind of pipe different things together, like SQL and Python or whatever you want. And those things have been really helpful and I've, I've really enjoyed using them and because the flexibility it gives you. So I can't see why that wouldn't be as well loved in like the transformation layer as well. As the consumption layer. And I think there's enough people crying out for it that you. I think you're pretty sure that it will be. And I think there's. And certainly even just at a batch level, yes, it's definitely going to be valuable, but I think it then enables us to talk about streaming maybe a bit sooner than before, and then that makes it even more valuable.

Whatever y' all are working on. Tristan, the dbt, excited to see it. I think I would disagree, actually, David, with the point that, like, the transformation layer should be as flexible as the consumption layer. There's a lot of things that you do, the final mile of data playing with stuff that I think, like, that flexibility in the way that you describe it is helpful, but flexibility is, like, flimsy that. I don't want the foundation of my house to be built with, like, a whole range of materials all sourced from lots of different places where some part of it was built with one thing at some point I wanted to be. You thought about this. You built one nice, big, sturdy thing that all works together really well. And then, like, you know, the rooms can be decorated with different types of furniture from different eras, but I don't need the house foundation to be built that way. And I think, like, there's some scariness to me of the pipelines being. And this isn't saying, like, it can't have Python and SQL, but there's some scariness to me in, like this. Well, we bounce around different things. We run it here, then we run it here, then we run it here, then we pipe it to this thing. Like, I kind of want a clear map in that place. And I think, Tristan, to your point, I agree with this. Like, right now you don't have a clear map because it bounces between different tools. I think that also applies, like, different modes of working. But if it's just like a much more structured thing, it's like the thing that I want to know is this thing is going to constantly work and I, like, don't have to go to bed wondering if it's going to collapse on me. And I think the more sort of flexibility you put in that in some ways the trade off as you make is you lose that rigidity.

And I agree with that. I don't think you can give it the same level of flexibility. Like, you have to be sure that whatever that node on that DAG is isn't, like, it's not going to, like, every time it runs, like, half the time it runs out of memory and fails or you can't do that. It needs to be more resilient. And that's I think why, I guess using something like Snowpot, which can guarantee more of those things because of the environment it provides, that then becomes feasible.

Tbd. We will see. We have to play with it more.

Yeah.

Thanks both. Let's leave it there. This has been fun and I'm very curious listeners. If you're into this format, let us know. If you think this was just a random conversation that Ben and David and I should have really had at a bar and kept to ourselves, let us know that too. But this has been fun. Thanks both for joining us.

Thanks.

Thanks Tristan.

The Analytics Engineering Podcast is sponsored by DBT Labs and is hosted by Tristan Handy and myself, Julia Schottenstein. Have comments, questions or guest suggestions, email us@podcastbtlabs.com Our producers are Jeff Fox and David Krevitt. If you enjoy the show, please drop a review or share it with a friend. Thanks for listening.