#252 Is Big Data Dead? MotherDuck and the Small Data Manifesto with Ryan Boyd Co-Founder at MotherDuck

Hi, I'm Maggie, VP of Curriculum at datacamp, and I know firsthand how overwhelming it can be to keep up with the rapid changes in data and AI. But staying ahead doesn't have to be hard. That's why our curriculum team has curated the best learning content to help you and your team stay ahead, covering everything from data and AI literacy for busy executives to cutting edge generative AI for developers, and everything in between. Whether you're upskilling yourself or your team, we make learning practical, engaging and flexible. It just works. Power your data and AI transformation journey on datacamp.com.

Now we're seeing like this need for a combination of local compute and remote compute. Fiber is only so fast. You're limited by the speed of light. So if you want to get like real time awesome visualizations of your data that are dynamic, that you could filter dynamically and things like that, why not run that on your local machine? Foreign.

Welcome to Data Framed. This is Richie. Everybody knows that the amount of data in the world is growing exponentially and that you need to continually scale your data platform in order to keep up. Except maybe that isn't true for most businesses. While some organizations have got ridiculously large data sets, not everyone's trying to analyze large quantities of video or create a large language model. In fact, for many businesses, core business data sets like financial state and customer data haven't increased much in size with the increases in computing power that have arrived since big data first became popular in 2011. That means that you can perform a lot of business analytics on a laptop. This realization means you might need to rethink your whole data strategy. It's the basis for the Small Data Manifesto championed by database platform MotherDuck. I want to know how big big data is these days, how you need to adapt your data strategy to deal with big or small data, and whether doing analytics on a laptop is better than doing it in the cloud. Our guest is Ryan Boyd, the co founder and CTO of MotherDuck. He's responsible for building the MotherDuck database product and building a community of small data enthusiasts. Ryan was Head of Developer Relations for Google Cloud platform, working on BigQuery at the height of the big data revolution, so his focus on small data is a notable change in position. In between his time at Google and founding Motherduck, Ryan was director of developer relations at Neo4j and head of developer relations at Databricks. That is, he has a rather extensive set of experience with a variety of databrick platforms. I'm keen to hear his insights on small data. And since Motherduck is built on DuckDB, one of my favorite databases, I also want to know how DuckDB is changing SQL. Hi Ryan, welcome to the show.

Hey, thanks Richie. It's great to be here.

To begin with, I want to talk about the size of data and for years there's been this sort of a stream of eye boggling statistics about how much data there is in the world and how it's growing exponentially. Do you think that's going to continue?

You know, I was kind of part of that messaging. I came from working on Google BigQuery for a few years, about 10 years ago or so. And I certainly said like, hey, data is going to grow exponentially no matter the definition. We knew that the size of your data was going to grow and huge bigger than you could possibly imagine. And BigQuery was going to handle that, of course, and I believed that message at the time. But I don't think it's actually grown as much as we expected. And I think that Moore's Law and compute and memory and distributed storage and things like that have all grown at a much faster pace than the data that we're collecting or at least the data that we're using.

Okay, so there's sort of different sizes of exponentiality there. So the data's grown, but maybe the computing power has grown faster than that.

Totally. I mean the data has definitely grown. Right. You know, we're starting to collect more and more. I mean there's obviously security and privacy regulations and things like that that put some constraints on that. But of course in the end, the more and more that we use computers, more and more you use smart devices, et cetera, the amount of data out there has grown. But there's only so much that people can analyze and only so much that is useful to analyze. And so it's that combination of factors is it hasn't grown as much as we thought. Plus there's only a subset of it that we actually care about analyzing.

So plenty of data there, but maybe not as much. Really interesting data.

Yeah.

And it also is in terms of the overall company landscape, the data that belongs to any given company. Most businesses out there don't have tons of data. So in Google BigQuery, 95% of our customers had less than a terabyte of data. single store, where one of my co founders was at, 80% of the customers want the smallest instance. One of the biggest requests we got when I was at Databricks was for single node spark clusters. Most enterprise data Warehouses are less than 100 gigs. So I think the standard businesses that aren't sort of the massive online businesses Silicon Valley is building technology for, they still don't have as much data, right? It's grown probably exponentially, but it started at a very small size for those businesses, you know, and even the startups that we've talked to, like Andreessen Horowitz, one of our investors, surveyed their portfolio companies. And on the B2B side, most companies had less than one terabyte of data. On the B2C side, most had less than 10 terabytes of data. And nowadays one terabyte of data, like I've seen SD cards advertised in my camera that are one terabyte. So I think a lot of it has to do with our bias. In Silicon Valley and in the tech industry, there are companies that have collected and analyzed much more data. Of course, and the people that are in that environment in San Francisco or other tech cities, New York, Seattle, Amsterdam, the people that are in that environment hear about all these big data use cases all the time. But that's not the vast majority of companies.

That is kind of interesting, I suppose. If you're Google and you own YouTube, you've got just petabytes of video data, you're going to think about big data problems, right?

You're the data 1%. And actually it's more like 99.95% or sorry, 0.05% is the elite group of people. And it's not just our background. The Redshift folks released an academic paper recently and they're actually talking at our upcoming Small Data SF conference. But Redshift is one of the more popular data warehouses and only 0.03% of the queries that are run query more than 10 terabytes of data. 0.03, like it's tiny. Other people are starting to see this, starting to realize this. We published these Small data manifesto on smalldatasf.com, which is the website for our conference. And huge roster of amazing speakers that are coming that really believe in this message, that the technology has allowed us to simplify our infrastructure and work with smaller data because that's what customers have.

That is absolutely fascinating that it was like 0.3% of queries run on 0.03%, 0.03 or even less than that. So basically everyone's just running, running relatively small sub 10 terabyte data set queries.

Exactly. So just because you have the data doesn't mean you're actually querying it and that's partly just what data you care about, but it's also partly the data technology has gotten better to be able to do partitioning and sharding of the data and things like that so that you only have to touch the columns and the row groups that you really care about.

All right, so you mentioned the Small Data Manifesto. Can you tell me a bit more about what that is?

The Small Data Manifesto is basically our belief that we need to think small as an industry and we believe in the simple joys of small data. But you know, some of this is also facetious, if that's the right word. Is that what was big data in 2012, you know, when a lot of the technologies that we're using today were created, is now small data. So it's a play on the words in a way. Like we say small data and people are just like. You mean just like Excel spreadsheets or whatever. I did learn that Excel spreadsheets now can have 2 billion rows, I believe. But that's beside the point. When we say small data, we do mean sub 10 terabytes or something like that, which is what most people have. But the manifesto is about focusing on the simplicity of our data infrastructure so that we can get the joy out of our data and the utility out of our data. And this is really about modern hardware is a beast. You can do so much just on your laptop. And this even carries over to the AI world is the bigger the model in an LLM, the more compute costs, that is the more data centers that have to go up, the more my Nvidia stock goes up. But the. But if you think about it, eventually this isn't going to be as subsidized and people like. One of our speakers at Small Data SF is actually my brother in law, Yulad Lotan at buzzfeed and he runs all the data work at buzzfeed. And they can call these large models, but the small models are 95% or something like that as accurate and cost a tenth of the amount. So if you think about that, like we can have small data, we can have small models. We can recognize that more data doesn't necessarily equal better results. You get diminishing returns with more and more data that you collect. We can recognize that single machines like your laptop are efficient and powerful. Like my laptop is over two years old, but it has I think 32 gigabytes of RAM and is, you know, sits 95% idle. Why not use that compute? And you know, we've seen people go back to that where developing locally is so much nicer than developing with remote systems and such. And the technology allows it. And especially in the database world, we're seeing folks where you can run DuckDB locally as a developer and you can push to prod and it's running the exact same database in prod on the analytics side and even on the transactional side there's a lot of startups like Terso, who's one of the co organizers for our Conference, who's using SQLite. And you can use SQLite locally and push SQLite into production in the cloud. Overall, we believe that small data and AI is more valuable than you think. And we believe that the technology changes that have happened really allow us to use the compute the way we want to use the compute, to have access to things locally to, to not have that latency between us and the cloud for a lot of things. Or to supplement the cloud. For instance, with MotherDuck we have this thing called dual execution and we can run queries that decide like which parts of the query should run locally, which parts should run in the cloud. Right now it's a fairly simplistic algorithm for deciding that, but we have some of the world's leading database researchers working on more cost based analysis of that. What is the bandwidth currently? What is the compute on both sides of the pipe? What, what is the cpu? And I have to acknowledge this is kind of a new style of distributed compute in a way. It's still distributed between your local machine and the cloud, but it gives you that advantage of being able to have really low latency access to your data on your laptop.

That's interesting that this sort of trend of everything going to the cloud is now being reversed and you've got the idea of hybrid computing. So some of it's in the cloud.

Some of it's local. Yeah, I mean honestly, the whole technology industry is a pendulum. It swings back and forth. When I started my career it was, yeah, you'd have local like sun desktops sitting in your office doing stuff. But largely everything was in the data centers, not in the cloud, but in your local data center. And you would remote desktop into those data centers when you wanted to do stuff or you would do like distributed SSH to push changes to a bunch of machines. And it was all about developing remotely in, in this data center environment. And then the cloud came along and it made it a lot easier to build things. But now we're seeing like this need for a combination of local compute and remote compute. Fiber is only so Fast, Right. You're limited by the speed of light. So if you want to get like real time awesome visualizations of your data that are dynamic, that you could filter dynamically and things like that, why not run that on your local machine? And especially with technologies like webassembly, that's possible. And DuckDB can run in WebAssembly. SQLite can run in webassembly and that means that it's actually running the same database in your browser. In DuckDB's case, it's less than a 20 megabyte executable, which frankly you see 20 megabyte images on websites nowadays. That's horrible. But you know, it's tiny, right? And that really gives us that power to go back to doing some things locally when we get advantages out of it. Like the pendulum shouldn't swing unless dependent variables swinging, but dependent variables also swing in our capitalistic society. I guess.

That's interesting. So I'm curious as to which bits of the data workflow you think should be done locally then? So it sounds like it's going to be maybe like late stage analysis where you're dealing with smaller data sets and then visualization. So it should be like dashboards, but local, but in your browser. Is that the, Is that the part?

So that. That is part of it. Right. So a lot of BI tools nowadays, black box embed DuckDB into the BI tool to enable those local dashboards. And this includes like the standard consumer user facing BI dashboards that the average data analyst can build in something like Hex, for instance, but also includes more of the dashboard as code tools like Evidence, for instance. Evidence. You can build a dashboard and it's basically you write and mark down your dashboard and it will actually give you a WebAssembly embedded DuckDB to speed up the local results of that dashboard. Because in my past world, working at one of the world's leading data companies, we had Tableau that was hooked into our lakehouse and I would drag a slider and I, you know, literally we had to go get a cup of coffee before the view refreshed. We don't want that. We want super fast access to your data. That really empowers people to do more analysis of their data because they're not waiting, because they're not frustrated. You can just look at your data in different ways. In Mother Duck, we launched a thing called Column Explorer, which is actually similar type of interface as some other data warehouses and such have. But basically it shows you the distribution of your data in different columns. What type Is it how many different unique values are there? Allows you to drill down and such. We're able to make it a lot faster because it's actually running DuckDB in the browser. So it's taking your result set back from the server and it's saying, okay, now let's figure out what the column view distribution and things is. And it's storing that so that you can drag sliders and such and see instantaneous results. And my favorite part of this is people developing tools like this no longer need to do all that data manipulation in JavaScript because JavaScript's a pain in the butt, if you ask me. So you can use SQL queries and allow people to have that level of interaction. So these are the types of things that we're seeing the. But we also see other things like people saying, well, there's this particular data set that I really don't want to store in the cloud for some security reason, as an example. And so you can take the data that is in the cloud and join locally against that data in one SQL query as if the data was sitting in the same place. And you know, that's amazing, whether it be like ID number lookups, not that Social Security numbers are private anymore because all of them for the US leaked recently, but you know, things like that, right, where you're saying, hey, I have this local Excel database and I want to analyze that and join it against what's in my data warehouse. You can do that nowadays. So it just enables like a new type of flexibility for analysts. Some of the stuff I've been talking about is mother duck, but that's really kind of been the theory around DuckDB as a whole is increase the ergonomics of working with data. It doesn't matter where that data is sitting. You should be to use the same type of queries to access it.

I like the idea of the ergonomics of data analysis, just smoothing things out, making sure you're not going from one tool to another and having to do different things in different places. And actually, since you were mentioning some of our competitors, I should probably also mention that DataCamp's data lab productivity tool also makes quite heavy use of under the hood. So that's like again, working with SQL in a browser.

And then, yeah, you can apologies that you had to. Honestly, the greatest thing that I like about working in the data industry is sure, we have competitors, but we're all friends. Everyone gets along so well in the data industry and it adds the smiles to the day.

All right, so just while we're on the subject of hybrid computing. I think one of the big pushes for cloud computing was just that it's kind of a pain managing software on your own machine. I think one of the big perks for me from moving to cloud was like, you don't have to like argue with IT departments about whether you're allowed this tool that you want to use. So how does, I guess it's going to affect software management if you're working locally and are there any of the processes that need to change if you're setting up hybrid computing?

This is another case where the pendulum swings like it used to be. You go and create a W S account and you have free reign on that AWS account. We recently went through the SOC 2 compliance at mother Duck and I'm a co founder and a leader in the company, but same rules apply to me as everyone else. There's so many of the AWS APIs that I'm not even allowed to call without getting explicit permission and making sure that it's written in wherever the documents say that I'm allowed to call it. And basically software over time becomes a little more bloated, a little more geared towards the enterprise security and compliance side of things. That obviously has great benefits and is necessary in many ways. But also privacy and security and usability are sort of at the opposite ends of the spectrum, right? You either gear your towards really good privacy and security. Google APIs is a great example. Like they used to be so much easier to use back when I, when I worked on them and over time like they added more and more API keys and restrictions on how those API keys work and verification by Google of your application and all sorts of things like that. And now they're kind of a pain in the butt to use if I'm being totally honest. And it's sad because like I was, I worked on, on off at, at Google, worked proprietary stuff called Authsub before OAuth existed. You know, I work on OAuth, I wrote a book on OAuth but I find it very hard to use the, use these APIs. So I don't necessarily think we're at the point right now where cloud makes it so much easier. I think that if you have completely unrestricted access to the cloud, sure it's, it's super easy. But some, one of our co organizers at the Small Data sf, the founders of the company, I think this was a llama and the founders actually created Docker Desktop and that was the era where they're like oh, we need to make sure that the same software that is running on your server is running locally. And so you would run Docker, you'd put containers in it. Somewhat heavy weight to access. We've seen lighter weight things over time. But the general gist is you can run the code that you would be running on your production servers, on your local machine and develop against that. And we want the same thing to be true for our databases. That's been much slower to happen as the cloud data warehouses and the lake houses and things like that don't just have an easy Docker image or a container that you can run to work with your data. And that's why things like DuckDB and SQLite and other things are so valuable, because you can run that same thing. Now, I can say the same thing for Postgres. I think a lot of people do run Postgres locally in their development environments, but we also find that a lot of people find it painful to get Postgres to do what they need to do for analytics, because it's not designed for analytics, it's designed as a transactional database. But a lot of app developers, because of that property that you can run it in your containers and your dev environment, prod environment, all is the same database. A lot of people just start there and start building things with postgres, and then they're like, oh, I need some analytics for my users. And they build it on top of Postgres, and then they eventually hit a performance wall where they have a choice of larger machine in Postgres, more time spent on indexes and optimizing all the queries, or going to a true analytics database. So we want to see the same thing happen in analytics, where you can build locally, you can push to the cloud if you want, you can push to a local data center if you want. And the technologies are kind of enabling that in the same way that postgres in Docker containers enabled it in the transactional world.

Okay, yeah, Postgres is just sort of amazingly popular. I mean, it's everywhere. So you mentioned that because postgres is a transactional database, it doesn't perform well for analytics. For people who don't know, can you give us an overview of, like, what's the difference between transactional databases and analytic databases?

Yeah, so typically transactional databases, the words can get kind of confusing because, like, for instance, in our analytical database, we also have transactions. But when you hear transactional database, what that typically means is that it stores the data in a row format. So Each bit of data you collect, you populate one row. You can think of like a web server log, or you can think of like a new customer or a customer's purchase has a bunch of different values and you want to store that all at once. And so you create a transaction usually ACID compliant, so you have your durability and reliability of the data and you store it to your data, but it's storing as one row at a time. And if you think about from an analytics standpoint, usually what you're looking at is columns. You're looking at, for instance, what is the total amount of tax paid by our users. And that is just in one column of your transaction. If you're a retail shop, for instance, and you don't want to have to load the entire row into memory, you want to load just that column into memory. And so that's exactly what the analytical databases are really good at, is storing the data in columnar formats and in addition to that doing things like we call vectorized execution, which is basically breaking those columns up even further for analysis and taking advantage of modern CPU infrastructures and something called simd. But that gets into details, but basically the difference is the columnar versus row based storage affects a lot of how performant and reliable a system is for transactions versus analytics.

Okay. Yeah, I mean, certainly I spend probably a bit more time working with Python or R compared to SQL and all that. You know, the data frames in both of those there, they're always columnar. So yeah, it's, it's good that SQL's kind of matching that well.

I mean, those data frames are usually meant for analysis. Right? Like if you're, you're working in Python and are like you're trying to do some data science or data analysis. And this is one area where I talked about ergonomics and DuckDB being able to access data wherever it sits. The ergonomics of using DuckDB in Python is you can actually analyze your data frames in SQL. So you can have a data frame that's in Python or R and write a SQL query against it. Or nowadays like arrow format is also really popular and you can run SQL against that, but you can also access your data that's in your SQL database as if it is a data frame. So you can go kind of both directions. And it's just, that's the beauty of it is no matter what way you're used to interacting with your data, it allows you to do that. And I don't know, even if it comes down to like CSV files, the world's most boring format, but the most accessible format. Some databases. It's really hard to work with a CSV file or a JSON file with DuckDB, they actually just make it so much easier. And that's one thing I love and I'm throwing a lot of praise on DuckDB. And just to make it clear, like I'm from a company called Motherduck, we partner with the folks called DuckDB Labs who build DuckDB and they're a shareholder in us, but they're a completely separate company and a company that I'm kind of in awe over. Both the company, I guess, and the creators of DuckDB, which is largely Hannes and Mark, are the two folks that created DuckDB.

It's very cool technology, I have to say. And I have some more questions for you about the Small Data Manifesto, but maybe we'll talk about DuckDB for now. So some of the changes to the SQL language that incorporated DuckDB, I have to say, very cool. Like in regular SQL, it's just a real pain to do simple things like calculate a median and DuckDB just has a median function. Makes it easy. Do you want to talk me through, like some of the other changes that DuckDB is making to SQL?

Yeah, one of my, one of my colleagues, Alex, has written a couple blog posts on this that maybe we can remember to include in the links with this podcast. But the blog posts are like friendlier, similar SQL in DuckDB, an even friendlier SQL in DuckDB. DuckDB has taken a philosophy of let's make things useful, let's improve these ergonomics so that people don't have to do repetitive tasks and things like that. So they started with a base of what is, what is largely the Postgres version of SQL and used the Postgres parser I believe is still in DuckDB. Then they said, okay, select star from. How many people have done that? Everyone has done a select star from. Why do you need to say select star? If you just say from table name, they return the results and it just skips the whole select star thing, which is, you know, it's. It's stupid things like that that really make it easier to work with your data more that you don't have to. It's not necessarily things you don't have to remember, but things you don't have to type so your fingers get a little happier. And they've also done a lot of other in terms of group by so you don't need to remember all of the different column names. When you do a group by, you can just say group by all. And that's a SQL feature which I believe started with DuckDB but then rolled out to the bigqueries and snowflakes and things like that of the world. So it is one thing DuckDB is doing is really building simpler ways to work with your data in SQL. And also those are rolling out into other database technologies. And honestly like it's, it's about time because you know, SQL, I think the 96 standard or the 99 standard is kind of the last ones that had any significant changes in them. And it's time for us to keep on improving our lives and how we work with, with SQL and our data.

Absolutely, yes. So SQL had its 50th anniversary earlier this year. Don Chamberlain on, on the show talking about it. But yeah, it was amazing at the time in terms of simplifying things, but things have sort of moved on since the 70s.

Yeah, no, I mean I, I'm a strong believer in SQL and it's funny because like I used to work at Neo4J for years and they have a language called Cypher for interacting with graphs. It's a graph database. I still very strongly believe that graphs are the most natural way for our brains to work with, with data. Unfortunately, commercially that hasn't won out. And you know, commercially people think of still of SQL and they think of tabular data. Cypher is now in SQL as a G SQL or something like that in the latest standard. But people still work with data in tables. Whether I like it or not, whether I think it's natural or not, that's life. That's what we build with DuckDB and MotherDuck. Even though you can actually use DuckDB to make graph like queries and all. There's some great blog posts on that and I guess I should do a call out to. We just, we have a website called duckdbsnippets.com where we put all sorts of things from the community where they find interesting SQL queries or interesting Python scripts for using DuckDB or R scripts. You can submit any of that and there's, I don't know, probably a hundred different snippets out there right now. Always looking for more contributions from the community of just fun things that you can do in SQL, whether it's DuckDB specific or not. And so check that out. Duckdbsnippets.com that sounds like a fun educational resource now.

So you've mentioned Python and R a couple of times and I'm curious Do you think there's an influence of Python and R on SQL now? Is that influencing the direction of the language?

I think it just comes down to a lot of people have worked with those languages that work with data and SQL is having somewhat of a renaissance in terms of people realizing that, okay, the NoSQL movement, yeah, it was good for some things, but SQL still persists. Then the Python and R folks are starting to pick up SQL to be able to interact with their data in their data warehouses or other places. And so tools like DuckDB actually started as an embedded database for the creators of DuckDB. Hannes and Mark just saw with pain how the R users were interacting with their data. And they're like, why don't you use a real database? And they're database professors, right? Like PhDs in databases. And they saw this pain of their colleagues and they're like, we're going to fix that. Cause the colleagues were saying, well, databases are hard to install or databases you have to connect to over the network and the network is sometimes slow and all of this. And that's really the origins of DuckDB is how to make life easier for analysts in R. It eventually became the Python version is a little bit more popular with DuckDB, but the origins really were there. So I'm sure whether consciously or unconsciously the Python and R data analysis stuff is rubbing off into SQL.

It's quite a nice story, like the fact that all these sort of analytics communities that were traditionally separate are now interacting and good ideas are bleeding from one group to another.

I actually have a hat over here, so data person, because, you know, we say data analyst and data scientist and machine learning engineer and data engineer and all of that. And vast majority of people we talk to end up doing components of all of that. So there really are data people out there. And I think especially in economic times where people are trimming down, like you're expected to know how to do all this stuff if you're on a data team in most organizations. So yeah, I think the Python and our stuff rubbing off onto SQL and SQL rubbing off onto the. The Python and R users, I think it's just naturally going to happen, except maybe some places in academia or what have you. But we do have registered for our conference, the Small Data SF conference. We have folks that are oceanographers and folks that are deep scientists in various areas and they're starting to use SQL, which might not. They might have been in MATLAB if 10 years ago. Right. Or things like that.

Yeah, that's cool. I Do love how almost every industry has some sort of weird thing, weird role, where you don't expect there to be data used, but somehow actually data's just invaded that space now.

I mean, I honestly think everyone within an organization won't necessarily have a data title, but will need to access data, and we need to make it easier for those types of people to access data. I mean, right now, Mother Duck is young. We just went GA earlier this year, so I'm not going to claim that we're there yet. But the aim is, yeah, every data analyst that just knows SQL should be able to use motherduck without a ton of data engineering experience and things like that, because they have data that's sitting on their computer or they have data that's sitting in HubSpot or in Stripe or whatever, and they might want to just download that and upload it to motherduck before they end up doing full data pipelines and things along those lines or hooking into fivetran, et cetera. And yeah, I think that comes everywhere within the organization. Like, I've worked in database companies where I didn't have access to the central data lakehouse, you know, largely because I didn't feel like filling out all the forms for the security stuff. I was a senior director, but, like, there's still a lot of paperwork to fill out to get access to things, but, like, I had access to the raw sources. So, like, I could go into our marketing systems and Download spreadsheets or CSVs of the data and then upload it to a different bucket and use it. I probably shouldn't admit all of this, but, you know, I think that everyone kind of works around and uses what tools that they have at their disposal, and some of that is just working around policy restrictions.

Yeah, I'm sure next time we do a security episode, it's going to be someone grumbling about what you've just said.

But yeah, I mean, there's more people who grumble about it, but, like, it's true. True. Like, that's, that's how organizations work. People want to get. Yeah, well, I was about to swear gsd, you can look that up. But people want to do their jobs and get home earlier in the evening and play with their kids, so why not enable that to happen easier? And that's all the way down the stack from what they're running on their machines to what they're coding their, their analyses in. And that's why I've always believed in sort of the. Well, developer ergonomics is What I've called it, but that developer is air quoted. Developer, data analyst, data scientist, whatever is like let's make this easier so everyone can go home at night, right?

Like, not very many people want to mess about feeling like with installing software or getting stuff set up. They just want get the insights from their data as fast as possible.

Neo4j, when I worked there was a, was purely a local install database, right? Like they didn't have a cloud version when I was there. And the first thing I did was like, okay, downloading a database, you expect this downloading install to take hours or days because like you've all or a lot of people at that time had worked with Oracle for instance, or Even worked with MySQL and the PHP, MyAdmin and all of that space, right? And it took time to install all these things, configure them, run them. One of the things I did with 4J is just downloaded install and ran my first query in 57 seconds and created a video out of that just to kind of convince people that we can head to this new world where it's easier to download and install. And the DuckDB side pip install works great. It's one command to run or brew and tools like that. So yeah, I mean, I think we've gone from the really painful world through slightly less painful with things like Docker through today having these frameworks for installing and managing our software like Brewer or PIP or what have you.

Okay, yeah, writing one line of code that sounds doable for a lot of people. Just yeah, brewing.

So chat.OpenAI.com you can run one line of code for pretty much anything now.

Yeah, absolutely. So I'd like to get back to the Small Data manifesto and I want to kind of test some of the limits of this. So you said 10 terabytes is sort of the limit of small data these days. Are there any types of data that aren't going to be that small? So I think things like image data and video data seem fundamentally big. And everyone's saying that it's an official data type now. Data isn't just numbers anymore. All these other types are. So what do you do about those?

It depends on what type of work that you're doing with that data. So I have a bunch of security cameras at my house and, and they all upload to, you know, an S3 bucket and become glacier objects and things like that. And I'm sure that is as many terabytes of data am I accessing those security cameras on a regular basis. Last Time I needed to access them was when my bike was stolen out of my garage and the police didn't even care. So do I really even worry about that data? So you have to think about that is like, what type of analysis are you going to do and when are you going to be doing that analysis? I do do like AI on some of the security cameras for people detection, car detection, what have you. But I only need that at the moment, right, Like I'm not like going back into history and doing that. So I did say 10 terabytes. The reality is, is the number will shift over time. I don't want to say, well, 1.44 megabytes on a floppy disk. If you're beyond that, it's big data, or 10 terabytes beyond that, it's big data in perpetuity because the, the hardware is changing. The max size of an EC2 machine is 25 terabytes of RAM. So if you're running on that machine, maybe you can work with a lot more than 10 terabytes in your data. And then there also tends to be a confusion between the amount of data stored versus the amount of data analyzed. And those are very different things. And the requirements for the software are very different. But as you mentioned, binary files and things like that. Nowadays we have great cloud storage mechanisms that provide the distributed N feature of storage, which allows us to actually run services on a single node because we have that underlying distributed storage network. And so I do believe in things like the Lakehouse, for instance, in providing like an organized way to store all the data and the binary data and things like that. Especially kind of the older the archive per se, what I might put in the glacier storage on an S3, the archive. And that's the one nice thing with both Motherduck and DuckDB, as well as a lot of the data warehouses nowadays do allow you to work with the data in a lakehouse and also the recent data in the warehouse. And that I think is important, you know, having easy ergonomics to access that other type of data. But the distributed storage systems are great. You can throw whatever you want out at it. You know, we just published documentation about Cloudflare R2 and some of the new things that they allow you to do there. The distributed storage systems are getting better. So it doesn't worry me that some people do have, quote, unquote, big data. You still have to ask the question, what are they doing it with? What is the timeframe that they care about? Is it just today or last week or the quarter and everything else is aggregated before that. There's kind of nuances that you get there, but I don't think it disproves the idea of small data. Even though you have some of these blobs sitting around.

You might have vast amounts of security camera footage, but you're probably not going to do crunching numbers on all of it.

Unless, I mean, I've debated doing that the heck of it because it might be fun to know like over time how many times do I leave and return to the house. But like, honestly this is. It's not our biggest problems that we deal with, I guess.

So I was curious as to how much data Data Camp has and where our biggest data sets was. Well, I start head of data and he was saying that by far the biggest data set we have is web events data because everyone clicking on the website, that's a huge raw data set. But you don't actually analyze like the whole thing, like individual clicks no one cares about. So it's always aggregated. So what does the mother duck setup look like then? If you've got big raw data sets, but then you want it processed in some way or aggregated before you analyze it.

I mean, I think you do sort of standard aggregate tables. So that's what we have for our own data warehouses. We have separate tables that are aggregates. The raw data is still there, can still be accessed oftentimes since you are just like the event data that you're talking about. Like yeah, it might be super large, but also for instance, you might have the user agent string in there. That's super huge. Right. But you might not care about that. Usually we don't care about the browser that the person is using nowadays we care mostly about what is the event and what is the user. Two columns, maybe time, three columns. Right. So I'd be curious if you asked your data people if like if you broke it down by columns, is it sub 10 terabytes, is it that you access on a regular basis? I would imagine so. And maybe datacamp is in the data.05% or whatever, but there's a lot of companies out there which don't have the amount of web traffic and all that you folks have. So I don't know. Yeah, I think that from a mother duck standpoint it's things like logs, like data that goes through our datadog and things like that. That's the type of data that's large for us. I imagine for a lot of other organizations it's things like sensor data from connected devices and such. But like you said, you're not analyzing all of that all the time, right? You're analyzing a subset. So the question is, what is the data size of those two columns for the last month? And then can I build aggregate tables for each of the other months? How far back do I need to store daily aggregates and how far back do I need to store all of the events? There still might be cases where you say, you know what, I stored all of this stuff and I forgot to analyze xyz and I want to analyze XYZ on all this historical data. Those situations exist. And I would say for some cases, maybe you'll use a distributed compute engine, maybe you'll use Spark or something like that for that. But also people are finding new and creative ways to use, like, lambda functions. We've seen people run thousands of DuckDB instances at a time for ETL jobs in Lambda functions. And there's actually a great story from the folks at Okta, Jake Thomas, I believe at Okta, who has built their entire ETL infrastructure using DuckDB using single computer instances. Now, granted, he's doing it in a distributed way, but in a distributed way where the ergonomics have gotten a lot easier due to things like lambda functions.

Once you're running thousands of instances of DuckDB in parallel. Is he just reinvented BigQuery?

It depends on how those DuckDB instances need to communicate with each other. So that's the real problem with a lot of these distributed systems is the overhead in communicating with each other. So BigQuery, for instance, when I was there, I'm sure these things have changed. Every query would hit 1100 different machines, and there would be 550 different sets of data that each of those machines had access to. And every query, every bit of data would actually be analyzed twice, and they would throw away the slowest one so that you didn't have one machine bogging down the whole query. So there was some fun things like that. And as a geek, like, sure, that's exhilarating, but you don't need that for the vast majority of queries is kind of the point, especially as hardware has grown at the time, you might have, at the time when the max size of memory on an EC2 machine was, I think, 38 gigs or 41 gigs, something like that, you wanted to have tons of machines if you were going to analyze terabytes of data. It's just the world has changed. And so, you know, I think, yes, folks like Okta are doing something that's pretty kind of out there. But they're, they are in the, the data 1%. You know, they do the, the authentication authorization for the vast majority of the, the Fortune 1000, I think, and they have lots of data and they just looked at it and said, you know what, we can break this data up and we can have DuckDB analyze it, write it back to storage and then do the aggregates like that and watch the video. I think we'll include some links here too, and share that. But it's just innovative the way that the folks have approached, like taking this very simple tool and using it in a fun way that does fit largely what you would call a big data use case, a distributed use case. But they found it easier to use than, than something like Spark.

Suppose you've convinced some people about the Small Data Manifesto. What advice would you give to them in order to adopt a small data ethos?

Not to steal an Apple slogan, but think differently. Like, look at your data, look at your infrastructure. Are you building your infrastructure for the size of data that you have? And for as fast as the computer technology is growing, we really do want people to kind of have that thought process. Where could I get better ergonomics for my analysts, for my engineers? Do I need the complexity of some of this software that I've been. Do I need the maintenance cost of some of this software that I've been running? Or just something much easier work, something much simpler? You know, I think the phrase that we now have on our own page is the ducking simple data warehouse. And, and that's a lot of these technologies, as they get built over many years at some point just become so complex that it's painful to use. And a lot of that is due to the distributed nature of technology. So there are some people that are still not going to believe it. My co founder Jordan wrote a blog post called Big Data is Dead. I thought we were going to get all sorts of crap for it from the community, but I would say 80% of people believed it. 10% of people said, you guys are full of crap. And 10% of people were like, huh, I'm confused, but I'll look into it. And there's still always believers or people that hold a definition in their head of what is big data that's different than everyone else. But to me, and to the 80% of people that I've talked to, we really do need to focus on simplicity instead of focus on how distributed we can make our technology.

Okay, I like that. Pick the simplest solution that's going to fit your use case.

Yeah.

Okay. Excellent. On that note, I think we'll wrap up. Thank you so much for your time, Ryan.

Absolutely. Thank you. Richie. It's great talking with you.