#201 The Database is the Operating System with Mike Stonebraker, CTO & Co-Founder At DBOS

Hi, I'm Maggie, VP of Curriculum at datacamp, and I know firsthand how overwhelming it can be to keep up with the rapid changes in data and AI. But staying ahead doesn't have to be hard. That's why our curriculum team has curated the best learning content to help you and your team stay ahead, covering everything from data and AI literacy for busy executives to cutting edge generative AI for developers, and everything in between. Whether you're upskilling yourself or your team, we make learning practical, engaging and flexible. It just works. Power your data and AI transformation journey on datacamp.com.

So one of the environments that we have DBOSS running on is the MIT Super Cloud, which is in Holyoke, Massachusetts. It has 32,000 processors. It has several terabytes, that's for the T of main memory and many terabytes of secondary storage. So the resources that the operating system has to manage has gone up by about six orders of magnitude in the last 40ish years. So without me saying another word, that makes managing Operating System State, which is keeping track of tasks, resources, processes, files, all that stuff, and keeping track of operating system state, that makes it a database problem. Without me saying another word.

Welcome to Data Framed.

This is Ritchie.

You don't need to be a data scientist to know that databases are everywhere. Well, almost. While the majority of the world's data is stored in a database, there's one final frontier that databases haven't yet conquered. That's the heart of the operating system. At the moment, databases are applications that sit on top of operating systems like Windows, Mac os, and Linux. However, today's guest had an idea to invert this. He asked, what if operating systems were built on top of a database? That guest is none other than Mike Stonebraker, the inventor of five database systems, including Ingress, one of the first relational database systems, and Postgres, one of the world's most popular databases. That is to say, he's been working with databases since their very beginning 50 years ago, and he's one of the most influential figures in the field. Since 2001 he's been a lecturer at MIT, and his fundamental work on databases has also given him the highest honors. In 2005 he was awarded the IEEE John von Neumann medal, and in 2014.

He won the ACM Turing Award.

On top of this, he's been the chief technology officer for seven firms, and at the age of 80 he's just formed another startup, dboss. The company shares a name with its main product, the Database Oriented Operating System that we're Here to discuss, let's find.

Out why you might want an operating.

System built on a database and how it can help your business.

Hi Mike, thank you for joining me on the show.

Oh, thanks for having me, Richie.

Wonderful. So you're most famous for your work on the PostgreSQL database, so can you tell me what you think led to its phenomenal success?

So, first of all, its phenomenal success had almost nothing to do with me. It was picked up in 1995 by a pickup team of programmers who have promoted it, shepherded it ever since, and so they deserve most of the credit. And you say, why is it taking over the world? Well, it's a much better database system than MySQL and hopefully Cream rises to the top. But also people were kind of afraid When Oracle bought MySQL and Postgres has remained purely community driven all this time and I think it's a perfect example of what open source is supposed to be. I'm delighted that the elephants, the cloud elephants, are pretty much standardizing on the postgres wire protocol and so I think it will become a very, very dominant database system. While the interface there will be lots of implementations.

Yeah, I love that. It is really, it's a community that's grown and created something amazing and it's still continuing to evolve sort of decades after its introduction. So is there anything that you're most excited about in the world of SQL databases?

Andy Pavlo and I wrote a paper, well, I wrote a paper in 2007 saying what goes around comes around and there aren't any new data models. Andy Pavilo and I wrote a paper that's going to appear in SIGMOD record that's 15 years later. And here's a summary of what the paper says. There aren't any new data models that are going to get fraction in our opinion and all the interesting ideas are in either hardware stuff or in new applications. And I think, for example, everyone is moving everything they can to the cloud as quickly as they can and that's for all kinds of good reasons. And so it seems to me the most exciting thing that I see is that if you're doing one of these cloud migrations, you have a once in a generation opportunity to try and fix the sins of your predecessor. And so you can either do a lift and shift, at which point your successor will inherit the sins of your predecessor, or you can refactor, rewrite, intelligently move to the cloud. And I think that's the most exciting macro trend. Other Things are when you move to the cloud, the cloud basically forces you to have disaggregated storage and that's forcing all the database vendors to completely rewrite their stuff. And the reason for that is that networking has gotten a whale a lot faster than it used to be and that enables you to do disaggregated storage. The second thing is the big cloud vendors make it financially very attractive to have software as a service functions, service serverless computing. And that will encourage all application writers to rewrite their stuff. Also will encourage database systems to adopt that model. So there's lots of changes driven by, by the cloud guys in terms of new applications, I think how is machine learning, large language models going to be supported by database systems? Where's that going to go? I think the exciting thing to watch, I think genomic databases are going to become much more prevalent. And how are those going to be supported? And then I think the topic that we're supposed to talk about is can the operating system really become a database system? That's basically a new application area for databases. So I think new application areas are fascinating and I think the cloud hardware changes are really fascinating. I'm not expecting anything to happen in data models. I think the relational model is the answer and I don't see that changing.

Lots to unpack there. I like your first point about how yours inherit the sensing predecessor. Like, I think this is familiar to anyone who's worked with data, that there's some sort of horrendous blob of data there that you're like, well, yeah, I'm not sure I want to touch that. So the idea that you've got to maintain things and improve them as you go along, avoiding the technical debt, that seems very important. And then on your last point about databases and the operating system, it just seemed like databases are pretty much everywhere and the operating system is one of the sort of last holdouts. So since your new project is around creating an operating system with a database at its core, can you tell me why do operating systems need databases?

So I have lots of gray hair. So I was a very early user of Unix in 1974 on a PDP 1141 processor, 48K of main memory, not M or G K, and 20 megabytes of secondary storage. So one of the environments that we have DBOSS running on is the MIT Super Cloud, which is in Holyoke, Massachusetts. It has 32,000 processors. It has several terabytes that, that's with a T of main memory and many terabytes of Secondary storage. So the resources that the operating system has to manage has gone up by about six orders of magnitude in the last 40ish years. So without me saying another word, that makes managing operating system state, which is keeping track of tasks, resources, processes, files, all that stuff that makes keeping track of operating system state, that makes it a database problem. Without me saying another word. So you want to apply database technology to what's obviously a database problem. So that's number one. Number two, Linux is now 40ish years old, Unix is 50ish years old, and that makes them legacy software that's been maintained, patched, extended over a long period of time, and the Linux community is having a very hard time making forward progress. So for example, there is no multi node version of Linux. So everybody who is running a multi node system, which includes most everybody, has to run a multi node orchestrator, something like Kubernetes. Also, Linux is well known to be a leaky security boat. So people layer all kinds of security stuff on top of it. And so what you have is a patchwork of operating system software that is a management nightmare and is still a leaky security boat. So the fact that Linux is legacy means that it's time to send it to the home for tired software. So scale and legacy are the reasons to start with a clean slate and start anew.

Okay, that multi node idea in particular sounds interesting. So I know, like earlier you mentioned, that this is a solved problem with databases is that now because databases are in the cloud, they have to work on multiple servers at once. And so I think I can see where this is going in terms of having the operating system have that capability as well. I am old enough to remember that this has been tried before. So maybe 20 years or so ago, Microsoft tried to put a database at the heart of the operating system with WinFS, and then they abandoned that project. So what's different this time?

So the project, which was called Longhorn internally, sort of sleuthed around a bunch as to what happened when we started building dboss. And the general consensus inside Microsoft was that Longhorn was a good idea. Nobody disputed the ideas, and everybody internally said that its problems were bad management and feature creep that got more and more and more ambitious before it ever worked. And so I think technically it's a very good idea, and was then. And it suffered from internal politics, internal management issues, and especially feature creep. I've been involved with or watched a whole bunch of startups, and the worst thing in the world that any startup can do is engage in feature Creep. You want to get something running, and once you get it running, then you can worry about extending it. So Microsoft didn't pay attention to that lesson.

It's definitely a big problem being tempted to add lots of features because when you're just building something from scratch and there's like so many exciting things you want to add to it, but I can see how that can become dangerous if you've not shipped anything. So one thing you said you were excited about in the world of databases is that there are lots of application specific databases for different use cases. And it feels like the same thing is also true of operating systems. There are different operating systems for different use cases. So what is your intended use case for DBOS?

DBOSS started as a academic research project in 2020 jointly with MIT and Stanford. And so we had a running system that we decided to commercialize. And like any startup, the mantra is get a product out as quickly as you can, and then in the vernacular, see if the dogs are going to eat the dog food. And if they're not, they will tell you instantly why they don't like it. So our goal is to get a product out as quickly as possible. And we did that. So in a year, we've shipped a commercial version of dboss, and we had the advantage of having a person named Michael Cohen be part of the research project and also part of the commercial venture. And until recently, he was the managing partner of the cybersecurity practice for Boston Consulting Group. And so through him, we got to talk to lots of enterprise folks, big and small in a whole bunch of different areas. And so here's who saluted to those conversations. First of all, the three letter agencies, the defense industry, who is really focused on security. The second place where people really saluted were in financial services dealing with moving money around. And so one thing I learned from interviewing a large regional bank here in the Northeast was they listened, they talked with us and they said, wow, you solved the once and only once problem. And that was the first I'd ever heard of that. But here's what he actually meant, which is, if I'm going to move $10 from my account to, to rich your account, then we're almost certainly in different systems. The way the transaction should work is you debit my account, send a message to your account, increment your account, send a return message, and then commit the transaction. So this is basically a distributed commit problem. And in the banking world, most of the systems that this regional bank used don't have XA support or any distributed commit support. So the bank was forced to do this themselves. And so they figure that somewhere between a third and half of their application logic is dealing with once and only ones. And it's brittle, problem prone, hard to get right. Distributed commit is not for the faint of heart. And so they would love to get rid of all that code. And we do it automatically because we run the network system is in the database, the database is in the database. And so we control all pieces of that interaction. And so we solve the once and only once problem automatically. And he was very excited about about that. So Fin Services worries about distributed commit, also worries about security. The third place people got very interested was what I'll call Scuff Shoe Enterprises, which are enterprises that bend metal and do real things. One particular enterprise that who I won't name the describe their current security system. They have 100,000 endpoints, they're a very big conglomerate. And so they have one particular security vendor who does event extraction off of 100,000 endpoints. That's a seven digit a year subscription in US dollars. And they then send those events through a their enterprise's custom workflow system that enriches the events with enterprise specific data. They then pass these enriched events to another security product, another seven digit a year subscription. And they and the vendor have written several hundred monitoring rules. And so in production, if a monitoring rule fires, they have a human analyst look at it to make sure it's not a false positive. And if it's not a false positive, then they take action. The time elapsed time between when a bad guy knocks at the door and them taking action is measured in multiple hours. And they are just terrified of ransomware attacks. And so they have known many of their peers to have succumbed to ransomware attacks. It tends to take all production down for multiple days at a time and costs a billion dollars if you're a big enterprise. So they are terrified about security. And the thing they love about DBOSS is DBOSS recovers automatically from ransomware attacks. And the reason we can do that is we have everything in the database, everything. All the operating system state is in the database. And we keep a log of that state. Historically we spool it into a data warehouse. And so if you want to back up the operating system 18 minutes, you just do it. And if this is fast enough for the operating system, it's certainly fast enough for your application. So if your application puts all state in the database, then when we back up 18 minutes, we back up everything 18 minutes. And so if you had a ransomware attack 17 minutes ago, you just back up 18 minutes, single step around the bad guy and let the system go. So they're very excited about our security story. Their problem is that they are dragging a huge legacy code base around. And so to take advantage of DPOs, you would have to restructure, refactor, rewrite a bunch of it. And that's a project for the decades and so that will be a very, very slow market on the uptake. So with that said, what we're aiming for, the initial commercial deboss users are the three letter agencies, financial services, especially startup and adventuresome enterprises who are willing to refactor stuff on the way to the cloud. So that was a long answer to your short question.

Yeah. So I have to say I was thinking you swap out your operating system, it's probably going to result in a fairly small productivity boost. But the things you're talking about there, these are pretty dramatic. Like the idea that just solving this distributed transaction problem that's going to enable banks to cut out half of their code base, that's going to be a huge productivity boost from the maintenance and then the manufacturing example of just saying, okay, we've got better security, this is going to protect us from run somewhere. This is like pretty amazing stuff.

The thing I found really, really exciting is that most everyone we talk to about the idea thinks it's conceptually fantastic. And so the getting to an implementation is a small matter of legacy code. So anyway, I'm really excited about the possibilities.

This does sound pretty cool. So it sounds like the big use case of this is going to be around infrastructure for cloud applications. So for your software as a service stuff. And so can you talk me through in general, what does the infrastructure for cloud applications currently look like?

Well, to start with, if you move to the cloud, then as I said earlier, you're highly encouraged to take a software as a service model and you get disaggregated storage stuff like S3. And so our point of view is that if the world is moving from on prem to the cloud, we should go to where the market's going to be, not where it was. And so right now dboss is a cloud only service. It runs software as a service so that you only are using resources when you are actually running and if you're idle, you're not using any. And the other thing to note is that transactional databases have gotten wildly faster in the last decade or two. So it's fast enough to put a database system at the Bottom, that's what we do. So we run on the bare metal. Well, we run on a microkernel and at some point we may well write our own microkernel so that we're really running on the bare metal. And so Linux is nowhere in sight. Right now we are running on aws. We're running on one of their microkernels called Firecracker. And the database system is the only thing running on top of Firecracker. And on top of the database system we've written a file system, a messaging system, a bunch of schedulers, and your application runs on top of that stuff. So just so everybody gets really clear what we have in mind, let me tell you how the messaging system works. It is not TCPIP at all, it is not a heavyweight thing, and it's all written in SQL. And what do I mean by that? Well, there's a message table with a sender, a receiver and a payload and send a message, you do an insert into that table. That's one line of SQL. We're running on top of a partitioned multi node, highly available DBMs. So that tuple ends up at the home site of the receiver. And so to read a message you just do a SQL query, another one line SQL. So that's the message system. Once you go outside of our environment, we have a gateway that goes out onto tcp, IP and the rest of the world. But inside us it's all just the database. So everything is just the database. And so the database system is the only thing running besides a small microkernel. And as software as a service. The way software as a service works is you have to structure your application as a graph of workflow steps. You can call them micro ops operations, they're just pieces of code. So right now we've decided to support. We used to run on JavaScript and Java and now we've moved to TypeScript because it seems to be more popular. So you write a collection of operations in TypeScript and you tell us the graph of those operations. So that's the way you have to structure your application to get software service to work. We accept that graph and those operations, we store them in the database and we have a tiny orchestrator that wakes up any given operation when its inputs are available and it produces an output. All of that's in the database. So you just have to write a graph of TypeScript and we take care of everything else. So we run it for you. If it seems to be running slowly, we give you more resources and so forth.

Okay, so I have to say, I love the idea that the messaging system is just a table and a database so you can write C4 queries against it. I'd love to get into that a little bit more later, but just for now, what are the implications for anyone who wants to develop applications on top of dbos?

All you have to do. Well, you have to run on. We run on the cloud at some point we will probably support an On Prem deployment. But on prem comes with just a ton of idiosyncratic behavior on the part of whatever your shop is doing. But we run on aws, we will run in the near future on Azure and on gcp. And so all you have to do is produce this graph of TypeScript. You have to be using TypeScript. And I expect in very short order we will support 10 languages, because TypeScript is certainly not. It's popular, but it's certainly not universal. We will probably support Java, will probably support JavaScript, we'll probably support Python, we'll support Go if there's enough. I mean, we'll support languages as there's interest in us supporting them. And in short order, we will run on the popular clouds in a variety of programming languages. And what you get is that every one of your operations is a transaction. One of the big problems people have with software as a service applications is that they're broken up into a whole bunch of steps that are running in parallel. And so if there's race conditions between the parallel operations, those are fiendishly hard to debug. But in our system they're transactions and we sort it out with the concurrency control system. So they're basically few to none race conditions. And so if you know the word heisenbug, it was a term coined by Jim Gray, the late Jim Gray. We avoid almost all heisenbugs. We also give you a debugger because if we can back up the operating system, we can also back up your application. So if you're in debug mode, you simply end. Something bug bad happens, you can simply back up three minutes, single step forward, change the code, change the data. So we give you a really nifty debugging environment, but we give you transactions for everything. If you want to ask questions about your application, the state of your application is in the same data warehouse as the operating system stuff. So you can just use SQL to ask questions about what's happening. You can use SQL to ask, for example, if you think that I'm possibly a bad actor, you want to know who I've sent messages to who they've sent messages to transitive closure of who I've ever talked to. And that's just SQL. So right now asking questions about what's going on is really, really difficult. And it's very easy to do monitoring. So we talked, we. Right now you have to sort of move what amounts to the event log into some proprietary system that talks a proprietary language. In our world, it all goes into a data warehouse. You can just query it and SQL. So you get much easier monitoring. You get a fancy debugger, you get super security, you get multi node support, you don't have to run Kubernetes and you're not running Linux. And so you get a much simpler environment to maintain, a lot less moving parts, a lot more security and you get a next generation programming environment. So it's very attractive to application developers. We've talked to a whole bunch of them and most of them think, wow, this is really neat. And then they say, yeah, but you don't support go or pick an objection. And so the real question is we could support all of posix, which is sort of yesterday's standard. I'm really reluctant to do that because most people on the cloud don't care about POSIX at all. They're focused on workflow standards, stuff like that. So we'd like to be supportive of the standards coming rather than yesterday's news. But the question is, how big an application surface do we have to support in order to get traction? And we'll find out, or we are finding out as we speak.

Okay, I think your point about Heisenbergs is really interesting because certainly as a user of web applications, I've often had the experience where something's gone wrong. You report the bug and then the response back is, well, I can't reproduce. This is maybe a temporary glitch. And then from the engineer side they're like, well, how do I fix this if I can't reproduce this? So the idea that those categories of temporary problems are going to largely go away because you've got that state, that just seemed like a huge step forward.

Well, database systems have this problem in spades. I mean, I worked for a bunch of different database companies. If you have a Heisen bug, then the user sends you the bug and you can't reproduce it. If the customer is important enough, you put engineers on airplanes and go to the customer site and put a print statement everywhere in sight. And so they're fiendishly difficult and fiendishly expensive. And if the Customer is important enough, you're likely to land on the front page of the New York Times. So this is serious business and so we make things a lot better.

Going back to what you said about how because everything's sort of SQL internally, you can start doing queries on it. What sort of queries might I want to run against an operating system who.

In my environment is using more than 100 gigabytes of space, counting only those files bigger than 1 gigabyte? You can't ask that now. And it's just SQL in our system which three users are chewing up the most resources? And is there anybody who has copied more than 20 files in the last 12 hours? Just, et cetera, et cetera, et cetera.

From the application developer point of view, can you also do debugging by writing SQL queries?

Yeah, of course, but we also give you this time travel debugger. But sure, you can write SQL against what amounts to the event log in the data warehouse. And so, yeah, that works great.

Okay, so it sounds like if you've got a few SQL skills, then it's going to make a lot of what's happening within the operating system much more accessible. I'd also like to talk a bit about cybersecurity. So you mentioned this before, that security is one of the main features and that you can prevent these ransomware attacks. Are there any other security benefits from running dbos as opposed to another operating system?

Well, there's a much smaller attack surface compared to traditional world, so there's a lot less gates to close. And the current world is you start with Linux a leaky boat and you paper over that with a bunch of stuff. And in our opinion that gets you another leaky boat. And so the way to make security better is to have a much simpler system with a lot less attack surface. And that's exactly what we do. And so there's much less attack service. You can do much, much better monitoring and you can get up from ransomware attacks easily. That's our main security story.

Okay, Just having a simpler system means less things can go wrong. That seems useful to know. I'd also like to ask you a bit about, like, what it's like to create a startup, because often creating a system startup is very much seen as a young person's game. And, well, you've been around for a while now, so could you tell me what inspired you to create another startup?

So my career has been in academia, and most people in academia want to get famous and write papers and whether or not they do anything meaningful to the real world is irrelevant. And somehow early on, I decided that it's important to try and make change. And I learned a long while ago that the very big companies don't invest in new stuff. They by and large buy startups after they're somewhat successful. So the way to make change happen is to do startups. And so whenever I've had an idea that looked like it was commercializable to make a difference, you do a startup and after a while they get easier and easier for someone like me to do. And nobody has yet complained about my age, which everybody should complain about since I'm old. But I'm going to keep doing this as long as I can make a difference.

That's wonderful. It's very inspiring that. Keep plugging away at this and keep coming up with ideas and creating stuff. All right, can you tell me a bit about what you're working on with DBOSS right now? What's coming soon?

Okay, so we've talked about sort of what the commercial guys are thinking about, which is getting the first Lighthouse customers, making them happy, and doing whatever they need to be successful. So that's exactly what you would expect. The academic research on dboss goes on, so that hasn't stopped. The thing that I'm most interested in is that if you look at database transactions, the high pole in the tent as to what consumes CPU time, it looks like it's pretty much moved to being the networking system. And so to go faster, you've got to redo the networking system. And so we're looking at all kinds of ways to go fast to send messages faster than inserting them into a database table and then reading them back out again. So we're working on high poles in the current transactional database stack. That's something that I'm very, very interested in. And then another thing is everybody on the planet is dabbling in large language models, as am I. And the question of the day is, what can large language models do for structured data in database systems? So I'm plugging away at that. So that's what I'm focused on in an academic context.

Okay, Faster networking and large language models. Exciting stuff. All right, so do you have any final advice for people who are interested in using dboss? Sure.

Get at works. It works and works well, and it has a ton of advantages. So go to our website and kick the tires and you can sign up to freely use the cloud version software as a service version. So we're excited to try and get feedback and it costs nothing except small amounts of your time. So. So have at it and tell us what you think.

All right, Super. Everyone get out of here. You've got a call to action there, audience. All right, thank you very much for your time, Mike. That was great.

Okay, thank you, Richie.