152: The Future Database with Sam Lambert

Programming Throwdown, episode 152, The Future Database with Sam Lambert. Take it away, Jason. Hey everybody. This is a super awesome episode. I'm really looking forward to this. I actually found out about PlanetScale, which we'll talk about later on, from Googling on the internet, you know, how to get a self-managed database. I, you know, burned myself too many times with trying to maintain my own database. And another thing I always tell folks is the class I kind of most regret not taking is databases. As people who've listened to the show for a while, you know, show veterans know, I elected to take all the theory classes. I took linear algebra, I took all these things, and I didn't take networking or databases or operating systems or any of these classes that probably would have been extremely useful. And I feel like I learned most of them on the way, but the one that I really learned way too late in life was databases. So We are going to talk about, you know, kind of the next evolution of databases and, you know, kind of along the way, we'll dive into kind of the whole history of how people stored things. And we'll dive into a lot of that and how we can continue to make that better and more painless for engineers. So I'm really excited that we have Sam Lambert on the show. He's the CEO of PlanetScale, and he's here to chat with us about the future of DBs. So. Thanks for coming on the show, Sam.

Thank you very much for having me.

Cool. So maybe just to kind of kick things off here, what is one of your greatest database horror stories? You know, everyone talks about the intern that deletes the prod database and stuff like that. Do you have any story from your experience of a crazy database mishap? You know, something you couldn't even roll back or something wild that you could share with the audience out there?

I have so many.

All right, let's do it.

This is partly why I'm doing what I— I've got 2 actually. One inspired a product feature that we have, and we don't need to go into too much about what PlanetScale does right now, but one was in my second day at GitHub, I took GitHub offline for about an hour while making a database change, and it was the worst feeling of my life, and I honestly expected to get fired that day. It was a very common issue that a lot of our customers face, well, did face until PlantScale. And it was a real passion for me to fix this for our users. But what happened was, GitHub was a Rails app, very large Rails app with a lot of users and lots going on. And one of our developers had been waiting for me to join. I was the first database engineer that actually joined the company, and they had a load of schema changes that they wanted to make. And they said, we've stopped using this model. I want to get rid of a load of these columns, and nothing's accessing these columns. So I went ahead and ran pt-online-schema-change, in those days was the kind of way to run an online database migration. And kind of it went through, it took about half an hour to do it, and then the column dropped and the website went offline. And what happened is the model hadn't been fully cleaned up and there was tables that were still used— sorry, there was queries that were still using that column and every single site on the— every page on the site just completely broke. It was a horrible feeling. And with our audience, people noticed extremely quickly because your application needs to pull from GitHub to build or to go onto the cloud or whatever. So everyone would notice, right? In fact, in our load graphs, you could see the cron ticks of the entire world. So on the hour, we had a huge spike. And you would see these spikes throughout the entire stack. If you looked at the load balancers, if you looked at the application, if you If you looked at the database, every graph had exactly the same spike. So the hour was the biggest, half an hour second biggest, 15 minutes, down to 5, down to 1. And it was people's applications, they had a cron and they were just building the app whenever to deploy it or whatever, or just to build it and build an artifact. And so that was one of those. The other, the funniest, and I can only be funny because of how crazy it was, was I was out at dinner, I was in Berlin with some GitHubbers, and I got paged, website's off, MySQL down. So I ran, it wasn't far from where we were staying, so I ran back to this Airbnb, jumped on the computer, got on the box, and it said MySQL clean shutdown or shutdown initiated or whatever. And I was like, someone has logged on and shut down our main MySQL server. And I was like, what the hell's going on here? This is crazy. Obviously no one had done that. There was a really weird issue. So there was another Percona tool called ptstork. And what that did was it would say something happened in the database, like a load spike or whatever, it would try and grab as many metrics from the database as possible, like what queries were running, the InnoDB metrics, just so that you could debug a bit when things went wrong. It was just a really good way of capturing a snapshot of what was going on in the system. It was also quite heavy. We used to run pretty sort of stacked database machines, like really big servers, so a lot of memory, and scanning buffer pools that are that size is very, very slow and was actually slowing the database server down. So earlier that day, I'd gone into Puppet to just disable pt_stalk, which, you know, next time Puppet ran, it would just go to the database servers and shut down pt_stalk. Great. So it did that on the first Puppet run, that was fine. Then we had this incident. What happened is there was a bug in ptstalk that it hadn't cleaned up its PID file. And so when Puppet went to disable it, it found the PID file and went to kill again to shut down ptstalk. What happened is a thread of MySQL had assumed that number, and MySQL forwarded on the SIGTERM to the master process and processes it as a shutdown request and just Gracefully shut itself down.

Oh, the entire database. Uh-huh.

So it was the weirdest issue I've ever, like, it just should not have happened. It was like 2 bugs piled on top of each other and then just a MySQL shutdown. And it was very like annoying at the time, but then we all were able to laugh because like, I mean, how do you see those things coming?

Yeah. What are the chances of that?

Yeah, exactly. Exactly. You just got to kind of roll with it.

That is wild. So you said you joined GitHub and then within a couple of days, You broke the site and you thought you were going to get fired. This is something that, that, uh, you know, a lot of people wonder about this. So I'm assuming you didn't get fired because you have a second GitHub story, but what actually happened? Like, you know, do you get called into the boss's office? Like, what was the day after like?

Everyone there were great engineers that understand that these things kind of happen. Like, we looked at it, made sure the problem couldn't happen again. and went back to it. Okay. I mean, we didn't have a very blame-based culture because what's the point? If you're looking for blame, you're never going to find the real problems. It's only systemic problems.

Yeah.

If you assume, and this was the case in both the circumstances, is you assume best intention and you assume that the person does the right thing with the information presented to them. So the developer, was presented with the information. They thought they'd moved, cleaned up the code base, and they had no other reason to believe they hadn't. So they made the next step with best intentions and with the information they have, which was to ask me to remove those columns. I probably should have checked, but at that time I didn't. I kind of just went forward with my knowledge, which was like, okay, I believe that this thing is done. I'm going to go and do this. And I went and did it. And so when you look back at this, if you just go for blame, you miss what could actually stop this happening in the future, which is like process, right? So after that, I wrote a checker that makes sure that no queries have happened in that amount of time since the model cleanup. So we knew then we were going to cause kind of an outage if we still removed those columns. And that's how we actually improved. And throughout my career, I've always encouraged people to do that kind of postmortem, because if you're just going for blame, you very— I've never seen an outage where in the postmortem someone was presented with the right information and then deliberately did the wrong thing. People just don't really do that. Yeah. There's a lot of interesting safety science around airplane crashes and how just giving pilots a checklist to follow in a very stressful situation improved airline safety massively, because it's that same insight of people are just going to do what they think is the best with the information they can gather at the time. And gathering information in a high-stress environment is very difficult.

Yeah, that makes sense. Yeah, I kind of have a downstream story of something like that where I was making changes to TensorFlow. This was maybe 6 or 8 months ago. I was forking TensorFlow, for a bunch of reasons and adding a whole bunch of proprietary operators and things into TensorFlow. And I accidentally put in the command to push my branch to the open source TensorFlow. And I got this message pop up and it says, hey, you are not pushing to the internal TensorFlow. You're pushing to the open source TensorFlow. Are you sure you want to do this? And you could press, and it was no by default, which is super smart. And so I just pressed no and we moved on. And that later on, about a month after that event, someone was in our company all hands talking about how they pushed something to TensorFlow and having a laugh about it. But that error, failure review board, or like the contingency of that error was to implement this extra check, which You know, save me from doing it and a whole bunch of other people too. And so you're right. I think instead of wasting time, like trying to blame, it's really an opportunity to fix, you know, a whole category of errors.

You can understand a lot more.

Yeah, totally. Very cool. So yeah, let's, uh, let's dive into, into you for a bit so the audience can kind of get to know, uh, more about you and your experience with, uh, with databases at kind of led to PlanetScale. So kind of walk us through, like, you know, what was your first time using a database? You know, what's kind of your kind of history with that? And at what point did you say to yourself, now's the time to go and start PlanetScale?

My first kind of interaction was with databases. I was at this company that was an e-commerce store provider. It was like more of a boutique. They did it for very fat, like fashion companies, and it was way before Shopify and the likes. And so it was a house-rolled e-commerce framework, and they had an incredible design team, and they would produce beautiful e-commerce websites, right? And really big brands would come to gain e-commerce sites. And this was where— oh, when would this be? Probably 2006-ish. So it was a long time ago.

Mm-hmm.

And it was PHP and MySQL. which is still the majority of the internet, and at that time was definitely the majority of the internet. When I was a sysadmin, I was there to just keep websites up, put websites into production. We had very basic tooling for doing this because back then, and MySQL was the database. And so I had to administrate database servers with phpMyAdmin and things like that. And that was my kind of first introduction to MySQL. And after I left that company, I went to an electric vehicle company that's now not in business anymore, but they were an early electric vehicle company doing commercial electric vehicles. So they were building these big electric trucks and Frito-Lay and people, these companies doing these local area deliveries were using these electric trucks and they were very fun. Electric vehicles, regardless of their size, still accelerate very, very quickly. And it is very fun to floor a 7-ton truck that's got a massive electric engine.

That's amazing.

So we got to do that as well. That was very Very enjoyable. But they had this huge telemetry system and they were actually selling the data to the Department of Energy because they were fascinated by the usage patterns of this very new emerging technology. So we built this device that went on the box and listened to the CAN bus. So in every car, there is a kind of a network called the CAN network, which all of the devices in the car communicate on and kind of diagnostics is sent around the vehicle, things that if you're getting a warning light or whatever. And we would listen to all of it and it would generate thousands of messages a second. We had a SIM card and we would stream this up to our data centers and we would gain all this data in real time. And we stored all that data in a massive MySQL warehouse, just huge volumes of data coming in and having to be stored. And that is when I really got exposure to the internals of MySQL because we were running it in such an extreme fashion. We would hit performance issues or bugs, and that's when I really got into databases and learned how fascinating they are and what a challenge they are. And that's where I kind of moved my career from just being like a general sysadmin to specializing in databases, and it was really great fun. And I eventually made my way to GitHub where I did the same. But I've had this observation throughout my whole career, The databases are too difficult. They solve a really hard problem for you and then pass on a whole other ton of really hard problems for you to solve.

Like what? What are some examples?

Well, I mean, look at how difficult it is just to get a schema change into production in an online fashion. Like, I've got a horror story about that. Everyone has horror stories. Every day you see someone fills out a contact us report on our website and they're having problems with schema changes or a schema change took them down in production. And it's like a really hard thing to do. So when we built PlanetScale, that's one of the first problems we looked to solve, making schema changes fully online and easy enough to deploy as if you were deploying code. And people love it. And we are very productive using the platform. We get more schema changes done than I've ever seen any other company, like multiple a day. Developers just push them. The normal process in every other company, and it's a process I've set up because the database is so fragile, has to be babysat, is you You'd open a ticket, the DBAs would look at it, they'd test it, they'd run it on a staging environment, and you're waiting days. And how often do you like waiting days to get the feedback of getting something in production?

Yeah, never.

Yeah, no, you just don't. It just ruins your time. GitHub had this magical culture, an incredible culture focused on shipping. You had to ship something in your first day. You know when you hit T on a GitHub issue and you get that Really fast— not issue, sorry, on a GitHub repo, and you get a super fast search.

Yeah.

Yep. That was someone's first day project when they joined the company. They shipped that by the evening.

Wow. The go-to-file thing, right?

That was our culture, was that it was all about shipping and user impact and getting the value of what we're building out there in front of people. So I had this challenge of looking after my SQL databases and having an internal audience that wants to ship really, really quickly. It led me to build a lot of tooling and learn a lot. And so PlanetScale is— this is reflected in our product values, which is we want to give you an extremely stable database. And the backend of our database has run some of the largest websites in the world and continues to do so. But we want you to manipulate and ship as fast as possible. And that's a really hard challenge. There's loads of baby databases, toy databases out there that are like, we are the best for DX and whatever. It's like, But yeah, that's great. I mean, yeah, you can build against this thing super fast. There's no schema. Really good. Good for you. You can ship super quickly. And then you pay that debt down terribly when you lose data, you ruin a customer's experience, all of those things, right? Like that trade-off, traditionally, it's been an either-or. You want a super fast, easy-to-use database that is just unable to do what databases do at scale, or you have a database that scales and is robust with data. And it's painfully hard to use and really hard to be productive with. PlanetScale bridges that gap. And that's something that really excites me.

Yeah, that makes sense. Yeah. This is a problem that I've also seen a lot with databases. I think it's because you can't do anything atomically, because the database is sitting as the engine for a massive dataset. You can't say, well, I'm going to push this PR. And this PR is going to, you know, update the code so that my column now says foobar instead of foo, and also update the database schema to say foobar at the same time. Like you can't, you just can't do atomic things like that traditionally because you have a, you know, zillion bytes of data that all are expecting column foo.

You also have open transactions. If there's an open transaction that remembers that column not being foo, Right, right. Do you clobber that transaction or do you clobber the ones afterwards? It's a very hard problem. That's one of the reasons that online schema changes are quite difficult in databases because of that issue. One very common MySQL issue was that you would have an open transaction so that it has a lock on the definition of the schema, because if you change that during a transaction, you are going to mess the transaction up and transactions have to guarantee repeatable reads throughout the transaction. So if you change the schema, you could break the running transactions. So that transaction has a lock to prevent that. But the transaction that's trying to do the schema change goes and grabs a load of other locks and waits for the lock on the table. But while it holds the other locks, it's then destroying other transactions that are trying to come in. So you get this pileup of long-running transactions that then break your website. So there's all of these really tough issues about doing this. And luckily, we're moving forward, tech is going forward, things are getting better, and this is getting close to a solved issue, but it's not easy. And I think the hardest thing, and the thing I like the least about building a database product is you have to just push loads of trade-offs back to the user. And there's no way of being too magic because if you are too magic, you're going to let people down and you're going to let people paint themselves into really awkward situations. And so you have to pass over a load of choice and then you have to educate people and that's difficult and it's not simple.

Yep. Yep. Yeah, that makes sense. I think it'd be really good for the audience to explain kind of what a transaction is. And yeah, maybe do you want to take a crack at that?

I'm sure I'll miss something, but here we go. Yeah.

Well, between the two of us, we can fumble our way through it.

Let's talk about the— I think it would be easy to frame it in why you would want to use one, and then we'll talk about why it would be achieved. I think that's the easier way of doing it. So the reason you'd want a database transaction would be in the following scenarios. Let's pretend you're a bank and you store your data in MySQL, which a lot of banks do because of transactions and because of the durability of the database. Imagine I wanted to move my money from my bank account to your bank account, I've got to go and delete the data from, like, decrease the balance in my account and increase the balance in your account. And as we know, computers crash. If a computer crashes midway through that process, after it's deleted from mine and hasn't quite put the insert into your account, the data, the money disappears and has not gone anywhere, right? It's just gone.

Yep.

So the delete, the kind of update has happened on my table, well, on my row, and has not yet happened on yours, and we crash. All that's happened is my balance has gone down. So an ideal way is that we stage them both to happen, and they don't happen unless they all happen, right? Unless the data gets into your account and out of mine, neither actually completes. That is why we have transactions. You do this inside a transaction. So you would begin a transaction that tells the database, I'm going to do a series of commands, and I want all of them to happen or none at all. And then you would issue those commands. So you'd begin this transaction, you would then remove the balance from mine, you'd insert it into yours, and then you would close the transaction. If all of those operations are able to complete, they all happen. If one fails, they roll back, and none of them happens. That is probably the simplest explanation that you then, you can peel the onion a little deeper and go into why you need repeatable reads and all of these things. But that is the simplest I can explain transactions. Yeah.

Is there ever a case where it's not possible to roll back? Like I'm thinking, uh, let's say like, is there some kind of, I guess to, to, to further your analogy, it'd be some kind of double spending thing where, let me think about this, like to roll back, you'd have to, you'd have to put money back in your account. Is there something that could have changed externally to make that not happen? I guess I'm wondering, can transactions somehow still not fully protect you? Can the rollback fail?

Well, if you have a locking problem, potentially. Oh, I forgot to talk about locks. Here we go. This is why. For a Monday morning, we're going deep, but we should. Right. So when, when the thing the database should do for you when you remove the data from my row is lock it to say, there's something going on here, don't do anything else. And that stops another transaction coming in and say, I'm doing it again. It stops me removing even more from my balance and then making the transaction fail. So it locks that row at that same time. So when you do the remove the balance from me, it locks the row to say, no, like there's something going on right now. No other transactions can go and add or remove balance from me. And that stops that happen. That means you can enable that rollback. If you didn't allow that to happen, if you didn't lock the row, you could have another transaction remove all my balance, and then I don't have any balance to transfer to you. And then we have another problem.

Yeah. Yeah, that makes sense. Um, yeah, or you could, I think maybe, uh, Like if for some reason people had a limit on how much money they could put in their account, then you could imagine like, let's say you're right at that limit. Let's say the limit's $10,000 and you have $10,000 in your account. You elect to pay somebody $1,000. So it debits $1,000. Now you have $9,000. Another transaction puts in $1,000. So now you're at $10,000 again. Then that first one needs to roll back and give you some money back from a failed transaction.

Yeah.

which would put you at 11,000, which violates some, some rule. So it's like you end up in really bizarre things like that, I think.

Yes. And, and the key point is it wouldn't be putting money back.

Because of the lock.

Yes, because the lock, it would stop the, the removal just wouldn't happen until the insert happened as well.

Yeah, that makes sense. So, so yeah, what ends up happening, what I've seen a lot in code is where, you know, somebody You know, there's a column in a database and someone tries to sort of rename the column or put a note or something that basically says, don't use this column. Like this is our busted, you know, our old column. You know, it's an integer and we decided we really need a double here. And so we're just going to make a new column and then slowly get everyone over to the new column. And then when you finally feel like no one's using the old column, which is also really hard because there could be things deployed using old code that aren't, you know, updating every day. So you can have old code running somewhere in your company. But if you could somehow, I guess, looking at the database logs, say, okay, no one's accessed this column in 90 days. So I'm confident enough now I can finally delete this column. And so it's like something that you wanted to just do overnight, some atomic change, it's actually going to take 90 days. And that's even for the most trivial thing.

Yeah. That's really tough. You even get scenarios where people's deployment infrastructure just doesn't work as well, and there's always one pod or something that's got an old version of the code that's asking it. It's really, really tough. We get around this by allowing you to push these things online, but it might still break something.

That makes sense. So one thing that we should cover a little bit before we come back to MySQL is all the different types of databases, right? There's Sort of, um, you know, these sort of common column— columnar— columnar. How do you say that? Columnar.

Columnar. I always get— I'm so bad with some of these words. Like, yeah, I could never say that right.

Patrick, you have to give it a shot. I feel like this is a word that you can get right. Columnar.

Yeah, that sounds right.

That might be the closest. I don't know. But anyways, so that's, that's those ones that look like, you know, if you've opened up Excel or something, you have this sort of sheet, you know, and you can put little titles in the first row of all the columns. That's kind of view of a table. That's the sort of canonical database. But then there's, you know, there's key value. There's, there's these document stores like Mongo and, and I'm totally drawing a blank on some of the alternatives to Mongo, but, you know, what's your feeling about all of this? I mean, is, is there, you know, a new application is presented in front of you. Do you have— like a process that you go through and you try and say, okay, this is the right sort of way of storing this data. Like I've, my initial feeling on this is I used to spend a lot of time on this and now I've just decided I can do everything in MySQL or at least in that format. But I was wondering what your take was on that.

I mean, it's so funny you say that because that's kind of the curve, right? Like best tool for the job. It's good to try and find the best tool for the job. However, it should really be the tool that mostly fits your use case that you can operate really, really well. Because if you're familiar and you can operate these things and kind of get it done, you're likely going to have a better experience in the long run. And there's this kind of funny progression that companies go through. It's like, you know, maybe MySQL was good in the beginning and then they moved to Mongo and then they always end up back at MySQL. The truth is that massive scale— You usually end up back at MySQL. So there's loads of graph databases out there, right? Like that do, you know, storing graphs. Great. The largest graph database in the world is at Facebook, TAO, and that runs on MySQL because they have a massive MySQL deployment. They are incredibly good at operating MySQL. And at the end of the day, if you can build a layer on top to store graph data, that is way better to operate than a brand new graph database that's still figuring things out, right? Like if you have got your operations down on a specific database, then that's the way to do it. And even key-value, we removed loads of Redis at GitHub and replaced it with MySQL because MySQL's faster, because it turns out multiple threads are quite good for performance. And so it really, it's a really nuanced conversation. Like Mongo is great for some use cases, There's other data stores that are great for other things. It's just how you apply these things, and trying to go for an overspecialized stack all the time. Like, I speak to some people, and it's like, we've got 7 databases that we operate. It's like, wow, you don't have 7 teams of database experts. You've got the one people mostly know, the semi-abandoned ones that cause outages all the time, and it's a real mess. I mean, Do you really want the bells and whistles features of everything? It's like, you don't buy a new car when you move house. You either rent one or you just kind of make it work. You just pile everything you can into your Prius and then just go, right? And that's often the right way to do it. You can't— overspecializing in your tech stack leads to horrific maintenance problems. And that's why every big tech company unifies in some way with a form of platform. And it'd be best for most companies to probably do the same.

Yep. Yep. Totally agree. Yeah. Um, I know that the, at one point the TAO team looked at, um, a document store and what they found was the P50 was a little bit better, which, you know, at Facebook means a lot of money, right? If you can get a little bit better, you can save, you know, millions of dollars, the equivalent of like 1 hour of Facebook's delta on the stock exchange, you know. You could save billions of dollars, but the P99 brought down the whole thing. So, so basically, you know, MySQL, at least the way that it was implemented and used there, you know, it was a little bit slower on average, like, you know, the transaction, each transaction was a little bit slower, but when things went wrong, you know, it gave you a lot more lead time and it could just handle things at the limit that we just didn't see from any other product.

Yeah. Yeah. Too many people evaluate the database from the point of view of querying it, which is of course very important. That's the thing your application is going to do a lot. But how does it fail over? How does it replicate data? How does it handle crashes? Those things really matter because no one— when you drop a customer's data, the customer very quickly stops caring how easy to use the database was for your developers. In fact, they don't give a damn.

Yeah.

They just want a reliable service that is up. So with our backend, it has experienced failovers hundreds of thousands of times at other companies that are not us and was built at massive companies with scale problems and extremely smart engineers. That's what you want to leverage. I promise you, if you're listening to this and you think, no, I still want the really fancy features, I can handle the rest. Ask yourself that at 2 in the morning when you cannot recover your only top customer's data or whatever. All the regrets come back, and too many people are bitten by this in the database world. It just— businesses have gone bust, very successful companies have gone bust because they did not have an ability to recover their data. And I've never seen MySQL lose data in a crash. ever, unless the operating system was lying to it.

Yeah. And that's also the developer experience and sort of the ubiquity of MySQL and Postgres and these other tools is unprecedented. I had this really dumb idea where actually, okay, it wasn't dumb. It was, I wanted to push myself and And so I thought I'll use a graph database for something that was, you know, not a really heavy hitter in terms of performance, but this would be opportunity for me to learn about graph databases. And it's kind of like, it was kind of like using, you know, OS 9, you know, where like you get error code -13 and you have to go look up what that is. And it's like, oh, you know, they're the, this, uh, you know, C++ isn't supported because they just targeted, you know, Python and And, and JavaScript. So, so now I have to use their REST API, and I'm trying to make REST calls in C++. And finally, I was like, this is, even as, you know, a learning opportunity, this is just not productive. So it's like, okay, I did the same thing in MySQL in like 2 hours.

Exactly. And I'm not saying there's no room for innovation. I'm not saying people should not be disregarding all of those things to try new things and try new ways to store and and query data, but it's a long evolution. The Postgres JSON columns is a really good one. Really good story. So this was when Postgres was a lot less operable. The origins of Postgres, it's an academic project basically.

Oh, I didn't know that.

Yeah, I don't know it fully, but I think that's where it came out of. I think it was, please, I'm sorry if you're listening to this and this is incredibly wrong and you're driving your car and getting really mad, but I believe it came from an academic project project, which was more focused on teaching people SQL and was very good at an implementation of pure SQL. It took a very long time for Postgres to gain some of the necessary operational primitives like replication, and it's still not great. And so they shipped JSON columns, which is brilliant, very useful. The developer community went crazy for it immediately, and they were like, MySQL doesn't have this, MySQL sucks. And so the Facebook team, again, Facebook, looked at that and thought, I like that idea. We would like JSON columns too. So they started working on it and they shipped it at the scale of Facebook. So what, 3 billion daily active users or whatever?

Mm-hmm.

And it worked great. When they shipped it in Postgres, it was really not production ready. People had horrendous problems. And the wisdom was don't use this in production. 2 years later, they were both production ready. One had been designed by Facebook and run at Facebook scale. The other had let users down for 2 years and finally got good. So the timeline was exactly the same, right? But it was the order in which these were shipped and tested, and the robustness of the implementation was what was really different about the 2 databases. And that talks about it philosophically. You can get all these hyper-cool features, they're amazing, like awesome. And there's famous document stores out there that were Amazingly revolutionary. And then there's a million stories about how they lose the data. You can get this stuff if you want it, but you have to just be careful because the downside of this newness is not good. You don't want a new database. Databases take a minimum of a decade to mature.

Yep. Yep. That makes sense. Yeah. The other document store I was thinking of was, I think, Orbit, OrbitDB. But yeah, totally right. And there are a number of really good plugins. So for example, if you want to do vector search, you know, there's vector search plugins, there's GIS, you know, if you want to do spatial, there's a Postgres GIS, there's a spatial plugin for that. You know, one thing that when I was trying to pick a database for the first time, at the time what they were saying was basically Postgres is truly open source, MySQL, like at some point they'll come bothering you for money or something like that. I mean, that might be really dated, but what's the deal with MySQL? Is it completely free to use? Is there some gotcha there?

No, it's never happened. People saw the purchase of, well, they saw Oracle buying it and they thought, oh my goodness, this is the end for it. Oracle have been good stewards. They know how databases work. InnoDB was built out, had a lot of origins with Oracle. They're very good at databases and they've kept it going. They're good stewards. I mean, there's commercial interest. When I was at Facebook, the whole Oracle PM team used to— no, not at Facebook, sorry. I wasn't at Facebook. When I was at GitHub, they would show up and listen to their customers. And they have some of, most of the largest websites in the world as a user base, and users that are not paying them any money, but they still go there to make it better. And MySQL gets patches and input from all of these large companies running at scale. So it gets better and better. So it's a healthy, good open source project with some branding issues.

Okay. That makes sense. Yeah. I mean, you're right. This was around the time. I mean, this is probably like 10 or 15 years ago and there's all this kind of drama. This was around the time when Android, we weren't sure if Android could keep using Java and all of that. But, you know, I mean, looking back on it now, I mean, yeah, everyone could use Java. Everyone could use MySQL. There's really no issue. It was a lot of, uh, it was kind of a scare for, for nothing really.

Correct. Yeah. It's great that these things are open source and that people can use them and they get contributions from folks that have got great ideas and great skills.

Yeah, totally. So, so you were, um, did you go straight from GitHub to PlanetScale or was there something in between?

I had a tour of duty at Facebook for a little while, which was really fun to see that level of scale.

Okay.

I just wanted to have incredible respect for the Facebook engineering team, and I think it's the best at-scale engineering team that's out there. They have so many phenomenally talented people and are also led by incredibly talented people. You have VPs that have 10,000 people in their org and they can talk to you about the intricate technical details of what their teams are working on. And it was just an incredible experience to be there and witness that.

Yeah, totally. Yeah, I could totally double, uh, double-click on that. Okay. So, so from Facebook, when you were at Facebook, at some point the spark ignited and you said, I'm going to, you know, quit my day job and start, start PlanetScale. What was that like? Uh, how did you take that leap of faith? And what were those kind of moments like?

So there's some nuance there, actually. I joined PlanetScale as the Chief Product Officer. I didn't actually start the company.

Oh, okay. Got it.

I'll tell you the story actually of PlanetScale. So the founders left YouTube where they built Vitess, which is our backend technology that PlanetScale is powered by. So Vitess is a layer on top of MySQL. It's an orchestration and sharding scheme for MySQL. that allows you to take MySQL to giant scale. So YouTube was growing, scaling, and building, and they needed a database that worked. Obviously MySQL was the choice, and they needed some sort of layer to shard MySQL on top. So they built Vitess. Vitess was built on Borg, the predecessor to Kubernetes. So it was an environment of pure impermanence. You don't get your disks back. You don't get to go and recover a server, it's gone. So they had to build an incredibly resilient system for orchestrating MySQL. And that's really hard, really hard to run state on stateless computers. They just fail. And they achieved it, and they built this incredible technology. And we started using it at GitHub, because of course GitHub was a MySQL shop. We started using it, and we were just so impressed. It just does what it's supposed to do.

So yeah, could you dive into that? So sharding is, I guess, like a way to do multi-node. Yeah. Like what exactly does that mean? And what goes into that?

So horizontal scalability is really hard with databases. You can vertically scale. And we did that for years at GitHub. We would basically buy the kind of best Dell server that they had from that generation and beef up these machines. Eventually that just becomes impossible. You also become very write-bound. So in MySQL's kind of terminology, you have a leader and a follower. So you write to the leader and the followers all get the update. And so followers are just a great way of scaling reads. You can just put loads of them out there. The read traffic hits it, but you are writing to a single place. And that is where you start to see big bottlenecks if you're writing a lot of data, or you just store more data than can live on a single machine. And those replicas, the replicas of the primary just have to be an identical copy. They can't have a subset usually. I mean, they can, but you—

Right.

It's a lot of messing around. They just have to be a copy. So they all have to be the same type of machine. And once you have more data than fits reasonably on that machine, or more connections, or more writes, you're in trouble. Your database cluster is now oversaturated and you have real issues.

How do you keep it so that if someone reads something from a follower, how does the follower know that the data's fresh? How do they know that the leader has a better version of it?

So you do get some level of replication delay. There's always some. I mean, normally it's very small, but then this is another scale problem, right? If writes pile up, the replicas don't get them in time, and you have your application reading its own writes. So it does a write and then a read from the replica and it's not there, then you get a failure or you get an inconsistency issue. So you can ask the replica how up to date it is, essentially. But again, this is another problem, another scaling problem is that replication delay happens and it starts to get annoying. And then, so at GitHub, we built this kind of API that tells you which replicas are delayed. So excuse me, query the right one, or it's very tricky. Sharding comes in when it gives you the ability to split tables or databases across multiple machines horizontally. So imagine, let's just take the example of a big table. Every app has one of these tables, notifications, statuses, timeline, Whatever, right? There's always just one giant table that you just put loads of data in. You probably only need the recent data from it, but whatever. When that table gets too big for a single machine or a cluster of machines, you then want to start sharding, which means you split the table data into chunks and distribute them across multiple clusters of leaders and followers. So you could have, instead of 1 leader and 5 replicas, you could have 5 leaders with a replica each, and you've distributed that workload across more servers and more leaders. So you get more write throughput and you kind of can keep scaling horizontally. Now orchestrating that is a lot harder, right? How do you tell the query where to go? Well, you don't even tell the query. How do you route the query to the right place, aggregate them, joins, all of these things are really tough.

So as the engineer, do you have to provide a sharding function?

So not every, like as the person designing the scheme, yes, you have to explain how you need your data sharded. That can be very simple or quite complicated based on your needs, but someone has to make that decision and they make it at the beginning. You can reshard and change, you can change the sharding scheme, but You kind of pick it up front. It's not too difficult to do. And then with Vitesse or PlanetScale, it's then transparent to the application. The app doesn't care.

Got it. Okay. That makes sense.

So the YouTube app had like one connection string. It thought it was talking to one database. It was actually talking to 70,000 servers across— 70,000 nodes across 20 data centers. And the data was aggregated back for it. Very powerful, exceptionally powerful. And sharding comes with so many benefits. We've done a benchmark that's on our blog where we do a million queries per second sustained. And there's a graph, and this is the really impressive part, is how it's linear. If you double the amount of shards, you get double the amount of throughput. That is very hard to do as a database. I don't know of anyone else who's really achieved it, that level of predictability. is incredibly difficult. And so sharding gives you these really, really nice dynamics, and it gives you isolated failure as well. If you've got 50 shards and the leader fails in one of those shards and has a couple of seconds failover, only 1/50th of your user base experiences a blip instead of all of them all at once.

Right.

So there's so many benefits of kind of breaking this, like breaking the problem down, into these smaller chunks. And it has a lot of, you know, lots of benefits.

Yeah, that makes sense. I guess this comes back to— I'm still trying to wrap my head around the sharding function because I'm thinking if it's not done in a healthy way, then every query is going to need to read from every shard. But to your point, if you shard, if you put, you could probably put certain tables in certain shards. So if you're reading from a table, then You don't need to read from every shard for any one table. But yeah, I think it seems like it really depends on how you set up that database and then looking at saying like, okay, here are my most expensive queries and they're expensive because they're accessing every shard and then realizing, okay, the usage pattern isn't what I thought. Let me do this really expensive reshard operation once.

Yeah, you do have, so if you get the wrong sharding key, you may have to reshard. Some, often we have to help users design a sharding scheme that works for their queries. It's much better off if you locate data. So it's not as hard as it sounds because say it's users, you just locate the users' data together and you can shard multiple tables into the same cluster, right? So you do your joins and whatever within the cluster. If you design it poorly, you may have to aggregate across all shards, which isn't always slow. Not always. But it can be. But then there's other ways as well of materializing. Another thing that Vitess does extremely well is to allow you, like, to materialize a table elsewhere. So what that means is, imagine you've got this beautiful sharding scheme, 99% of queries just amazing, like go to the local shard, it's fantastic. This is one query, this one dashboard that needs to count up every like a user has done on the platform. Vitesse can, you tell Vitesse you wanna materialize where the result set of that query will be into another table, and it can do that on the fly. So then you issue that gnarly query against the perfect materialization for that query, and it works super fast. So you can get around that. It's not, not as difficult as it seems. Again, the building of that, that is some really hard tech. That is some really hard engineering to get that right, but it has been done. And that is a code path that's like 8 years old. It's been used by billions of users around the world as these large companies use Vitess. So there's other ways of getting around it.

Yeah, that makes sense. Well, that reminds me of a product that I've been using recently, which I'm really a big fan of called dbt. Where you, yeah, and please correct the record on this. I'm going to try my best as just a user of this to explain it. But the idea is you have queries and instead of just running the query interactively, you check this query into source control and it creates this destination table. And so you have 2 modes for dbt. You can either say, like run this query, you know, periodically or on a trigger or something like that. Or, uh, you run this query, let's say periodically, and generate this output table. So for example, I have this expensive query. I don't want to run the query every time a user hits F5 on their keyboard, so I'm going to run it once per hour, and then it's going to create this table. And then now when I go to the website and hit F5, it's just querying the result of that query instead of having to actually execute the query. And then they have this other mode, I guess it's like ephemeral or ghost mode or something where, you know, you query a table, the table doesn't actually exist, but when you query the table, it goes and executes that query. And that's more of like an aliasing thing where it doesn't save you any compute, but it's just easier for you to read the query than if it was some like massive nested thing.

Yeah.

I think dbt has been really, really impressive. I'd never even heard of it until a few months ago, but I've been a big fan.

Yeah, that is exceptionally cool. Now, I don't know if you tell your users this, but we're recording this a little bit ahead of time and PlanetScale has a big launch next week. And so hello from the past for anyone that we're talking to.

Yeah, that's right. If you're, if you're listening to this, this is maybe a month or two Later. So we're a month behind reality.

And if we keep shipping, maybe I'll be able to say hello to you from the future one day if we keep up with the time-based magic. But for now, we are bound by the known constraints of time and the universe. And what that means is next week we're shipping a new product. And there's lots of people, we're showing it and demoing it to people right now, and they're getting very, very excited. We've been tweeting their quotes, and they're very, very, very hyped by this. And it's in the realm of what you've just described. So what we're calling it is PlanetScale Boost. And what Boost does is allows you to choose a query that is accessing your database. You go into our insights panel and you say, oh, look at this query, it takes 5 seconds to execute. You would press a button to boost that query. And what happens when you boost that query? is that we tell Vitess the explain plan, the execution plan of that query, and we ask Vitess to materialize that query in memory all of the time. And we stream any updates, deletes, or inserts that that query— to the result set of that query, we stream it into memory. So you get an up-to-date, real-time version of that query in memory forever. So we see people, we and with the customers we've been testing it with, get thousands of percent improvements in their queries by just materializing it in memory. And this replaces caching logic, this replaces invalidation logic, it replaces running Redis, it replaces all of that stack and just gives you blazing fast, in-memory queries with the same consistency as a database read replica.

Wow. That is super cool. Yeah, that makes a ton of sense. This reminds me a little bit of, what was that? Scuba, right? Scuba was this thing that would load into memory. And I think someone, some Facebook folks left Facebook and started an open source version of Scuba. I don't remember the name of it. But yeah.

Scuba is an amazing metrics store that Facebook uses, and I think it uses similar technology, but this has never been applied to a database product before. The simplicity of just saying, make this query really fast all the time, and here's the amount of memory I want to allocate to do that, it's mind-blowing. So we always knew it'd be possible. So if you think about what's amazing about Vitesse, And you think about resharding, you just talked about resharding, right? When you have lots of disparate shards with copies of the data, how you stream data consistently between those shards is extremely important. Like, if you screw that up, you really mess your database up. And if you wanna reshard, for example, you may have 256 nodes, 256 shards, sorry, there's more than a node in each shard. And you want to reshard it, you have to fan that data back in and fan it back out in a new sharding scheme. And that's like a really hard thing to do. Machines die, you need to restart from the right place, network hiccups, all of this stuff that just makes that like quite a hard problem. That's a solved problem in Vitess. So that's called v-replication. And we built this on top of v-replication to say, you know what, you're not materializing to a table, you're not materializing to a node, you're materializing to an in-memory store. And keeping that up to date. So when I first saw this, when I first saw it actually working, when the team built it, I just burst out laughing because there was no other response than, this is actual magic.

Yeah.

So we're ahead of time in the sense that we're a week before it launching, but I cannot wait to see people's reactions when they just take that really horrible query that is their user's newsfeed, which is really slow, and they've tried to refactor it and it doesn't get any better. or they've broken it down into 4 queries and joined it in memory and it's flaky and buggy. For them just to choose that query and boost it, and then it's in memory forever and they've solved their problems, is going to be really awesome. And I just cannot wait for people to experience it.

Yeah, I mean, it reminds me a lot of things that we had to do at Facebook. There was this ML model that was very expensive, and what we ended up doing to make the latency was We actually executed it for every single user and then put it all into this, this in-memory key-value store internally called Laser, which is very similar to like Redis or these other ones. And that sounds kind of crazy, right? Because every day we're running this, that we're running this model a zillion times. Um, and the vast majority of those people won't even be on the site tomorrow. So it's just wasted. But, but it did guarantee the SLA that We needed to guarantee. So, so, um, so that's what, that's what we did. It's probably still in production. Yeah. And so what you're describing basically automates that. I mean, we had to write a bunch of PRs to do that. You know, like if we could have just clicked a button, that'd be awesome.

Yeah. Right. And that, that is why it's so like, there's companies that have achieved that level of like caching or speed, but they have, they put like a 20-person team of MIT grads on it to get it done. Not everyone has that.

Yeah. What's the connection here? So I kind of interrupted you with questions, but you were talking about some folks at YouTube started Vitess, which was either maybe a company or a product at some point that became PlanetScale, I guess. And then that, and then you joined. So yeah.

Yeah. So it was at, it was at YouTube. They had the same realization. This is incredible technology. Like they saw that Facebook had built their own MySQL sharding. They saw that Yahoo had built their own MySQL sharding. They saw that Etsy had built their own, like. Everyone had done this. It's just like an inevitability. If your company is growing massively, you should just sit there and internalize the fact you'll be running on sharded MySQL one day because it just happens. It comes for all of us. You can stop lying to yourself that you'll get Postgres to run to that scale. No one's ever done it. Right? One day maybe, but you can't— we're talking at a different scale. We're talking about hundreds of thousands of servers at some of these companies. There's public cloud level, like there's clusters at these large companies that are public cloud size and they have one use at that company, right? MySQL is there doing that at a lot of places. You can't fake it just because people like GIS or whatever, right? Like, I know that's a dig, but like when people say, when I ask people, why do you— here's a rant, but why do you like Postgres? Oh, it's like extensions. It's like none of that is going to work. And I know you might not have scale, so fine, I'm wrong, you're right. right. Okay? It works for you today. But if you want to succeed and you're building massively, I had a founder come and talk to me last week and they were like, we're scaling really, really quickly. Do you have any advice for us? And one of the things I said was, internalize the fact you're going to run on some sharded database and it's probably going to be MySQL. And they were like, yeah, that's what everyone's already told us. And that's kind of how it is. So anyway, they knew this. And they built a really awesome solution for this on a containerized system. Kubernetes then gets open sourced. If you look back at the history of Vitess, and Google has documentation on this, and it's also intertwined with the history of Go, Vitess is one of the oldest Go applications, certainly one of the largest. It was built on version 0.1. So our CTO gets credit in the Go docs for giving some of the most awesome feedback on Go.

Nice.

Because he was crazy enough in a wonderful way to say, why not build a massive sharding system to run YouTube and choose a language that is completely brand new? Well, he went and did it.

Rob Pike's eternally grateful, I'm sure.

Yeah, I mean, they genuinely were, right? So they built this amazing thing, they then donated it to the CNCF, and then they started PlanetScale as a company to kind of commercialize it. I and my team at GitHub had then experienced Vitesse, and so had Slack, so had Roblox, so had— all these companies just started using it. Suddenly, this tier of hyperscale companies were like, this is good. We should all try and use the same technology and contribute to it. And so it got even better and even mature, more mature, very, very quickly. And we know, and you know what, There's one website that's out there that's doing 32 million queries a second, serving billions of users using this technology. And when you can harness the downstream effects of that and say, oh yeah, they have taken this to an extreme scale, and all of the kinks have been worked out, it's just so powerful. So we started using it at GitHub, we all loved it. And I contacted the founders and said, this is amazing, I would like to invest in your company. And they came to the GitHub office, And they had this clarity of what they wanted to do, and they had the product. So they started building it and kind of getting it into people's hands. And now this is where Vitesse isn't as optimized. If you want to put a 7 or 8 person team and train them up on Vitesse and get them to run it in production, it will be absolutely fantastic for you. And if you are Slack, if you are JD.com or whoever, That's really good and really beneficial. And the cost trade-off, considering that if you're at one of these hyperscale companies, your database infrastructure probably costs you a couple of hundred million a year, hiring up a few, like hiring a couple of people to manage this really awesome open source project is a great trade-off rather than building it yourself. So a lot of people did this. And at GitHub, we did the same thing. We thought it was awesome. The database team at GitHub was phenomenal. And I trusted them. So I reached out to the founders, I asked to invest, and then I became an advisor in the company. And the thing that I was seeing was that just presenting Vitesse as is, even as a cloud host, was not enough to get it into people's hands. And I had been at GitHub for 8 years and I'd built user products, user-facing products, and GitHub Actions was the last thing that I worked on before I left GitHub. And I had this such passion for building products for developers. Because developers are amazing, creative, wonderful, and unreasonable. And if you can, in the most wonderful way, you can't bullshit them. You can't just like wave a brand on top of something, right? Like this is the issue that we're seeing now is there's an explosion of database companies and they're just building great UIs with like a jank backend. And people are like, oh, this UI is amazing. I'm going to use this. And it's like, cool. That's like, that's not going anywhere. Like you've, like, this is like, you're in trouble with that.

Yeah.

So that's kind of an issue. But I thought to myself, if we can get this incredibly powerful tech that is almost an inevitability when you get to scale, but actually make it the best thing to use on day one, this is going to be like a game-changing company. And so I said to the founders, I would love to come and join you on this mission. Bring me on and we'll build this product. So I joined, a bunch of people from GitHub came too, Like the guy that, like one of the earliest engineers, sorry, not engineers, designers, Jason, who has been designing developer tools for decades came. And you can see why PlanetScale is so beautiful. Like these people that understand developers and built this product that has endured in the eyes of developers for so long. Like GitHub has got 80 million developers using it. And they're happy and they enjoy it. So we took that as a design inspiration with the same people and said, how do we take this incredibly powerful tech and put it into everyone's hands so that you're not reasoning about the same things Facebook has to reason about? You're just using a database that feels like MySQL. We came up with database branching and the way that you should— we thought, why can't you just branch your database, use it for whatever environment, whether it's 10 minutes of testing, whether it's a 4-month-long feature development product? Why isn't there just an environment there that feels like production? And then when you want to change the schema, why can't you deploy it fully online like you're deploying code? We asked ourselves all these questions and took it on as a design problem and started building.

Yeah, this is actually, I wanted to ask you a little bit about the branching. So I've never seen anything like that. Actually, let me step back a little bit. So the way I found out about PlanetScale was I was doing a Side project and I typed in, I can't remember exactly the query, so paraphrase this, but basically I looked up, you know, free hosted SQL database, something like that. And I just wanted something really lightweight. I knew I wasn't going to have a lot of usage on day one. And so PlanetScale came up and I started using it. It's really fun. We should, now is probably a good time to talk about There's a free tier. Um, I'm still on the free tier with this side project, which I haven't had time to get back to, but completely free. And it's still up and running. It's been really nice. But, uh, the branching thing I never quite understood, like if I make a branch of a database and I start messing with the schema, what happens to the data? Is the data shared between the branches? Is each branch Have like start with no data? Like how exactly does that work?

Every branch is isolated. It's a different Vitess cluster. By default, it just gets the schema. If you want the data, you can have that too, and it will add data to the branch. This is a beginning of a journey, right? There's a lot more that will get a lot better about this in the future. But the idea is that you should be able to connect your application to a branch and it's isolated. Do you remember like Databases are scary. When we do user experience testing, when we interview people, the word fear comes up a ridiculous amount.

I bet.

People are terrified of their databases. So you have to really explain to a developer, that's why it's branching, right? Because they understand, we all understand if it's a Git branch and I push to a Git branch, I'm not pushing this into production. It's separate. So we wanted branches to be a playground where you get the power of the database that you use in production. And because the more complex these systems get, the better cloud tools get, the more abstract from, oh, it can run on your laptop, it is. It's unfortunate, but at the same time, it can be magical. Like at Facebook, in my second week, I pushed to facebook.com.

Hmm.

I can't build facebook.com on my laptop.

Right.

That would be crazy. It's millions of machines. So you have this isolate, you get a dev server, right? That is a slice. It's a powerful machine that is a slice of production. And then it carries, you takes your changes off to production through all the testing and whatever. That's awesome. That's the kind of the way we should go. If you look at Codespaces and all of these tools that allowed cloud development, we're getting towards that world. Where everything's cloud native, you can't simulate it on your laptop anymore. And so that's why you wanted branching in the cloud. And it gives you this isolated environment to play and make schema changes without breaking anything. And then when you want to get that schema change into production, you deploy it on PlanetScale through a deploy request, which is like a pull request that will then, no matter what you ran on your branch, it doesn't matter if you ran 500 different schema changes, All we do is look at the end state of the database and say, we will get that into production for you in the quickest, easiest way without any locking or blocking. So your application will experience no downtime, it will just deploy. And that is extremely powerful. So we have really large users that tell us, and we have case studies where they say, yeah, I mean, schema changes have gone down from 2 weeks to 2 hours. No one's scared of the database anymore. They just roll it into prod. And that was so important to us initially, to give people incredibly powerful technologies and the isolation and lack of foot guns to move extremely fast while using them.

Yeah, that makes sense. So then I guess when you make a branch, you should also have some kind of a program that will seed that branch with some seed data so that when you go to your— when you point your website, your dev website to the branch, it's not just empty. So you create some program that either copies some data out of master or out of the main branch or just seeds it with some synthetic data.

Yeah, what we found was a lot of people already have those seed scripts, And so it's kind of easy for them to just run them. People use actions to do this. This is all configurable on our command line. You can create branches, do whatever from there. So people often have like, okay, give me an environment. It creates a branch, it puts the data in, they test it, and it goes away when it's done. That sort of stuff is all very possible. Or you tell us which backup you want to add to a branch, and it just happens. Again, branching is a powerful primitive for many things.

Yeah.

When you restore a backup, you can choose to restore over your main branch, which you shouldn't really do, and there's not many people that need to completely roll their database back. So instead, you add a load of backups to different branches and sift through the ones you need and just restore the data that way. So it's just a really powerful way of giving people really cheap, easy environments and getting rid of staging. If you've got complex infrastructure, staging is just a second production that breaks every day and ruins the developers'—

Yeah.

Yep.

Yep. Yeah, totally true. I read a really interesting article about foreign keys, like, because Vitesse and PlanetScale don't support foreign keys. So you can't do like cascaded deletes and stuff like that. Kind of like tell the audience, you know, I mean, the article did a pretty good job, but kind of walk the audience through that because I, I remember when I first learned about MySQL, I thought foreign keys were amazing. I thought, oh, I can. have a credit card for a person, and when I delete the person, the credit card gets deleted. And so you kind of walk people through the dangers of foreign keys, or maybe just on the technical side, what's the limit there?

So we allow you to have foreign keys in the sense that you can have relationships, you can have joins, you can have all the things you need in a relational database. The thing we don't have right now, and it's something we know about and we're working on, is foreign key constraints. As they are implemented right now, They are unscalable and do not work with online schema changes. So it was, you know, we want people to default in things that are going to work for them in the long run. And foreign key constraints at this moment are not a scalable pattern.

So, so what does that mean actually? Because I didn't know the difference. What's a foreign key constraint versus a foreign key?

A foreign key constraint is when the database will automatically do that cascading for you.

Okay. Got it.

If you use an ORM, the ORM can handle it. Like dependent delete in Rails does exactly what you're talking about and isn't relying on the database to do it.

Oh, I see what you're saying. Okay. Okay. Now I think, let me see if I can paraphrase this. So if you have a foreign key constraint, like the one I gave is cascade delete, then when you just need to issue a delete command to database and database will effectively turn it into a transaction where it'll guarantee that it'll delete all of these dependencies. But you're saying you can do the same thing just at one level up in the abstraction and say, I'm going to create a transaction that deletes Jason and deletes all of Jason's credit cards and then closes the transaction. I have basically the same constraint now, just one layer up.

Correct. And a lot of people rely on this for their database to do it, or to check that the relationship is there, the other side, for example.

Mm-hmm.

Like, this is a functionality we do not support right now. Now, there's a chance we will in the future, but it also makes online schema changes not possible while you have those. Because the way online schema changes work is you create a copy of the table, you migrate the data into— you do the schema change and you migrate the data in. And then swapping those tables in place breaks the constraints that are already there. So you'd have to remove the constraints and re-add them, and it's tricky, right? Right? So we don't love that you can't have it, but it's very much within our philosophy to say, let's inform users and give them things, primitives to use that will serve them well in the long run, and maybe has a little bit of an upfront kind of extra complexity, or at least cost, but in the long run, you pay it down. Like, it's the same with these NoSQL document stores. They would tell developers, you don't have to think about a schema, about schema is a waste of your time. And then you throw a load of documents in and now you just have the most horrifically badly laid out data that is really hard to scale and use, and you have to do horrible hacky things to get around it. Actually thinking about a schema upfront is a good thing because it lays things out and it means that you have structure to your data and keeps it a little bit neater. And if it's easy to modify the schema, then you get the upside without really any of the downsides.

Yeah, that makes sense. Actually, this gets to another kind of question, which is, you know, my background is as a sort of developer, someone who very, has very limited use of knowledge of databases. And so I tended to be kind of dismissive of stored procedures. I kind of felt like stored procedures just seemed like an anti-pattern. Seemed like code that really, you know, logic that really belongs somewhere else. I was wondering what your take is on this. Like, when are stored procedures useful? Like, when should people be using them versus just putting that logic in their application code?

I have run MySQL at scale across many companies and never have seen a— never have stored procedures been worth it.

Yeah. Okay. We're on the same page. Okay. I guess there's not much of a debate there. Yeah. Every time I've seen it, it's been a project that for 100 other reasons has not been engineered correctly. That just seems to be the pattern I've found.

Yeah, yeah, it's just— there's just some, some real abuse happens to databases, and it's never really a good idea. And there's like some shortcuts, but not many. If you can, you just have to try and keep it sensible. The worst— the customers that are in the worst pain that require the most help are the ones that for us that have got too clever with what they're working with now. And it's really, really hard to get. You're only losing at that point. In the pitch deck that we made when I joined the company, I had this graph that goes up and it's a graph of innovation and kind of velocity for your company goes up. When you start a startup and there's like 3 of you, you're shipping code all day and you're talking to those first users and they're like, why don't you do this? And you're like, cool. And you do it in 2 minutes and they're like, Wow, you're amazing. That's awesome, right? Look at GitHub, just rocket ship growth from the first week. They just put something out there. Chris's early tweets of just like, I'm going to set up a Git server. Oh, that was very difficult. Don't like that. I'm sure other people have these problems. Let me build this thing. And then everyone's like, yeah, I love it. And it goes really quick. And then on this graph, there's a plateau of the middle years of your company, which is where you have to hire tons more engineers. You have to start paying down all the bad decisions you made in the early days. And you're losing at this point. Your company is losing its potential maximum valuation while you solve database problems, because your users do not care. We all use those products, it's like, oh my God, this is amazing. They're building stuff for it all the time. It's like they're building my dream product, and then they go quiet, and they stop shipping. Some part of their stack has stopped scaling. That's what's happened. And they're just spending all of their roadmap time bailing themselves out.

Yeah.

And then people are looking for other products or other ankle-biter kind of startups are starting and exploiting the fact that you are now slow and old and they're doing the— that's really tough. This was the vision for PlanetScale, was that you don't have that middle-year plateau where you inevitably replace that Heroku Postgres database that was super easy to set up and get going on day one. And now Just fundamentally just doesn't work anymore.

Yeah, I think there's a lot of parallels between this and our interview with Guillermo Rauch from Next.js where, and I was telling him the story that I'll rehash now where when I start, and again, I'm also not a frontend guy either. So when I started with Next.js, it just kept giving me errors. Like it just wouldn't let me do things. And I was like, what the heck? You know, this is so annoying. Um, but then once I, once I found how to do what I wanted to do in that pattern, it took like half of my website and made it a static file so that there didn't, it didn't need to run any JavaScript and it could get cached. You know, you could Google like that page and you would get the contents, not like, you know, JavaScript loading or something. Um, it did that and 100 other things that you know, would be things that I would eventually have to tackle if this got that popular, right? And they were just done, right? And so I think that's one of those cases where, you know, when you use an opinionated framework, it'll keep you from making a lot of these mistakes. The stored procedure thing I was thinking of, this was a really long time ago. I mean, this was in the '90s, but basically we didn't have, I was basically the first application developer. We had a database administrator. And then we had a couple of people who also didn't really know what they were doing on the app dev side. And so, you know, because the DBA was strong, you know, he did a lot of the logic and stored procedures to limit the complexity on the app side. But then, you know, it just didn't really work. And so then now you have to play catch up. So it's like, okay, now I have to, instead of starting from, you know, a foundation, now I have to start from— from nothing because we had been walking on the crutches of stored procedures. So, yeah, it was really hard to refactor and get out of.

How big was that engineering team?

That was maybe 5 or 6 people. It was pretty small.

And one of them was a dedicated DBA?

Yeah, exactly.

Crazy. All DBAs are there to do, mainly, is to mitigate downside. Right? Like DBAs are there to stop outages. It's a very hard job, by the way, like saints that do this work. It's extremely hard. Having like a fifth of your engineering team be assigned to just so something doesn't break is insane to me. This is why we do what we do because it just shouldn't be that way. Like it's wild that databases are so unapproachable, but essential. Yeah. That you have to spend a fifth of your engineering team on just stopping it going wrong. The value to a product of the database working well is very clear, but no one, there's not many features that are solely enabled by the database in a sense that you need them to build the features, but it's not like this is a really rad feature that we're exposing to you because it's a feature of the database. Like, that's not true. You abstract over the database every time. So they'd have someone dedicated just to make sure it doesn't do the really bad thing that they predictably always do. It's wild. It's just an insane sink on innovation. And it's still, we're not there yet where I think people are still okay with this, or they're not okay with this and they don't understand that there's a better world. They are, you know, you talk to companies that they do not realize how deep in the mud they are. Because they're their database choice.

Yep. Yep. Yeah. Imagine if you had a, I'm the, I'm the electrical administrator. And when the electricity doesn't flow the right way, you call me up. Like we would just never get anything done.

You know, I stare at the green light and if it goes red, I do something.

Yeah. Oh man. Really, really cool. So let's talk about PlantScale as a company for a moment. So, you know, what is roughly the size of PlantScale and where is it located? Is there like a headquarters? Is it kind of like, uh, uh, you know, one of these companies telling everyone to come back in the office? Is it on the other side where, where it's completely, you know, like globally, uh, distributed? So what is PlantScale like?

PlantScale is around 80 people and we're based on the internet.

So, all right. Nice.

It's an extremely remote culture. We don't have an office. We're distributed around the globe. Partly that was because a lot of people came from GitHub, which GitHub was that way and was probably an early pioneer of that culture. I don't know, it's just the best place to do work really, at home, isn't it? I think now that's—

Yeah, it's the most available conference room. That's for sure.

Exactly. You're right. I love that. I've never heard that before. That's fantastic. I loved going to the GitHub office. It's an amazing office, kind of legendary in terms of how beautiful and wonderful, and they put loads of time into it. No office has ever felt like a place to me where you just sit down and really get the work done. Facebook was an in-office culture and it worked well, but they had to push it, if that makes sense. It was in a way that it was impossible to be effective if you weren't in the office, or one of the offices, because your team was located around you. And look, I love everyone that works at PlanScale. If I could click my fingers and get us all into one really expensive building in SoMa, I'd probably do that, if I could, because it would be really cool to see everyone every day, and it would be really great for bonding. It's just not an option I have, right? The person who's built— the team that are building Boost are over in Europe right now. One's in Spain, one's in the Netherlands, a few over in New York. I want the best people. We've hired some of the best engineers that worked at GitHub. They're not moving. They don't want to. They have their life. So I just don't have a choice, right?

Yep.

So we work with it how we work with it. We manage. It's nice to be asynchronous. It's great to, I work long and strange hours and I would feel really stuck if I had to be in an office to have any effect of the work I do. So I love it. You can get a lot done in Slack. Yeah, it has downsides. You can't just tap someone on the shoulder and whiteboard something out, which is something that's very, very powerful. So you have to make up for that. I'm not one of those remote people that pretend the remote is the solve for everyone. and that everyone should be remote. Like, I just don't believe that either. I think it has incredible upsides, some quite severe downsides, and so is the in-office culture as well.

Yeah, really well put. Yeah, I remember one time where, um, I walked into Facebook in the morning. I took the bus, which, um, you, you sounds like maybe you lived in the city. I don't know if you worked in the city office, but I had this huge bus commute that took forever. I lived, uh, more rural and, um, Got into the office and just, you know, there's certain days like this where you feel really in the zone. And I remember just being there the whole day and, you know, running into different people and, you know, being able to eat all your meals there so you could really focus. And then, you know, worked like crazy hours, got some product out, and then, you know, spent like 2 or 3 days at home where I was like just kind of recovering from that kind of like crazy blitz. And so I can't, like, likewise, I can't really say, you know, there's pros and cons to everything. I can't really find anything specific about working remote or working in the office. I mean, clearly, like, having the accessible food is really nice. Not being able to get a conference room is really annoying, but there's just not really any way to fix either of those things. So it's like, it just, it's just going to be 2 different environments and we just need to make the best of both of them.

You can build some great energy in an office culture, right? You can feel excitement when there's things to be excited about. We are incredibly social creatures. We pick up on social cues that you're not even aware of, right? Presenting on a Zoom, I'm telling you, a Zoom all-hands with 80 people looking at you, it feels not so great. But when I presented in-person all-hands, GitHub, you can hear people's, like, you see their nods of approval, you can hear their kind of, like, subtle gestures of approval, or, like, when people clap, or they get excited, or you just feel the energy. It's way more energy in person.

Yep. Yep.

I'm not gonna just, like, pretend that's not real, right? It's not the same. At the same time, what you have to do to get that is not always what companies are willing to do. And yeah, like, it was awesome. of Facebook, right? You'd walk past, say, the 2 of the best networking engineers in the world arguing on a whiteboard, and you listen, you get like, that doesn't happen in a remote company. You may hop on a Zoom and get it done, but it's still not the same. It's different. And junior folks, early career folks struggle a bit more in a remote environment. When you've got people that are new and they're learning the craft, You can see when they're getting stuck. They kind of sit near their manager and they're looking a bit disengaged or a bit worried and they seem stressed. It's easier to check in on people.

Yep.

I've seen in remote cultures, early career folks getting really stuck and they feel the barrier of scheduling a Zoom call is too much formality compared to just saying, oh, can we just walk to the kitchen for a quick coffee and I just got a couple of questions for you. Like, that's such a lower barrier than jumping on a Zoom. So it's, again, it's just all ups and downs and trade-offs. And there's some really weird and bad takes on the internet about how you don't need offices or remote is terrible and you can't ever build a company remotely. Both takes are completely wrong.

Yep. Yep. Yeah, I totally agree. So So PlanetScale is remote, totally distributed. And so are you hiring at the moment?

We have a few roles open. Yeah, they, uh, we've got folks in customer success. We've got folks in sales. If you like computers and databases and want to work with some of the biggest companies in the world with their tech stacks and their most important part of the tech stack, the database, it's a really fun company to be. You can get super nosy.

Cool. Yeah, that's awesome. And what sort of, uh, so, you know, in general, um, you know, not just hiring, but in general, like on the engineering side, what sort of, I'm trying to figure out how does someone get into, like, what sort of skillset are you looking for? Because, you know, my guess is you're not necessarily looking for people who are good at using a database. You're looking at people who have that, but are also really good at the insides of it. And so what is that most closest to? What's the skillset there?

There's really varied skillsets in the PlanetScale engineering team. You have the people that build our query parser, right? And they are working on really deep computer science that I just struggle to understand. I'll go and look at what they're planning and it's all math notation. And it's like, I don't understand any of this really. Like the complexity's there.

Yeah.

It can be really tough. So they're working on the core of Vitesse, but they're not really called upon to build the UI, 'cause it's a completely different skillset, right?

Right, yeah.

So the folks that build the UI, like our designers, they code, they hand it off to some of our services team that are fantastic JavaScript engineers. Our API and app is a Rails app. With Next.js up front for the actual user interface, but the things that talk to our CLI. Then we have our middleware that schedules all this stuff to happen in the backend. That's all written in Go. So Go backend engineering skills, understanding how to build services at scale that are reliable. It's a big varying skillset, and people usually are specialists. We don't really have many people that are jumping around. the stack, doing it all. People have—

Yeah.

We have hired such incredibly tenured and smart people that everyone gains respect for them mastering their craft and trusts them to do the things that work extremely well for them. And we don't do anything by half measures. Like I said, our product designers have been building products for decades and have built famous and loved products. No one, no database company has the ratio of designers and, and, and, you know, cause they just think they're, they're wrong about what they're delivering to their users. They think they're there just to provide the backend. We, we just can't accept that. So we have to have great people doing all of it.

Yeah, that makes a ton of sense. Yeah. I noticed that, uh, to use PlanetScale, at least today in November, you know, I had to use, I mean, to administer it. Um, you know, I used, uh, the, uh, MySQL Workbench and you have to do kind of some finagling to, to get to the right branch and everything. But I think there's a huge opportunity there. Anytime I'm using a desktop app that takes, you know, more than 3 seconds to load, there's an opportunity there. So, so I'm really looking forward to seeing what, what comes up in the future. Um, so before we head out, I wanted to talk about the book that you recommended. This is something that's a programming throwdown kind of Mainstay. We always ask folks what are books they recommend, and you recommended Amp It Up. So kind of talk us through Amp It Up and what's it about and how did it affect you and inspire you?

It really resonated with me, this style of leadership and management that Slutman talks about. You know, we have one value at PlanetScale, every day matters. We're going to add more, but I don't believe in just like, you know, you've seen company values rolled out where it's like 10 things that it's like no one could ever—

Yeah, that's right. Yeah.

It's like empathy, cool. Yeah, empathy is great. You should definitely have empathy, but no company is going to disagree that you shouldn't have empathy. So why is it a product value? Why is it a company value? The company values should— people should look at it and go, oh, I completely firmly agree there and I would love to work at that place, or, oh, I really disagree with doing it that way, so I don't want to work there. They should be very divisive. If they're just a hand-wavy thing, Yeah. thing that everyone's going to agree with, useless value, not worth it.

Yep. Yep.

And so I read this book and I thought, this aligns with my values. I believe that you should work extremely hard and push for outsized results. And so does everyone at PlanScale. They work so hard. We don't burn ourselves out. We take time. In fact, this Friday that just passed is one of our first Fridays. We all take one Friday off a month together so we can all have real downtime. It's impossible to take real vacation in the modern world with no notifications and you get phone calls and stuff. So we have to give everyone a day off together to really get them to chill. And that's great. It's funny though, people start creeping back onto work by Sunday because I think they get lonely or bored, but no matter.

They get antsy.

Yeah. I do certainly. But that's the thing, is that it just felt right to me that this spirit of aggressive motivation towards the goals that the company has, And we believe in that. We want to run as small a company as possible with as good and the best people we can possibly find. And that takes running a really disciplined culture. You can't hire amazing people and give them no autonomy. At the same time, if you have a highly autonomous culture with no accountability, you go wildly off the rails. And so just a lot of the principles in that book really resonated with me. And I read it Only recently, we've been running PlanetScale the way we run it for a while, and I had never really seen that style reflected in book form. And to read it, I had a lot of respect. And also, it's from someone that has built an incredible company. I mean, I can't really read business books of advice from folks that haven't done something that I find impressive. And I do very much find Frank's work very impressive.

Very cool. Just for context, so the person who wrote this book is the CEO, is it former or current CEO of Snowflake? Which is awesome.

And has made many, I think this is his 3rd billion dollar company that he's created.

Wow. Wow. Amazing. Cool. Yeah, I'll have to check this out. This is right up my alley. I've been trying to read more on the product management side and trying to exercise that part of my brain. So this is, Personally, I think super relevant. I'm excited to give this a read. I'm going to add it to my Audible, which is a great segue to our shameless plug for Audible. If you're listening to this and you don't have an Audible account, definitely grab one. Patrick and I have had them for a zillion years and I'm still an active member reading a ton of books on there. And you can also catch us on Patreon and support the show that way. So either way, We really appreciate folks supporting the show and beyond supporting the show, even better is when folks write in and a lot of our interviews and a lot of other topics have been because folks have written in and suggested things to us. So we really appreciate that. So yeah, Sam, this was amazing having you on the show. You really were able to dive deep on databases, explain a bunch of different concepts to folks. I love some of the things that we covered around transactions and sharding and, and then moving on to PlanetScale and how we're able to do kind of a more coding-like environment with branches and PRs and merging and all of that, but on the data, in the database world. I encourage all the folks out there to check out PlanetScale. It is totally free. I've had the same PlanetScale database for probably like 12 months now, and it hasn't given me any trouble and they haven't asked me for a cent. So maybe, maybe if my website gets popular enough, I'll hit some threshold. And if it does get that popular, I'd be more than happy to contribute. But up until now, it's been in the free tier, which has been extremely generous. So folks out there, check it out. If you've never used a database before, this is an opportunity to use it for free without having to spin up a server or install a whole bunch of packages and deal with all of that. It's super accessible, definitely the easiest setup I've ever experienced. So So check out PlanetScale. We'll put links in the show notes. And, uh, once again, Sam, thanks so much for coming onto the show.

Thank you so much for having me. I really, really enjoyed this time. Very, very enjoyable. Thank you.

Sure. All right, everyone. I'll catch y'all later. Music by Eric Barndaller. Programming Throwdown is distributed under a Creative Commons Attribution-ShareAlike 2.0 license. You're free to share, copy, distribute, transmit the work, to remix and adapt the work, but you must provide attribution to Patrick and I and share alike in kind.