PlanetScale Postgres with CEO Sam Lambert

Welcome back to the Database School Podcast. This is a very special episode for a couple of different reasons. One, this is the first episode that we are putting on the new YouTube channel. So we're going to have a dedicated YouTube channel for the Database School Podcast. So please subscribe to this new channel if you enjoy these podcasts. And don't worry, there's not going to be any Laravel content, there's not going to be any PHP, it's just going to be the Database School Podcast. So So I'm trying to grow this channel specifically. So if you like databases, please subscribe for more content like this. The second reason that this is special is this interview is with my former boss, Sam Lambert. So I talked to Sam Lambert, who is the CEO of PlanetScale, and I talked to him about PlanetScale moving from being very famously a MySQL company to now having a Postgres offering. In this interview, he mentions the Metal release, which I have a full interview with a PlanetScale employee talking about PlanetScale Metal. I'll leave a link to that down below. This is a great episode talking about why in the world move to Postgres, uh, what is different about MySQL and Postgres at a technical level and at a, at a community level, and how this change has affected kind of like the internal vibe of the company. So I think you're really going to enjoy this one. There's kind of a surprise announcement at the end that you should stick around and listen to. So this is a very special episode. Please subscribe to the channel right now. I will stop talking and let you listen to this interview with Sam Lambert, the CEO of PlanetScale. Hey y'all, like I said, this is a very special one. This one may surprise you a little bit for anyone who's been following along very closely Today we have the CEO of PlanetScale, Sam Lambert, the company that I used to work at. And you may be surprised what's going on here. Don't be surprised. We're still friends. Business is business. And we're here to talk about some very exciting stuff. Sam, do you wanna introduce yourself a little bit further?

Hi, Aaron. Hi everyone. Yeah, my name is Sam. I'm the CEO of PlanetScale. We're kind of the world's most scalable database platform. I've spent my career obsessed with databases and data. Um, before this, I was at Facebook as well as a little website we all, I'm sure we all use, which is GitHub. And, you know, kind of joined there really early and then saw it all the way up to, I think, having 100 million developers on the platform, which was really fun and saw a ton of scale challenges doing that. It being, I think, 40th most visited website on the internet. So pretty large.

Pretty large. Ben, before, before we dive in, can you go back to GitHub just for like a week or two and just make it a little bit faster? Can you just go back and fix the stuff, please?

I wish. Yeah, I, I'm feeling those issues too.

It's sad, you know, sad to look back and see things like that, right?

Like how things change and how they evolve. But luckily we've still got a load of the crew over here doing, building nice products that look a little bit— Yeah.

So you, you and several of the former GitHub people, um, walked over, started another company called PlanetScale, which has historically, and that's what we're gonna talk about today, historically been the world's most scalable MySQL platform. So historically it's been MySQL with Vitess, um, and y'all are running these massive, massive workloads, winning these giant deals. And then out of nowhere, we get this like revelation, hey, we're doing Postgres now, which when I was there was always like a joke. The joke was always, people are like, hey, this is great. When are you going to do Postgres? We're like, ah, guys, don't ask me that. And then it happened. So let's back up just a little bit. Tell me what PlanetScale has been up until just recently.

Yeah. So up until Postgres, like you said, we were very, well known for running like very, very large web-scale workloads for a lot of consumer companies. Like we have Blizzard as a customer. They run their entire backend on Vitesse. So if you play like Call of Duty or any of the Blizzard games and you rack up scores or anything to do with your profile, that's all stored in like a giant Vitesse cluster. Slack, every Slack message in the world is stored in Vitesse. Cash App, all of their transactions, like immense volume. MyFitnessPal, Cursor, all of these like kind of apps that have just massive amounts of users that have to be online all of the time.

Wow.

where like downtime is extremely painful and impacts just a really large audience. We were very, very good at running these at huge scale. And like you said, the scale is astronomical, petabytes. In fact, the largest Vitesse cluster I know of is at 900 petabytes of data. So we're— yeah, right. So yeah. So we're getting close to exabyte scale. The largest MySQL I know of is multi-exabytes. But yeah, Vitesse itself, like absolutely astronomical scale. So to put that into perspective, we see customers spinning up single databases that will have 1,000 machines behind them.

Good.

No problem. And that's like 1,000 machines looking like one connection, like one connection string. And that's 1,000 different servers pretending they're one and working together in unison to be one, which comes with like a ton of complexity and some trade-offs involved in that. But yeah, it's like a, it's a two-sided thing. There's the Vitess, the open source project, which is kind of one part of enabling this. And then there's the backend of PlanetScale, which is our operator and our infrastructure. Which, you know, you've seen this, right? Like it's, it's very multilayered and it's not simple at all. Like our edge network has to be incredibly well managed, right? And fast. And how we load balance to even get a connection to Vitess, then what Vitess does with that connection, what the backend does, taking backups, right? Like I watched someone take a 260 terabyte backup the other day. It took an hour. Which was just amazing, right? Because you get this incredible parallelism across 256 different shards. working in parallel to do this. So the thing I love about this and get really nerdy about is that kind of coordination of watching all of these servers like kind of dance together and build and do one thing that looks very simple for an app that might still be assuming it's like one database server behind the scenes. Yeah, it's really fun. Really fun.

Yeah. So that's, I think, one of the craziest parts to me, and this may be helpful as we move into the Postgres discussion. But before that, like, talk to me about kind of the layers of PlanetScale. So at the very bottom, we've got, you know, we're still talking historically. At the very bottom, we've got MySQL and then Vitesse and then all of PlanetScale's secret sauce, kind of the internal stuff on that. 'Cause Vitesse is open source. So while that is, you know, very, very important, it's not secret sauce.

Right.

But on top of that, you've got all the PlanetScale stuff. And, um, you know, I know some of this based on my time there, but for the rest of the people, like, When you say there's 1,000 machines behind a single connection string, kind of lay out who's responsible for what in that statement so that we can understand the different layers.

Absolutely. So at the very base level, we get like an Amazon— like let's talk about, we'll talk about Metal being the first one, right? So PlanetScale Metal is, it'll be a nice bridge to the Postgres world because it was one of the catalysts that took us towards Postgres. But basically Metal is a physical machine given to us by Amazon and they run a very thin, like Firecracker VM on, on that machine just to give us some primitives. And that gets registered into a Kubernetes cluster. So, so, so first of all, Kubernetes is our first kind of underlying foundational piece that gives us a lot to orchestrate when it comes to all of these hosts. And inside there, we run a proprietary operator, which kind of instructs Kubernetes how we want machines to run and allows us to instruct Kubernetes on, on what the state that we want a cluster to be in. So we define all of these various states by tagging machines in Kubernetes. and making sure that the cluster resolves to being that state at any one time. And that, and that will allow Vitess to understand what it needs to do to get our cluster always into one state. And overall, our architecture is what we'd call a shared-nothing system, which is we aim for nothing to be shared, like no single coordination node or coordination layer. So you essentially have this kind of Swiss cheese, these like various holes through the system. You need enough of them to be up to get a successful query. all the way down to MySQL and back out. We have to make sure there's layers of redundancy at every layer to be sure that you can have zonal failures, regional failures, single node failures, proxying failures, load balancing failures on the way in. So, you'll have every single piece is in triplicate. So, we, every database cluster runs across 3 availability zones at a minimum. So, each shard within that cluster will have a node in an availability zone. So, you'll have a primary and 2 replicas. each one of those are kept in separate zones. And that's first of all the job of the operator and Vitesse. Like it's told it has to be in that state. So if you imagine you lose a node, the cluster knows it has to resolve giving a node back in that region or that availability zone to maintain the specified state that we want it to be in. So we have to build all of these various orchestration pieces so that can happen reliably every single time in the cloud, which itself is— You know, every database server, MySQL, Postgres, were all built in an era where you could assume the application and the database may be on the same machine or in the same rack where there's a reliable, like, Ethernet connection between the two. The cloud gives us absolutely none of that. Like, the machine failure rates that we see are obviously very high for the millions of machines that we have. And that means that, you know, you have to just assume there's been a constant level of failure. Like, Continually, like nodes will fail while we're recording this podcast and a user won't notice because of that kind of level of like robustness that's been built up. So, so underlying Kubernetes with our operator that has gone through all of these various mechanics for provisioning nodes, bringing them up, restoring from backup, and making sure they can be registered into a cluster reliably over and over again. That is also our answer to node maintenance. So you remember this one, I used to say this all the time. No node at PlanetScale lives for more than 29 days. That is the longest possibly a node can live. Most never even make it to 29 days. So our answer to rolling out a new Vitesse version or a new MySQL version or Kubernetes version is just kill the nodes that have the old version. So we'll TTL every node that has that old version. And so like the first thing that comes to mind is why don't they just all die and everyone goes out when you do that? That again goes down to the operator and the rollout. So you can TTL every node in the cluster. And the operator goes, great, every single node in this cluster has now got a TTL. I need to now go through and respect that TTL while staying online. So it knows to stay online, it needs to replace a replica first. It's smart enough to know, like, how to do that. So it'll roll out the replica, whether it needs to restore from the backup to bring up a new one or detach EBS, do a version upgrade, come back in. It then needs to know it should do the next replica. in the cluster too. So, because otherwise if we promote, if we did the master next, we'd have to do a failover and then we'd have a replica that was an older version. So it knows how to do that. And then it knows how to do those 2. And then it would do what's called a planned reparent, which reparents one of those nodes as to be the parent of the cluster. Then it will roll the other replica. So you can see how like we're just getting, I could just get, I'm going to bore everyone to death if I just keep going.

No, not the people listening to this, but— Right.

You can see how just one scenario of an upgrade has handled 3 failures, promotion of a primary. Vitesse has to know that's happening, so it redirects queries out of the way of that rep— it drains that replica before it does it. It then stalls the connections to the primary so that they can do that switch. And that's all in the process of a planned reparent.

Mm-hmm.

You then have things like emergency reparents, which means like the parent primary is dead, the cluster needs a new parent. We buffer off the queries, we promote a replica, we make sure the next most caught-up replica is caught up so then it can— you see, again, it's just these constant layers of just millions of lines of code that handle all of these different failure scenarios throughout the stack. So then we— so that's Kubernetes and the operator. Vitess sits on top, and what Vitess really does is pretend to be MySQL to distribute queries across many, many MySQLs. So that gets us into doing sharding. So Vitess is mostly known for being a, like, an application-aware sort of proxy that will allow you to do sharding. So Vitess implements all of the query passing logic that MySQL implements and knows how to receive and understand the MySQL protocol and the query, um, the query standard. And then it also knows where the data resides. So it has a lookup keyspace. So it knows if we've got the shard that has your user, my user, and we're trying to aggregate. And Vitess actually rewrites your queries. So it doesn't, it doesn't often send the actual— unless it's a very simple, like, select query going to a single shard.

Mm-hmm.

It actually is gonna rewrite your queries. And now if people have taken your course, they'll understand things about query statistics and query planners, but it doesn't actually have an understanding of the query statistics. It has to use its own heuristics and understandings to plan the optimal version of a query, push it down to the shards, aggregate those back, and send them onto the user. So if we do a select star where users are in an organization, for example, and they go onto one shard, that's extremely neat and easy and it gets served within a shard and there's not lots of round trip. tripping. If it needs to say you want to get all users within a country, no matter what organization they're in, you might hit every single shard, of which there could be thousands, and those need to be aggregated. But it can send a simpler select query down to each shard, pull those back, aggregate them in memory, and then send them back to the client, all within milliseconds and microseconds, to go and look like it's just a regular single database server and send it back to So Vitesse is the one that is providing the facade of a single database. Correct.

And so Vitesse is saying, hey, you can ask me anything under the covers. I'm going to go and figure out who's got it, but then I'm going to go get it, put it back together, and give it back to you, and you never have to know that I went out to 256 machines or I went to 1 machine.

Absolutely.

What does it matter? Obviously, there's performance characteristics, but from that perspective of this is a single database, that's Vitesse's job. Now, at a, at a layer above that, you're doing all of this stuff and you're not, uh, you're not dropping my connections. So what's going on? What's going on way up here on your edge network where you're keeping all that stuff available for me?

Yeah.

So application developer.

Absolutely. Yeah. So we have an edge, edge network that can terminate your connections based on essentially where you are in the globe, right? So you wanna make sure that we, if you, request from a read from the AZ that you're in, that we send you to a replica that's in that AZ. So we do all this kind of intelligent routing that will then send you to the gate layer, which is the VT gate layer of MySQL, of Vitesse, sorry. But it also has to be aware of load balancer failure, like network load balancer failures at that level, handle authentication and expiry. Like you want to revoke a password on thousands of machines, it has to happen extremely quickly and you have to terminate those connections. Or you have to keep them alive so you can shuffle proxies under the hood. So it's this extra layer of like awareness of keeping, unfortunately, don't do this folks, but unfortunately people wanna keep very, very long database connections around, otherwise their application breaks. Like a well-written application can handle a database connection going away, but a legacy application may not. Might again, being built in the data center, like we, we're a bit older now. I'm sure we built some apps that run in the data center when, when, when they were, we could just leave them connected to the database server.

Mm-hmm.

And the database server's very likely not gonna go away. We've just said like PlanetScale is gonna make sure your database server goes away. So we have to have a place that does hold that connection for you and reestablish it for you to go back to the layers that have kind of scaled up, scaled down, moved around and shifted. Another option that's not in the edge, but another thing that Vitesse does is resharding, which is again just a really hard problem where it will redistribute your data based on a different sharding key and have to move tables and actually move your traffic in a single operation over to another cluster. So we did this with a customer. They're very public about using us, but at Cursor, right, we just did a reshard. They wanted to scale up. So we did this resharding operation, which involves rebuilding another set of shards elsewhere.

Mm-hmm.

And then kind of reading off of that, having the gate layer and the edge layer direct us to one place and then switch to another atomically so that no connections like go wrong or no transactions get left behind. And so you have to like drain off of one, move to the other, and give all of this status across so that you know where you're moving traffic. So you have a move tables and a switch traffic operation that allows you to kind of manage where you're routing your logic and where your queries to allow you to do these kind of scale-up operations. So we've had people reshard to change their sharding scheme, to add more shards. There's just all of these kind of, it's all a massive orchestration game.

Mm-hmm.

Just a huge, like, shuffling things around and making them happen and work in the right places. And then sometimes you have to do that cross-regionally. Like we watched a 600 terabyte cluster fail cross-region, across the country. So we moved them from—

Really?

Yeah, we moved them to east to west, which means that we have to kind of recreate the entire setup in the west, which is again, I think that cluster is probably like 900-ish servers and have them connect to their parent shards back in the east and make sure that we drain connections successfully and do a flip there. and orchestrate and go through all of these things. And, you know, very proudly I can say that this is all managed by a tiny team of 45 engineers over at PlanetScale.

Incredible. Yeah. I remember when companies would publicly tweet or send emails like, hey, we're doing database maintenance. We're going to be down for 6 hours. Everybody internally was like, uh, we should go talk to them because this is crazy. Like the fact that, the fact that PlanetScale can do all of that without incurring that downtime is That, I think, has always been to me the most impressive part. And the fact that like, you'll take a healthy, beautifully operating node out and shoot it in the head 29 days later and be like, sorry, we do this all the time. I feel like creates that operational discipline of, boy, we've got to have this stuff nailed the freak down because we're going to be doing it so often. So—

You remember as well, by the way, if anything went wrong, the paranoia layer level of like what that what happened when that goes on. It's like running an airline business, right? You know, that like we would get the, you know, the voice recorder and go through it as like crazy because you could just see it again at massive scale and it would cascade and be terrible, you know?

I would watch some of the, some of the like active migration channels where everybody's like all hands on deck and I would just like DM Holly, who's, you know, Holly and I were marketing and so we weren't like needed to do this part. And I'd be like, I'm so glad this is not my job. I am so glad this is not my job because boy, that looks stressful, but it always went well.

Yeah.

I guess moving forward now, you've built all these different layers, very complex, very highly orchestrated, very based on a little database called MySQL. Then we had Metal, which I talked to Richard about Metal a little while back, which was awesome. Right after Metal, right after the big announcement for Metal, which seems to have gone very, very well from, you know, the feedback I saw on Twitter. It seems like everybody was freaking out about Metal.

Yeah.

So congrats on that. But after that, I saw, hey, we're doing Postgres now. And I was— I, I probably DM'd like 5 or 10 different people and I was like, wait, what? Is it April 1st? What is going on here? So walk me through, walk me through. You've got this company that's been alive for many years that is just crushing it in MySQL world. You released this new thing. that further crushes it, which is Metal. And then you're like, hey, what if, what if we did Postgres too? What happened there? Like, walk me through that, that decision point.

Yeah. And believe me, I was with you on, wow, this is crazy. Okay. So it's a whole form— it's, it's lots of things that really went into it. So first of all, we ship Metal, right? Which is like running, we're very resilient. So most people have to use EBS, right? To run, uh, which is Elastic Block Storage. on Amazon, and then there's the Google, I think they call it Persistent Disk, although it's very, it's a weird name. Anyway, that's like network storage, right? So that when your nodes disappear and die, you're not truly screwed and you lose data. And that's what every other database service runs their data on. Aurora runs their own kind of version of that file system around the back, but it's essentially a network file system, like almost like a SAN that gives you this kind of highly available disks so that machine failures are not an issue. Well, as you, we've just gone through and as you've seen in the past, like we have got to this level of maturity through going tens, maybe probably hundreds hundreds of millions of failovers at this point. We're fine with nodes dying. We know we can always keep a version of your data around. And then we can do a couple of other things for like persistence and like spooling logs off of disks that we could always recover your data if we had to. But we basically calculated there's a, across an 8-year window, there's like a 1 in 100 million chance that we'd even power a cluster, a full set of clusters, a full shard down and lose, and we'd never even lose data in that scenario, right? So we felt very comfortable with with taking the completely ephemeral nodes that Amazon gives you, these, these metal nodes as we call them, that are very cheap and have insane amounts of IOPS. Now when I say insane, I think I tweeted the other day that we have one cluster that's out there with 280 million IOPS, or 700, crazy numbers anyway. Um, so yeah, so we went and shipped that. We shipped Metal. You should go and watch the podcast with Richard, it's really good. People loved it. Like the, the release I was proud of is very simple. It was just benchmarks, or not even benchmarks, it was just comparisons of taking— we took Cash App.

Mm-hmm.

On the live stage.

It was customer stories.

Yeah, right.

If I remember correctly.

Absolutely. So we had Cash App, we had Intercom, 2 just absolutely incredible companies. Um, and we showed the performance delta of just switching them over, over to Metal. It's just incredible performance improvements. And we got like tons of applause. Everyone was like, yeah, that's really cool. But the crazy thing happened is like loads of Postgres companies came in. One were either asking for us to do this for Postgres, but like really asking, or were moving to trying to move to MySQL to do this. And I watched a load of AI companies just moving really, really fast. And like, on the first— in the first couple of days, we were— me and Nick, our CTO— up to like 2 in the morning helping companies migrate over because they were so excited by Metal.

Mm-hmm.

You know, I had this nostalgic memory of actually watching GitHub migrate to Fastly. I remember we had them as a second, like a backup CDN, and we had, um, Akamai as the primary one, and Akamai went down. And we scrambled and migrated to Fastly and like never went back. And I, then I started thinking about how we used to buy software in the early days of GitHub. It'd be like, put it on my credit card, put it on someone else's credit card and just go crazy. And I just remembered all of these startup days and I kind of realized that pretty much every startup in the world starts with Postgres now for like whatever reason, Postgres is won when it comes to new developer adoption. So it's like, yeah, that's probably true. So then I, you know, a friend of mine, um, who runs a really awesome startup that's doing extremely well. I said, yeah, you're on Postgres, right? Yeah, yeah. Why? He goes, well, this is what everyone else was doing. Just like I was on the RDS creation screen and it said Postgres on MySQL, it's just like everyone else I knew uses Postgres. And it really— and I should never have forgotten this, I guess, but it really reminded me that in the day one of building software, there's no criteria. I was always And you remember this, I was always like, MySQL's better, people want it for this, they want scale. What about when you do this? They don't give a shit on day one. It's like, it doesn't matter. It just doesn't matter. Like, they just want to build. And so there's no criteria. The criteria is vibes, and you can't fight the vibes. Like, it's truthfully it, you just can't. Things tumble and you just look. I remember, we remember Mongo now is incredible business, incredible product. Used to lose data, used to have all sorts of problems. Doesn't matter, people wanted it, it worked. So then I had that realization. And then I also, after talking to people that were running seriously large Postgres clusters and saw the situation that they were in, and them asking to do it, like very genuinely, like people would try and troll about the Postgres thing and that's fine. And then I also just realized that most people who love Postgres don't run it themselves or haven't. MySQL got really successful in the data center era, the tail end of the data center era for a lot of the web-scale companies. Postgres was pretty much made successful by Heroku adopting it and taking away the backend pain of managing it. Like I realized that managing in the backend is significantly harder than MySQL. And if we're this good at doing what we do with databases and that every startup is building this way and building fast-growing AI companies or whatever, there's a real place for us. And they want it. Like they know what the brand is. Like the thing that was most surprising about the Postgres launch was it, we just said PlanetScale for Postgres. Like it was all we did.

Mm-hmm.

And you saw that, right? Like, I'm curious to see what you think, but I think people just got that immediately. Like, do you know what I mean? Like there wasn't really, They just, they just understood what PlanetScale plus Postgres would mean. That I think was the best part about it. And so that's actually why, that truly is why, is because I think Postgres hosting needs a PlanetScale and we had to go and build it. And so we had a week where we basically turned things, like we decided we're going to just try and get things running. It went quicker and better than we expected. And then we started building and told the world and now it's GA. And it's been very quick. Yeah.

So fast. So fast. So let me, before we get into the technical part of it, so you, you all launched Metal, you looked at the current setup and you thought, we're getting these trade-offs that we don't need in terms of like the, the EBS volumes, right? Like we're fine if it goes down because we have other things that are handling it.

Yeah.

Let's not incur this penalty for a benefit we don't need. And so you just went to Metal. Metal. crushes, everybody loves it. The launch was very counter-current trends, which is, you know, hype launch. This was like, hey, here's what our customers did, here's some graphs. And it was like, Yeah. That's kind of, that's pretty good, actually. So everybody loved that, everybody freaked out. And then you're seeing, you're seeing everybody wanting that, but not wanting, or, you know, having to switch their actual underlying database. Some want to, some don't want to, some are pushing you to do Postgres, some are like, whatever, I don't care, I'll switch to MySQL just to get this. And so you have this realization that's like, yeah, the, the vibes— that's the hard part for me is like the vibes are hard to beat. I remember talking internally about like, uh, MySQL is, you know, easy to scale. Not easy, easier. Postgres is very, very difficult to scale. Why do all these people want Postgres? It's like, because they've never had to scale it themselves. They just get to use it and they love using it. And I've seen a lot of that. People love using it. It's a very usable database.

Absolutely.

And so you all looked at it and you're like, wait a second. And this is the part that, that I want to like dive into now is how do we adapt the stuff that we've built to this completely different database? Like, in this whole stack of stuff, how can we use not only our operational experience, which, you know, you've got 45 engineers who are, you know, world-class, but how do we leverage, like, Insights? I see tweets from, you know, Ben and Holly that's like, Postgres is now plugged into Insights. And I'm like, what? How? What are the primitives there that can be shared across everything?

Yeah.

So, talk to me about this one week of I assume it's some sort of like internal hack week where you're like, let's see if we can make this happen. Was that kind of the story behind the scenes?

Yeah. I mean, it started out with me kind of ensuring everyone that I hadn't lost my mind, that I was still like—

How did that go?

Yeah. I mean, I'm sure they don't believe that or would believe that on any week, but let alone Postgres Week. I was like, let's just try it. Like, what's the worst that can happen? Like, it can go so badly that we can, we talk ourselves out of it. We talk ourselves into it. Like, I don't know. All I know, and I just showed them like DM, like my Twitter DMs, the emails, like ever stories. Like I had a CEO from an incredibly successful tech company in this room, right? Like that I'm in right now saying, I spend hundreds of thousands a month on Postgres. You guys should build this. I will immediately, the second you do this, I'll become a customer. And so I told them all these stories and just said like, why not? Like, you know, we have, we actually have people on staff that used to work at some very, very large Postgres deployments. So they were interested. Uh, certainly. And I said like, let's just spend a week, let's just see if we can get Postgres in prod and receiving one query by Friday. Like, if we do that by Friday, we'll, we'll feel amazing, right? Um, we had that done by Tuesday, and so then by Friday we had failovers and insights working. Now we've had to refine those things obviously, but like, like, and it was a very version, like basic version of insights, and I think it was running on Rafer's laptop, but it still was like a beginning. And we'd really surprised ourselves by the end of the Friday. And then I said, okay, we're going to be GA by November. It's September and we're GA.

September?

Yeah, yeah. But you, you remember this too, like very much our style. We never hit our dates. We either go back or forward, and very often it's forward, right? Like very often we just, we don't sit around waiting. If we feel comfortable, we get it shipped, right? So we just went and did it, and we got to the date where we, we knew we could like tell the world, and like it was going to be a one-way Street after that. And so we had to be really like deadly sure that we can hit that date and hit certain milestones. And we got there in July. Yeah. And so we really, the main underlying pieces that, that contribute are the operator, the extreme amount of kind of good operations around Kubernetes and, um, running in the cloud. That, that itself is just like a massive amount of work. That's been 4 years of, of refinement. Vitesse is not involved in the Postgres stack. That'll be Naiki, which is on its way. And we can talk about that. We've only just started talking about it and it's going to be a journey. But basically it's our operations, the way we manage and handle data, just data operations, backups, the product that is Insights, right? Insights is incredible. You remember how the reception people would get. And in fact, like now it's, I love that we're bringing this to the Postgres world because it's lots of small startups that I think gain even like a lot of our customers are already experts. The reason I'm so proud of the logos that we have and the products that run on our product is because you can't be— you're clearly not like pulling the wool over people's eyes to sell to them, right? That they are incredible engineers themselves and we have to clear the bar of them externalizing running something. That's great.

Postgres startups— The best logo you have is Cash because they are dealing with money.

Correct.

And if they trusted you to handle the money, I'm fine.

I'm good.

Yes.

They know down to the second how much they're losing if things go wrong. And it turns out if you look at their public filings, it's a lot of money.

It's a lot of money. Money, yeah.

Yes, yeah. So we take that extremely seriously, right? We have to, like, we were like, can we get there in the Postgres world? But the cool thing is they don't have the expertise. They're a 5-person startup, a 10-person, 20-person, and they don't want to. Like, what's the cloud for if they have to learn this stuff? You know, you get it, right? Like, you run your business, you externalize as much as you can. And if you're doing too much, then it's a shitty vendor, right? And so insights provide so much. And there's like people tweeting and blogging out there sharing, like, I moved to PlanetScale, not only did it get faster, because of Metal or, uh, our EBS offering seems faster than most others as well. They're also like fixing their work. We're telling them what their worst 10 queries are and they're fixing them super quickly. That itself has an impact because newsflash, no matter what we do for you, if your queries aren't that good, then, and you're, I mean, you know this very, very well. There's so much goes back to the user and the more we can help, the better.

In most cases, if the database or the platform doesn't tell me that I have bad queries, I got a bunch of other stuff I'm doing.

Yep.

I don't know that. What? I'm building a whole business and application, whatever. Um, as much as you and I would hope, oh yeah, everybody's like, oh, kind of almost a DBA, and they're like, they really want to dig in. Like, no, come on, we gotta— we— they have concerns way up here. So Insight's telling me, hey, this one's real bad, go fix it. It's like, all right, I can do that.

Yep, it's amazing. Like, in fact, Cursor moved over one of their Postgres databases to our Postgres product, and one of the engineers pinged us and said Insights told us about this really long-running query that we didn't realize was running against RDS. Like, they were— RDS did not tell them this was happening. They got rid of it and it solved like downstream weird issues that they've been looking to find. Like, you just find it, right?

It's all fungible. Compute and memory, it's all fungible. So you fix one huge problem, everything else suddenly gets a lot better.

Absolutely. And that's actually another funny thing with Metal. People have moved to Metal and then turned out they don't need as many frontend servers because they're not queuing and they get Like, they're not queuing their requests behind database, far slower database, and you need less for capacity. It's all like, yeah, you're right, you're completely right, it all knocks into each other. And so it's been amazing to ship for a community, and the feedback, the product feedback and the iterations have got faster because the, the people that— they're like younger companies that are deploying significantly a lot more than the bigger customers, and that's normal. The bigger customers are deploying to millions servers at a time and on products that just, they've already made it. They can't, like it's innovator's dilemma. They can't afford to go down. The smaller Postgres companies, they're there to build quickly and reliably and not let their customers down, but while shipping continuously. And so it's been really amazing to have them as an audience enjoying the DX and enjoying the things that we do. And it's been amazing. And the Postgres ecosystem is Phenomenal. Like plugins, I can see why dads love it.

The Postgres ecosystem is crazy to me. So I, my, my history is not as deep as yours, but I've always been MySQL guy, which is why I came to PlanetScale. MySQL guy. Now, like dipping my toes into the Postgres world, the, the community first of all exists. I'm sure that like at some layer the MySQL community exists, but it is like, I, I'm, I think it's all DBAs. It's all way down here. But in the Postgres world, in the Postgres world, there's an active community everywhere. The whole extensions all the way up and down the stack. But then like the actual ecosystem is a lot more, it's a lot more rich. There's just so much more stuff that you can plug in. I think there's a, you know, there's a guy on Twitter that I really like, Cheyenne, and he's like, just use Postgres for everything. You need cron? Postgres. You need jobs? Postgres. And I'm like, yeah, it's kind of true.

It's true.

to me. So when you enter into— when you as PlanetScale enter into this Postgres world, what are you thinking in terms of extensions and like interacting with the community? Because this is kind of like now you're living in both of these worlds, and I imagine they're very, very different. So what was that experience like entering into this Postgres world in a very serious way?

So first of all, I completely mirror your optimizer, uh, like your, your, um, observation, um, about community. Like it exists, like there is a MySQL community, but it is dwindling and it's a set of people that are working on really massive scale deployments where they've gotten more bespoke over time and they're less applicable. And just, you know what I mean? Like Facebook—

It's that woman I talked to at HubSpot, uh, you know, when, you know, they're running, they're running the Vitesse over there by the It's like these little niche communities of people that are uber experts, like super duper experts, but then the general community is just gone, I feel like.

Yes. Yeah. And it used to be there, right? Like when I used to go to the MySQL meetups in San Francisco in 2013, it would be Facebook, Yelp, GitHub, Uber, Square, like all just every cool company that was still young and building was in that community. So then great things would get open sourced in that community. That's all gone. That it just, it is, it's just gone. Those, those, a lot of those companies are now very large at-scale companies that are still building amazing things, but they're, they're pro— everything is different, right? Like everything about their problem set and technical problems is different. And so that, so the benefit of community is not so much there, but the Postgres community is absolutely ripping. And in, in more ways than the MySQL one did. And I think it's down to the extensions ecosystem, like you mentioned, right? There's people building OLAP, there's people doing queues, there's people doing everything in extensions. Some are a good idea, some are silly. But the point, the point being, it's there and it's happening and it's creating this really dynamic ecosystem that's everywhere. And no matter where you go, people use it and care about it. And that— I've loved it. Like, I go to so many more events in San Francisco now because it matters to go to these events. You meet new customers, prospective customers. They're having their own cool events, they're doing cool things. It feels great. It's a truly alive community of very, very smart people, very warm. I thought we would get outright—

Rejection.

Rejection for joining. I thought people would really dislike us. Actually, that's just been the opposite. People have been really reaching out to try and do as much as possible with us. The reason we launched the way we did with the benchmarks that we did, which we benchmarked basically all of the Postgres options that are out there, is because I wanted to show why we were showing up. If I showed up with like, oh, it's just like Postgres but really good DX. I mean, there's loads of that. Like, I mean, it's out there. I think people respect what we've managed to do at the numbers and the volumes and the scale that we've done it. And the way we've brought ourselves out there, the way we talk about what we do has always been there to educate and has always been there to kind of just, it's about the tech, right? Like it's truly about the tech. People appreciate that. And so it's been nice to be welcomed into the community. We want to keep giving back in generous like generous and genuine ways. And yeah, like we're gonna just keep improving things and, and, and just showing what we can do and most importantly, make our customers successful. That's really the main thing that matters. But when it comes to extensions, we wanna support as many of them as we can. We iterate. I think we've got, we've probably got most of them, the major ones now. There's a, there's a really long tail. We're pretty much gonna try and get parity with what AWS is supporting. So for those that aren't so familiar with the Postgres ecosystem, you can build extensions pretty much to do anything.

Anything.

Anything, like anything.

And people have, they really, really bad ideas.

Terrible.

Yeah. Yes.

And like even fun, silly ones, like putting Kubernetes inside or putting SQLite inside of Postgres, which is like hysterical.

But what are we doing?

Yeah. Yeah. People are putting SQLite in lots of places it shouldn't be, but that's fine. Um, and so the thing is, there's one major issue with that is that it's also the number one way people get their data, data, or used to get their data siphoned out of AWS. was that they would just install random extensions that had vulnerabilities and would go out there, right? Like their data would just get— they'd get hacked and it would be terrible. And so they have to have these validated and verified extensions that allow people to get the functionality they want and use what they want, but not have bad things happen or just be unreliable. So this is—

I imagine there's a lot of performance risks with just letting people install anything.

Right. Well, even to actually today, we got blocked on an image rollout for our latest Postgres image because an extension— I won't name names, people— like, they had an issue with the building of that extension and it wasn't working. So it blocked us being able to roll out Postgres because that latest version of that extension had an issue. So we're now kind of— you have this new vector, this new— you know what I mean? Like, we're now—

Lots of tendrils into the actual thing.

Absolutely. It's like, it's not just like Postgres, it's Postgres plus a ton of extensions.

Right.

And we have to back that, right? We are running that in production for people. The other thing we had to get around is permissions. Like, we don't want to give you superuser. We want to give you a sudo version of superuser so that you can actually run the extensions that you need without— again, we don't want to give you too much control that you can just screw up your cluster. Like, we want to take back enough so that you can't disconnect replication yourself. Because that's a failure mode that sounds simple, but people can do all sorts of— like, we can't let you, like, go in and change the replication hierarchy of your cluster because you might do that in a way where we can't— it become— goes into a state that we don't expect and we can't keep you online.

Right.

So we have to kind of keep some of the permissions back, but a lot of these extensions need the permissions or they need other extensions to build themselves and do— it's just our new way of doing things. We're getting used to it and we've got a good process for doing it. And people, people like request them and we turn them around in a day once we can validate them, what they did.

Which compared, compared to your MySQL offering, this has been stated publicly many times, y'all have like your own kind of fork of MySQL that you're like very, very, very carefully updating. And now you're out in this wild, wild west, frankly.

Yes. And it just takes, it's like, we love these challenges, right? It's just like a management challenge. I think we'll, we'll evolve that. to— I would love to get to a point where we can be— we can isolate things in a way that people can run their anything themselves. Maybe we'll get there, but it's a hard challenge. AWS hasn't got there yet. Um, it's tough, but it's fun. And also, yeah, like, I mean, like you said, it's— they move faster in the Postgres world. So keeping ahead of Postgres releases and doing that in a stable way— they don't quite have the easiest upgrade path from version to version. That's what actually— that's one big thing we've seen that's very different is You know how like iPhone, like they're very proud of the fact that pretty much everyone's on the latest version of iPhone. That is not the case with Postgres. We, we have seen so many people that we're, we're on the cusp of 18 coming out, but now we're, we're seeing so many people that have, they're on version 13. One thing I have noticed that when, yeah, right. When you stay, people pretty much seem to stay on the major version of Postgres that they pick and never move. We don't want to do that.

Wild.

Yeah, right. So we have a way of migrating you from 13 to 17 as an online upgrade.

I'm guessing it involves failing of nodes. That's my guess.

Yes, it does.

Bingo, nailed it.

Yeah, yeah. But like, because we want that, because we don't want to run long-tail versions because we take, we take a lot of responsibility for your uptime and your reliability and security. So we need to, and we, and it just becomes this giant management overhead to like support all these old versions. So we've had to do stuff with there. There's various differences in the Postgres architecture that means online upgrades are very difficult. Even online failovers are much more difficult than MySQL. Again, it's just different, right? And these things will change over time. I think Postgres is entering an era now which will be like a more of an operator era. They're definitely making more changes in core that go towards operating things, but the history of the project is just different from the MySQL project.

Sure.

And they're, they're catching up with a whole bunch of things where— and MySQL stopped catching up with the good things Postgres has done, which is a shame, but Postgres is moving along really well.

And where are you getting— where are you getting this expertise in-house? So you've talked about the team being very small. You've talked about a lot of the history of the team coming over from various MySQL shops. And now we're running Postgres at a very high level with a lot of important things on it. Where's that expertise coming from inside of PlanetScale?

Yeah, so we've done hiring. So we've got dedicated Postgres experts now. We've got folks that, you know, help build Heroku's infrastructure, which is like one of the largest Postgres installs out there, and done a lot of learning ourselves. Like, we, we've read a lot of the Postgres source code, a lot of the tooling that's around there. You, you've, you know them, right? Like, the, the thing about having folks like ours is that everything, including the kernel, just looks like software. Right? Like, if you're not scared of it and you understand— like, you, you remember, like, we'd find bugs and go, and the explanations would come back in, like, bytecode to figure out what's gone on and happened. So we're a little fearless when it comes to that because eventually, essentially, you can just go— if you can read and understand what's going on at every layer, you can actually get to that.

I remember doing research for the MySQL course and coming across, you know, bug reports from, like, the early 2000s. and the name on it is Shlomi.

And I'm like, there you go.

Wait, I work with that guy. And so yeah, that, like, that, that is very baked into many of the employees at PlanetScale. So you've gathered some of the Postgres Shlomis, we'll say.

Yes, exactly. And Shlomi himself is working on Postgres stuff.

Amazing.

It's very funny to see his reactions to certain things. Um, but he's like that. I mean, when Shlomi was doing MySQL stuff in the other days, MySQL was really not very good and People like Shlomi made it so much better. So like, we can do it again, you know? And so it's, it's that really. It's just like really digging in and learning it. And like I said, it's— there's some things that are really good about Postgres, like the parser being, you know, like isolated. And, you know, it's like really, really good things that make— you know, I don't want to paint it as like we're, we're picking up something that's like wholly broken and trying to make— there's, there's great stuff that make it, and most importantly, it's worth it.

Yeah.

Because the customers—

it's important to remember the point of view. Like, you're coming— you and PlanetScale are coming from a different perspective than most people coming to Postgres, right? So most people coming to Postgres, just, just by the nature of the, the distribution of builders, most people coming to Postgres are application developers using Postgres.

Yeah.

Very few people coming to Postgres are operators trying to build out an an entire platform based on Postgres.

Yeah.

And so you're talking about different challenges that people like me who are not running a database platform have never had to encounter. And so obviously there are going to be different challenges there than like, hey, I like Postgres because I got— I can use vectors pretty easily and I've got better data types. Like, that's awesome. And so that's not what you're talking about. You're talking about, well, when I'm failing over millions of nodes, things are a little bit more complicated. Which is fine. That's totally fine.

No, absolutely. Absolutely. We really take what we're on the hook for really seriously. And so, yeah, and, and it feels so good to do that work so that people can do all the things you're talking about and move fast and build like amazing applications against the incredible interface that Postgres gives them. We just have to take that really seriously. Like, I think people, you know, people think I'm sort of some crazy MySQL lover and then dislike Postgres, I can hate on MySQL better than any single one of you listening to this podcast, unless you work here at PlanetScale, because you might be, you might probably spend a bit more time thinking about it than me now. But my point is, like, a friend of mine, he works on the world's largest MySQL install, and he said the same thing. He's like, you know, we are the best haters because, like, at least we know everything that's wrong with it. And that's true. I know all of MySQL's deficiencies, and we are there to learn Postgres's deficiencies and know that we love it enough to make it worthwhile. You, you wouldn't like like, you wouldn't go to a surgeon that's like, you know, I fundamentally think bodies are infallible, you know, I can't really screw up, it's going to be fine. Like, you know, or like a plane mechanic that's like, these things never really fail, you know.

Yeah.

It's hyper paranoia about the failures means that we will build something that's end usable and all of the good parts of Postgres can be exposed for what they are and people can use them and we will sit up till 3 in the morning finding all of the crazy things that are wrong with it, fix it, and hopefully send a patch back to actually improve things. That's, that's the point. We will stay paranoid and critical.

You want the old crusty mechanic who's going to go in there with the wrench and bang on the engine because he knows exactly where— he's just got to beat the heck out of it. Like, that's what you want, somebody who is so familiar with it that they know all the deficiencies and they know which knobs to turn. So let's talk about, let's talk about this Postgres stack now. You said, uh, you said Vitesse is not a part of it, and then you said a word. I think it was Necky. I've never said it out loud. Yeah, Necky.

I've— the pronunciation's gone away from us already. It's whatever, you know, there's no official one. Whatever, whatever works for you.

So you've got, uh, you got Postgres at the bottom being orchestrated by this same orchestrator. So you've got the primary and a couple of, uh, replicas, and then All the way up at the top, you've still got the PlanetScale edge network and all of the, you know, the insights and all that on top of it. What's going on here in the middle? Is that where Necky is going to live, is living? What is it and where, what role is it going to play?

Yeah. So in the, in the middle for the current, so the key, the, I'll call it Horizon cuz it's the internal code name. When we were doing Postgres, we did it in super secret. Like if you look in our JavaScript, it'll refer to things called Horizon because we never wanted to talk about Postgres. We did not want to leak it because, you know, in South Park where we are located is the kind of core of technology, I guess, for San Francisco, which means like partly for the world, right? Like most tech companies have got some presence around here. And so we couldn't really sit in coffee shops and talk about Postgres all day long being PlanetScale people. So we had a code name for that. Um, so Horizon already has a few extra pieces of the stack, which is one of them being Exosphere, which is our Postgres-aware addendum to our edge network, which allows us to do the kind of buffering and failovers between proxies. We're running a modified PgBouncer. Um, it's one of the things you have to do for Postgres is bouncing. Like, you— I'm sure you, when you came, there's— in MySQL, we kind of, we have the concept of routing. You route to the right nodes. But now there's this whole other concept of bouncing for Postgres because of using processes instead of threads. means that Postgres can handle a lot less connections, but obviously people need to persist connections, so they need to bounce connections. So, we run that, and then that's orchestrated as sidecars. We have tier 2s of those coming soon. And then like the operator, and then a load of orchestration to handle the failing of those nodes. Naiki will be the test for Postgres.

Okay.

But it will be an opt-in. Now, one thing that we did, if you use the MySQL product, the Vitess product in PlanetScale. There's no way of doing it without Vitess. You always get Vitess. That was actually a mistake. Like, we got that wrong. For certain people that didn't need sharding, it meant that you'd get additional latency above normal MySQL, and it would add a layer of like incompatibility.

You meant quirks? Yeah. Yes.

Like no foreign keys, stuff like that. We do do foreign keys.

Correlated subqueries or something was weird.

Yeah.

Yeah, yeah, sorry, my bad.

Yeah, they've run most of that down, but it's still not the same. And that was a mistake, and we didn't want to make that mistake again. So now we actually are building 2 Postgres products, which is Horizon, which is just unsharded Postgres in all its glory. We're not really hiding Postgres. You get it, it works, we keep it up, reliable, highly available. Um, and then we're building Nike, which, uh, will be open source in the same way Vitess is. It's, it's because Vitess was built really for MySQL, around MySQL's faults and benefits. We needed to do the same for Postgres. It's the same people that build Vitess. It's the learnings that we've had from Vitess and PlanetScale, most importantly, like running Vitess in an applied— and I bet you've seen this too. Across the entire database world, there's this giant vacuum between core database people and then the running of those things in production. Like, those are very separate.

Yes.

I remember at GitHub, the MySQL team used to come to GitHub and I said, what's the largest MySQL installation? Because like, they really— the upgrade from non-GTIDs to GTIDs was just stupid in version 5.6. It was like, turn the whole cluster off and bring it up. And, and I, and I said like, what's the largest you like thing you run on MySQL? And it was mysql.com, which was just like a message board and an issue tracker. So it's like, yep, that's very clear to me. And that's basically the case for every database that's out there. They're just not running it in prod, like, like at a high scale. Like they're doing— they're geniuses. Like, I mean, the Postgres core team, the stuff they're doing, it's genial, but they can't do that and run at large scale.

Yeah, I mean, frankly, the number of companies running at that large of a scale rounds to zero when you add in all the other companies. Now, when we talk about machines, that's probably different because these companies are running, like you said, many tens of thousands of machines. But as far as companies go, we're talking like a handful that are running at that scale, fewer than 100. While meanwhile, there are millions of startups that are just like, give me a database, I got to go, go, go.

Exactly. Vitesse existed before PlanetScale and before the last 4 years where the company really grew to running these these workloads. And Vitesse has improved immeasurably. If you look at like the contribution, like graph of Vitesse repo, you see the amount of activity. It's because the team are incredible at seeing failures or seeing issues that happen for very large customers, which happens constantly with the amount of machines. Like we did this really amazing blog post, the true failure rate of EBS, because we have millions of the volumes. We see the actual, like how these things fail. And because we're running at such large scale, you can see these all the time, which means we get a great feedback loop between the Vitess team and production. Like, we had a minor, just a very minor issue with a customer a couple of nights ago, and the team patched Vitess and we were in prod within 2 hours with that change.

That's crazy.

It's wild, right? But this is— that is so rare. There's just so few companies that have the core database product and the production and the feedback loop between the two. Vitesse, like, so that learnings means that we've now got a fresh view on what we want to do with Nike. We're, we're not really talking too much about like its architecture right now, but it's inspired by Vitesse. And most importantly, it's the people sort of behind it. It's the sister restaurant of Vitesse, right? It's the new concept. Um, uh, and yeah, it's the core team that built Vitesse at Planescape are now, are now building for Postgres.

And so internally, you still employ a lot of the core team of Vitess, which is one how you're able to do that 2-hour story that you told just one second ago, um, is you've got a bunch of the core Vitess maintainers internally, basically. I mean, not sponsoring, you're paying them salaries, but they're working on open source as a part of their, their like PlanetScale-funded job. And now those Vitess maintainers that are internal to PlanetScale are now working on the, uh, the sister, which is Nakey.

Yes, Nakey.

I've got 4-year-olds, so Nakey is like, oh yeah, that's what they say all the time. Nakey.

Yeah, I always think of that. I always think of that, um, there's that line from Bill Clinton about Republicans in the woods hugging trees naked, and I always think about Nakey.

I'll be honest, I don't know that line from Bill Clinton.

I'll send, I'll send it, I'll send it to you. It's very funny. Some of them are still on the test. Like, we've got all of the active Vitess maintainers work for PlanetScale. So some spend their time on Vitess. Vitess is extremely important and deployed across hundreds of thousands of servers for many, many companies. Like, the market cap of companies running on Vitess is like over a trillion dollars and with billions of DAU. So it's incredibly important. So there's dedicated core people that don't think about Nagi at all, like that are just fully on Vitess all the time. And then there's a group that did a lot of the core parts of Vitess, like the query planning, the parsing, the bits that— like, we had an offsite recently and we did a whiteboard session, and when it got to the team that did like the query planning and all the relational, I just sat there thinking, yeah, I mean, 20 years in computers and you can learn—

you can know nothing compared to what these I mean, just step outside and grab a soda for this part because—

Yeah, exactly. Yeah.

Way over my head.

Right. I like— I literally think it— I, I did actually leave the room to go and get candy for everyone because it made my sugar drop just trying to think about all this stuff. So they're all doing the, uh, like that, that kind of thing. And we're continually hiring, um, for the team to keep growing. But yes, it's being definitely seeded by a lot of the people that have spent so much time thinking about. And we've actually— the cross-pollination of the Nike team is sort of half people that have spent a lot of time on production, right? So the operator that will operate Nike is in development alongside it with the people that have done a lot of production work, so that the practicalities of the two are even more closely combined, which is going to produce something really amazing. And they're moving very fast. Like, we have already got a very kind of early, narrow, naive case of sharding for Postgres working. You remember Beam, our sort of internal message board? Like, that's up next to be sharded.

Cool.

Which is hilarious. We sharded our 20-gig message board database. But, you know, it's just like refining through it. But it's gonna be a long time. Like, this— we are not trying to rush this one. When— you know this. When we ship something, it's just not like— we don't just put, like, push it out there for a few people to use. Like, it's— this is gonna— like, it will go into prod for a very large customer, be probably or possibly before it's open source. I don't know, it's far— we might change our mind on that one, um, because we— because it's that loop that's going to make it the best. Like, you know, there's certain sharded companies with Postgres that are moving their, their own self-sharded Postgres to PlanetScale for Horizons, so they'll They'll kind of replace their shards with Horizon shards, which means that when you see their queries and we have an ability to like kind of shim in, in with Nike and kind of make sure it works for their workload. So we'll do a ton of production work ramping up. We're probably a year at least before people see how well it works.

So without trying to like lure you into talking bad about other extensions and stuff, you looked at the The ecosystem, and there are, there are lots of like, let's shard Postgres extensions, there are non-extensions, there's a lot of solutions for sharding Postgres. So between you and the team looking out at that ecosystem and saying, not quite, what, what is the, what's the insight there that where you're like, not quite, we got to build something like Vitess?

The most common contemporary one out there is Citus. right, which is an extension, and it achieves a lot. Like, it, like, it gets a lot done, and it's, you know, definitely people have used it, um, at some, um, scale. It kind of violates the principles of a shared-nothing system though, right? We need something that lives outside of an individual node so that you can do things around those nodes, right?

Yes.

Like, it doesn't quite sit within our architectural principles. Actually, this is one of the things that we suggested in the early days was why don't we teach for tests, even with MySQL under the hood to fake being Postgres.

Hmm.

But that again violates our principles of send, like, get the traffic to the real thing that you are abstracting as easily as possible and let that query planner also do its own work, right? So, like, that really, we looked at what's around us and what's being built and it didn't fit the principles that we have for how we think we should work. Some of those principles are old and were developed in the— when Vitess was created. Some of those principles are new from running databases at scale and failing and doing things wrong at times. And so we needed to like kind of build within the walls of PlanetScale to have that really fast feedback loop. You know, it's really hard to build software in the abstract. Like, I don't know, when you see a new tech, right, like, and you want to play with it, I don't know about you, but I just sit there forever thinking of something that I actually would value using it with.

Yes.

And I know I never get it done. I just like— I really admire people who can just like mess around and like build anything and then throw it away.

That's one of the things I love about Prime the most, is he can just play with stuff and he's so good at stuff so fast. And I don't have that ability. I'm like, what actual outcome do I want? And let me build towards that. And he's just like, I'm entertaining, I'm gonna build with anything, everything, all the time, and people love it.

I can't Yeah, I'm the same. And, and I, and I think it's fundamentally because I've spent all my time building products or production applications, and it doesn't really feel meaningful to me unless it's going out to something with traffic and can break. And, um, that's kind of what Na'ki is going to be, right? Like, it will get these, you know, these underpinnings that will incrementally get adapted to people's workloads, and then we will be able to test it and, and figure out what goes wrong. So we needed to have that loop. that's controlled by us, developed by us, and, um, with our architectural principles and with the, with the architecture that we're kind of going for, that will also, um, be in unison with the operator that we're developing. And, and it will have its own proxy layer, it will be different to Horizon's. Horizon will have different needs. Um, we have to solve like various different questions, you know, extensions that is interesting in a sharded environment Like, it's just, there's lots to do. And so we just wanted to really take our crack at it and go for it.

So if I may, what I'm hearing is a lot of control for the sake of reliability. Is that directionally correct?

Correct. Yeah.

Yeah.

Like when we sell you a PlanetScale database, we are backing that. Like we are gonna give you 5 nines, which is our guarantee, which is a very, it's higher than any other database provider commits to. We've gotta stand by every decision. And also like, Why adopt someone else's project and then argue with the things they've wanted to do and show up and try and make a change or fork it? You fork it and make it more successful. That's not— doesn't seem very fair. Or, you know, you know what I mean? Like, and, and try and change it. I, you know, I really just wanted to have our control and to— in, in the kind of— in the, in the, um, pursuit of just pure reliability and performance.

Mm-hmm.

But then we will open source it and, and get community contributions when it's ready.

How has the, how's the, like, I guess the tone or the tenor or the velocity of the actual company in the day-to-day changed? Because when you're just a MySQL company, you're competing with, let's be honest, just RDS. I mean, who else is out there doing a MySQL platform?

Yeah.

I can't, I can't think of many or any. Now you're competing with the entire world. And so what does that do inside of the company? Because there are as many, as many Postgres hosts as, you know, you could wag a stick at. So like, how does that change the culture inside of the company and the, like, the velocity and maybe the pressure, maybe not? Like, what have you seen inside?

That's a really great question. That's a really great question. It's definitely reminded us like the need to stay nimble and stay humble, right? There's like many great companies out there. Doing really good stuff in the Postgres community, and it's different to us. And we could be wrong, like we could lose, like something else could completely— we have an opinion about how databases should run and operate to an end goal. People have made other trade-offs and made really great businesses not caring about those things, right? Or thinking about them differently. So it gives you like, it gives you the need to stay humble and, and still really care and be so user-focused because the the community's shifting the whole time. Like, there's no, there's no room to like stay buried inwards and just arrogant to what people want. We have to like really adapt. Like, there's just no— like, there's so many times where we've, um, come across a, like, a thing that is more standard or done in a certain way, and we've had to catch ourselves from going, oh, that's just dumb, like, we just won't do that. It was like, okay, well— I mean, if most people are doing it, we have to have an answer for it, right? You can't just, you're wrong. You remember, like, the foreign keys thing, so we can talk about that a bit. Like, we, plans, Postgres did not, sorry, Vitess did not support foreign keys, 'cause they're really hard in distributed systems. You shouldn't do them in a sharded environment. No one does do them in a sharded environment, but people really wanted it. And we kept saying, no, you don't need them. You just don't need them.

Mm-hmm.

It doesn't work. It's against the vibes again, right? So it's about being much more active and humble, and there is a lot more competition. Although the benchmarks, I think, and the immediate— like, if you look at some of the comparison posts that people have put out already in terms of how more reliable we are than the others, I don't think there's many doing it the way we are. Um, so we're different enough, and I— the size of the Postgres market and the size of the database market means I think there'll be 5 or 10 companies that just absolutely crush it for the various things that you need. Like, truthfully, like, there's some, some companies building full backends as a service that have Postgres included, right? That is great for certain people. I know the current customer base of PlanetScale couldn't use something like that. Like, it's too— you, you know, right? Like, you see that a lot of big customers, they have PlanetScale for the database because the depth of the database needs to be so good. The analytics cluster is going to be ClickHouse because they've gone so deep on that. Lambda, or, you know, you see, like, it's just going to be— there's certain companies that are at such a scale that they need the best of every single thing, and that's never usually built by one person, not even RDS, right? Um, so I think our ICP and the customer we're going for stays the same, and it's different to most of the other non-large cloud Postgres companies we're seeing. It's actually, I would say, 50% of the people that have migrated successfully to PlanetScale and are loving it, of the Postgres version, are still coming from RDS and Aurora.

I think, um, I think the thing that PlanetScale has always had as like a, uh, unique value proposition is frankly, you're just a database. Like, that's it. That's it. And that, like, that works for a lot of people. That doesn't work for a lot of people.

Right.

And that's okay.

Like, that's okay.

I can't, I can't install PlanetScale auth. I can't install PlanetScale image uploads.

Right.

And that's okay.

Yeah.

And it's okay that you, you may go somewhere else and do like have that entire, like basically, you know, framework, that whole stack built into your database provider.

Yeah.

So I think those are, those are just different games that are being played. And I have always seen from the PlanetScale, both when I was internal and external, like just is, is not meant to be a pejorative, but we're just a database.

Right. Right.

And we do that part really, really well. And if you want auth and uploads and whatever else, like, there are great providers that do that, but that's not what we do.

Yep, exactly. We don't presume to be good at anything else other than databases. And any company has to spend innovation points somewhere, and we get basically the same amount, right? Like, eng teams get too big, coordination becomes too difficult, like, and at the end of the day, you could still if spend 100% of your focus on one thing and be really good at it, right? 100% of our focus is just doing what databases do, and that's incredibly important to us. There was a bit of a meme after the Postgres thing of like, what are you going to do next? People took that really seriously. Like, I'm— the amount of like other backends people emailed me saying, please do this next. I even had the people like, well, people want us to do Elasticsearch for some reason.

Oh yeah, that checks out.

And I'm like, you know, they do a load of good stuff. I don't think I can take what we're good at and learn to be good at running the JVM in prod. Like, that's just not going to be what we're good at. Like, other people are better at, you know, they're just going to be better at that stuff, you know. Um, and so we try and, like, we're trying to be humble about this and saying, like, you don't want us doing other things, you want us to spend 100% on databases. Luckily, Postgres and MySQL have similar-ish architectures for how a cluster runs and definitely similar principles in that way. Um, so a lot actually surprising amount of transfers, and we're going to just stay there. And we might earn the right to do something else, but I— our internal bar for doing anything— like, again, that's why with the Postgres thing we had to convince ourselves we could do something. We might have done that hack where you can convince we had nothing for this audience. Like, it's been— we've been bested, and that's fine. We shouldn't do it. The benchmarks told us that that's not true, and that's why we're here. But we are going to stay just insanely focused on, on Postgres and MySQL as relational databases, and maybe earn the right to do something else long in the future.

But My guess, I just want to go on record and say, long in the future, if you do consider something else, it's going to be Redis. That's just my guess.

Maybe Redis. We could add an extra thread to Redis and go more—

That feels way more approachable than Elastic to me.

Right.

So, this is being released on GA day, and we're recording this just slightly before, obviously, so we can get it all edited and stuff. So, talk to me about, talk to me about what is GA now, what people should do, what it took to get here. Just tell me, like, uh, let's celebrate GA for a second.

Yeah, it's been a, it's been a journey. It's been really fun. It's been tons of learning. So what's happened since we announced Postgres? So we, we had a waitlist, thousands and thousands of people signing up for that waitlist. Like, it was just awesome just to see the reaction was huge. I think from, in terms of like volume and, and reactions, Publicly, it was probably the biggest launch that we ever did, and it was really just launching a signup page. Um, yeah, I think we took over sort of tech Twitter for at least the day, maybe even the week. Um, and people came inbound, and then we'd see these, like, people would email, sign up, and DM me to try and get access.

Trying to jump the queue.

Yeah, and we did let people in. Like, we, we have a number of very serious companies running production workloads today. Because we weren't in beta. Like, we wanted to do— this didn't get— we didn't communicate this very well. I think it's very, it's very confusing. Um, we don't really put much stuff in beta. If it's in the data path, you can't really do beta, you know.

Right.

It's like, it's not the right— you remember this too. It's like, it's really diff— like, you can do certain features, then we'd throw them behind a really explicit opt-in that made it really clear that this is not like— this is a beta feature, you, you know. Run with it, um, you kind of go, go use it with that knowledge. Um, so we put it out in what is production, you know, production grade. And so we have a number of companies running full production on, on it today, which we're really excited— some very large and exciting companies, especially some very big AI ones. Uh, and it's gone really well. Um, they're observing, you know, significant performance improvements and reliability improvements against the other providers they've migrated from, which, I mean, if we weren't doing that, what are we here for? We would've stayed private preview for a lot longer until that's, you know, done. We've shipped a lot of things based on feedback, like tons of extensions. Insights works fully now. Suggestions are coming. Not all of the suggestion types are there, but most, most of them. It's very, very robust multi-AZ single region Postgres right now. Right now. Like, the things that are coming soon will be cross-region replicas so people can bring their data out to the edge. Um, we've got, and like, extensions like Zero and things like that work really well, which is very, very popular with folks. Um, yeah, it's robust. You can sign up and use it, and, you know, for $39 a month you can get a very reliable and robust Postgres offering that does just that. Like we said, you know, it just does the database part of what you need. There's, we've really refined the migrating in story. Like, like I will say that the replication story for Postgres is different and, you know, less friendly, I would say maybe than the Postgres one. Like, what's your take here? You've, you've analyzed both now. Like MySQL just seems so much easier to like set up and never think about.

It does seem to me way easier to set up and never think about. And one of the things that has served me so well is I've spent most of my career up at the application level.

Yeah.

And so I, I really haven't had to dive into the, the nitty-gritty of hosting very much, but that's one of the reasons since, since my youth I picked MySQL is because it was like, oh yeah, this thing's rock solid. It's just gonna— it's easy. I'm just gonna run it forever. Um, so yeah, hearing all of these Your stories, if you're talking about the foibles and pitfalls of running Postgres at scale, makes me glad that I still live at that layer. That's pretty compelling for me to live up there.

Yeah. I would say replication has been one of the things that we've run into issues with the most. The primary is caring way too much about what the replicas are up to, like the concept of slots and just these sorts of things, and that makes the import story a little bit more difficult. But we've built an open source— in fact, Richard did this, which was like built and open sourced a ton of the scripts to allow you to migrate in. The good thing is because it's— we don't have a layer like Vitesse in front of it, it is just Postgres. You can use a load of this, the tools for like getting in. We've had to like figure out some of the quirks of platforms like Heroku that don't expose logical replication so that people can migrate that way. So it's like really been a journey of just like You can only know what you know, and then when users come in, they break it in like a million other ways, and then you figure out how that's going to happen. Um, we've added a lot more configuration than is in the— than the MySQL product. Like, you can just configure so much more in the Postgres product. And so we've got like new UIs around being able to like change configuration. It deploys out, you get a tracked— you get a tracked list of everything you've deployed, when it happened. And so people are really enjoying online upgrades for Postgres, which is not really— not really any other vendor has that, which is another kind of wild thing to me, which— but we, we're allowed— we're giving people the ability to change variables and configuration of their cluster really simply without taking their cluster offline, which is just really robust operations, which is like, we just don't— we refuse to make offline modes of any— there's no offline operations at all in PlanetScale. Which we're really proud about. Um, and so it's like lots of refining and running through those error cases. Um, and so yeah, the journey to GA has been lots of discovery, lots of hiring up of Postgres experts that are here and onboarded and, and, um, really making customers successful. If you look out there, the testimonials are out there of people that have seen success. You can see the graphs, you can see the kind of, um, just the improvements they've seen. And so it's been really fun, and now it's about shipping. The other one is demand. It's— I mean, we, the— we've had to hire more people to answer the phone for sales. Like, we just can't keep ahead of it. Like, the, the, the— just the— we have like an MRR bot that shows just Postgres MRR, and the like daily growth of it just growing rapidly. Like, having 7 figures of ARR on a product that's not even shipped Yeah, it's pretty nuts.

Quite good.

Yeah, like just from literally hand rolling people in has been really good. And, and, and I think that's the thing that I've been most proud of is that people come in wanting PlanetScale for Postgres, which means we did something right with our brand to get here.

Totally.

And have them understand what that means. And now it's about holding that high bar of delivering it while we grow. I, you know, it's interesting, like we We have single clusters, MySQL clusters that do more QPS and have more data than some of our competitors. And like single clusters have more nodes than other database companies have for their entire fleet of Postgres nodes. That is huge. But some of those, like that could be one customer that has that volume for us.

Right.

Now we're gonna see a different type of volume, which is sheer numbers of individual companies.

Right.

Right? That's the crazy thing about the database world. We can have someone pay us $10 million a year and they have 6 admin accounts, right? Like, user numbers don't map to anything in the database world. Absolutely. Like, like, they will— those, those 6 people will deploy— will manage more data than an entire Postgres host will have for everything, and they could have 100,000 users, right? So it's a different world for us now though, because we're seeing so much more diversity of, of smaller databases or, or companies with lots of small databases. It comes with UX and, and other interesting challenges that we have to rise to the challenge of, and we will, but it will take time and we'll, we'll have to learn and stay humble. Yeah.

When you were talking about all those configuration options that you exposed, I thought, boy, that's a bunch of forms and a bunch of state and a bunch of tracking and a bunch of just like Boy, yes. Good. My heart goes out to the Rails team that's building all that, you know, has to keep track of all of that. Um, and documentation too. Plenty of documentation. Yeah. Sorry, Holly. Uh, so, um, I have one, one more announcement that's pertinent to me here at the end. But before that, uh, what is your, like, what is your, I guess, charge to the audience, call to action for people like Signing up, let's say that somebody's like, oh, PlanetScale has Postgres. I'm gonna go sign up for GA now. What is, what would you say to these people?

You've heard me say a lot of confident things about the stuff we've built and, and I talk a big game. Come in and hold that standard for us and see how we do. And if we don't, DM me, email us and give us the feedback. We'll be there. We, and we love engaging with customers. We love to hear how we if we fall short in any way. It's really important to us to get that feedback. But, you know, if you think, if you, if the provider you're using isn't as reliable or isn't as performant or is too expensive or whatever, come and give PlanetScale a go and see if we really do match the standard we think we do. And if we don't, let us know.

Okay. I like that. Um, so the final announcement, which is, you know, this is, this is self-serving. This is exciting for me.

It's a great journey.

Um, yeah, you are entering into the Postgres world. And exiting the education space. Now hang on a second. When I was at PlanetScale, I did this big, massive MySQL for Developers course.

Fantastic course. I have to interrupt you there and say it's a fantastic course.

That's very kind. Uh, that course is changing hands. So I got a message out of the blue like a couple weeks ago, uh, right before I was at a conference, and I got a message from Sam Lambert, and I was like, I haven't heard No, I haven't heard from Sam in a minute. And he says, do you want the MySQL course back? Well, yeah, I do. Hell yeah, I do. So what's going on there? Talk to us about that. The course is coming home.

It truly is your course. Like you poured your heart and soul into that course and it has your personality all over it. You've done an exceptional Postgres course and now, and you've built the MySQL course while you're at PlanetScale. you know, just like we have one thing in the Postgres world and one thing in the MySQL world, it didn't feel right for you not to have that yourself. And because you did such a— truly, it is an incredible course.

Thank you.

Everyone listening should go. I, I learned, like, I, I've been a MySQL DBA for like 20 years, like, and I learned so many things. It's just yours. Like, you really, like, you— everything you do comes with your personality attached to it. And with you not being at the company anymore, it felt really important to me that it was yours and that you can add to it, change it, modify your voice as part of it. And so, yeah, there's just— it's, it's in its rightful place right now alongside the Postscripts course. Um, yeah, part of your suite of amazing things you put out. Like, again, you've done it again with this podcast, right? Like, I've listened to nearly every— I'm a bit behind now just due to sheer, like, getting ready for GDA. Yeah, yeah, busy, busy. But, um, but like, It's just another really incredible production of yours, and you're building up this kind of library of things that you've built, and it just felt right to sit among all of the other treasures that you're building.

Well, that means the world to me, and I was, I was truly shocked and so grateful when you reached out and offered that, no strings attached, hey, do you want this thing back? It's yours. And I was like, yeah, I really, really do. Um, so I, I'm very grateful to you and, and Holly for helping me get that transferred over and it just, I, that means the world to me. So I now have, I now have the trifecta. I've got SQLite, Postgres, and MySQL.

Oh yeah, you did the SQLite too.

That's right. Yeah, that's really right. So for the—

what's next for you? Redis? Are we doing Redis?

Yeah, Redis is next.

Yeah, we'll have fun with that.

So what's next for me is consolidating. Everything is going to go on databasesschool.com. So the name of this podcast is Database School. I've done standalone sites Mastering Postgres, High Performance SQLite. All of that content is coming to databaseschool.com. And I'm— my goal is to be the destination for database education. And so I'll bring in the MySQL for Developers into that brand as well. And then, yeah, I'll do Redis. I'll do— right now I have Mastering Postgres, which is kind of like intermediate and up.

Yeah.

I'll do an Intro to Postgres, which is very, very short and like talking, you know, to people who may be new to databases altogether. So trying to round out the content offering there. But yeah, I thought like it would, it would be malpractice for me to not have a MySQL course, but I don't want to compete with myself. Like that kind of sucks. And so this is just, I'm very, very grateful. So thank you for reaching out.

No, of course. And we're grateful you made it. It was just amazing. It was so good. And everyone's going to really enjoy it. And yeah, like, if you're a DBA listening to these, you should buy these courses for your engineers. Because if you are spending time answering their application-level questions, like, it's a waste of your time. They should just get trained.

Well, I appreciate that. So, this is nice, fun, full circle moment. It's been a joy to have you back here. And I'm excited to see the response for GA. I have no doubts that it will go very, very well. So, where can people find you, keep up with you, keep up with PlanetScale and any final words you want to leave.

If you want to try our products, planetscale.com. If you want to tweet at me or debate databases with me, it's @isamlambert on Twitter. And yeah, I love engaging with the community and kind of hearing your feedback. And my DMs are open if you have questions about anything we're building or need assistance from us. Uh, we've been helping a lot of people migrate, um, as they grow. And so we're here for you if you need us.

Thank you for being here. And to the listener, if you're on RSS, there's a YouTube channel. If you're on YouTube, there is RSS. You can put it in your podcatcher. And you can find it at databaseschool.com, which is where you will also be able to find MySQL for Developers very shortly. Sam has done his job. I still got to migrate it over to my platform. So it'll be on MySQL for— I'm sorry, it'll be on databaseschool.com very shortly. But we'll leave links to everything down below. And for y'all listening, go try out GA. And also, here's my ask to the audience: who else should I have on? Let me know. This is very much a— this is very much a— I reach out to people and find interesting people doing interesting things. Nobody's ever paid to be on the show. So let me know who else is doing interesting things that I should have on the show. Until the next time, see ya.

Bye.