Scaling MySQL with Sam Lambert from PlanetScale

Truthfully, I don't think many databases have been built for developers. They've been built to operate and have built— been built for operators. But the kind of developer experience and the daily life of using most databases kind of sucks. It's serious to go and release database software. You can't mess around. You can't just yeet it out to production. And one of our company values is no passengers. We mean that.

Hey folks, this is Alex DeBrie. I'm super excited about today's show. My guest today is Sam Lambert. Now Sam is the CEO at PlanetScale, which I think is one of the more interesting cloud-native database companies out there right now. Uh, we just cover a ton in this episode. We talk about what I consider like the 2 parts of PlanetScale. We talk a lot about Vitess, which is the technology underlying PlanetScale. So we talk about the history of Vitess, the architecture, how it fits into PlanetScale, all that stuff. And we also just cover a lot of stuff about how PlanetScale is such a high-performing culture, right? And you see this in their Their cadence of feature releases and the great educational content they put out, just their, their nice design aesthetic, all that great stuff. So we cover that in today's episode. This is the first episode in this series that I'm doing on cloud-native databases. So if you're interested in that, make sure you subscribe to the podcast. If you have guests that you want on, if you have notes on the show, anything like that, feel free to reach out on Twitter, email, whatever it is. Otherwise, let's get to the show. All right, Sam, welcome to the show.

Hey, thank you for having me.

Yeah, absolutely. Glad to have you here. So you are the CEO of PlanetScale. You've got a lot of great experience at GitHub and other places. Can you give listeners just a bit of your background and what you do, your history, things like that?

Yeah. So like you said, CEO of PlanetScale. Before this, I've worked at 2 kind of very significant companies that were very significant Uh, in tech. I did a short stint at Facebook, uh, on the traffic engineering team, which was just an awesome experience seeing that's the scale of Facebook. No, like, you can imagine the scale of Facebook and then quadruple it and it's still not there. Um, it was just absolutely incredible. Uh, and then before that I was at GitHub, which, uh, also another site at scale. I think, you know, around the time I was there, It was like the 32nd largest website on the internet, I think. And I was there for about 8 years, started as the first database engineer at the company when the company was still small, back in the kind of no managers culture, very flat organization, and just a very disruptive, interesting, and cool tech company to be at. And that's kind of, I would say, where the formative years of my career were spent. And now I'm at PlanetScale trying to solve some of the problems I've seen plague every company that I've ever been at.

Yeah, absolutely. I'm sure you've seen some pretty amazing stuff. So I'm excited to talk about PlanetScale. And just for the users, like, I think of PlanetScale as sort of like 2 different things, right? It's like, originally I think of it as like Managed Vitess, which is like a horizontally scalable MySQL solution. So if you have these hyperscalers, like, GitHub, like Facebook, YouTube, somebody that has terabytes and up of data right now, they need to horizontally scale their MySQL. And that's what Vitess is doing, like an open source solution, but PlanetScale is providing a managed version of that. But then in addition to that, there's just like a bunch of like other delightful database add-ons that seem to have nothing to do with Vitess, but like are available to all your users. And that would be like, Hey, a better workflow for schema changes, you know, thinking about like non-blocking changes, branches, deployments, reverts, things like that. Also PlanetScale Boost, which came out recently-ish, right? Which is like a materialized cache with automatic incremental updates all the time. PlanetScale Insights, which just gives you like very good query visibility compared to like other solutions out there. I guess like my question is, was it always this way or were you all— did you start off with like, hey, we're going to manage Vitesse and then realized, we can provide a lot of stuff to other downstream customers as well. Or what's the story there?

Yeah, the original sort of vision for the company was that Vitesse itself is just an extremely powerful database technology, of which there's not many like this on the market, or not locked inside kind of a single cloud platform. There's a lot of power to being able to take a technology like this and deploy it kind of anywhere. across clouds. And so it was originally like managing that, and that, you know, picked up some really great logos and customers that are Vitess users. And, you know, if you've sent a Slack message today, that's gone into a Vitess database. Slack have been very vocal about how they're a huge Vitess shop. GitHub's primary database is Vitess. And then we have people like HubSpot, Um, Roblox, Etsy, all of these companies that have been very vocal about their use of Vitesse. And that's one thing that first of all is just really cool to see contributions from major companies running this technology at scale. Means that the kind of the rising tide raises all boats. You, you kind of get bugs, you get decent, um, incremental features built on top of the platform that's useful to a broad audience built by very, very talented engineers and maintainers. But then we wanted to see way past that vision, which was build a— like, you know, in our first kind of tagline was database for developers. And everyone, I think, was like, well, what were databases for? And yes, it was like a slightly kind of tongue-in-cheek, but truthfully, I don't think many databases have been built for developers. They've been built to operate and have been built for operators, but the kind of developer experience and the daily life of using most databases kind of sucks. And so we really wanted to improve it. And I did a little post on this recently talking about developer experience, and developer experience is not really just like floopsy fun little things and— a little avatar.

Dark mode.

Yeah, dark mode or whatever. It's doing what you're supposed to do for years and years and years on end. And so bringing a database product to market that has great developer experience requires being a great and proven tested database and doing what it's supposed to do. And having Vitess as our foundation enables us to do that. And then we build all these incredibly cool features on top that make operating and using the database, really joyful even. We came out with database branching, we even patented that. It was just crazy that we patented it and were able to in 2021. Not that we'd ever probably use the patent, but it's just a kind of marking your territory saying, this was us. We came out and it was crazy that that was done so late. And I kind of put my head up after being at GitHub and realizing databases just had not moved. They were just still doing the fundamental things. And so I'm proud that we've brought a huge shift in terms of the minimum bar for features. You can see that by pretty much every other competitive company is scrambling to add these features while we build the next set.

Yeah. Did you have, when you were at GitHub or teams at Facebook, I imagine, did they have branching-like workflows? Like, I remember Ghost coming out from GitHub to help with like database migrations. Like, did big teams have those sort of tools to make branching pretty easy, but it's just like not available for the masses?

We never quite went as far as branching, but we did get to good staging environments. I think a lot of— I think what the impression that GitHub left on PlanetScale is the unrelenting drive towards shipping software constantly and deploying software constantly. So when I joined the company, the company had grown really rapidly. And this was around the time that we raised that record-beating kind of Series A round. The company was bootstrapped and— was just winning hearts and minds of developers everywhere. And so things had grown really quickly, and the database had kind of— there was like the usual kind of patterns and things that had come together inside the database, and that had to be fixed. But it was— we could never— we never had the option to fix infrastructure problems while slowing development down. And that to me was such an amazing, tough, but amazing lesson to learn because it taught you the appreciation for shipping velocity and what shipping velocity does for a company. So when you see things like branching, like deploy requests, all these things in the plans, it's all in service of you shipping really quickly and taking away fear of the database. We really believe in just push it to prod, push it to prod. And if your database allows you to do that, then you will gain huge speedups and impact for you and your user base.

Yeah. What's the history of Vitesse? Was it— is it YouTube that developed it?

Is that right? Yeah. So YouTube was growing rapidly. It's now the 2nd most visited website on the internet, 2nd most trafficked search engine, and has, what, 2.5 billion monthly active users? And so they were using MySQL naturally. I mean, if you look at the top 100 of the internet, it's pretty much exclusively powered by MySQL. And they were scaling up and they needed a solution to go and sort of shard. And it was originally like a proxying project, then became sharding and all these things got built into it. And yeah, they just did an outstanding job. It was built on the earliest version of Go. If you look up in the Go history that they talk about the early Vitess creators being instrumental to the Go project, even because they were giving feedback on Go 0.1.

Yeah.

So it's one of the— what people don't realize is one of the oldest Go projects in existence. It was also built on Borg, which is the predecessor to Kubernetes. So it was built in this highly hostile environment. Everyone was used to running databases with RAID controllers, and it's like, oh, we're going to recover that machine and bring— not in the Kubernetes or Borg world. So that meant that they built an extremely resilient cloud-native, air quotes, database technology, which could recover from pretty much any type of failure. And then they put very smart Google engineers spending nearly a decade to go and build it. And now we build on top of that extremely strong foundation, it trickles down in so many ways and why it's important. But yeah, it was picked up by Slack, picked up by GitHub, picked up by so many other companies and kind of became the ubiquitous MySQL scaling solution out there.

At GitHub, before pulling in Vitess, did you have kind of like a homebuilt horizontal scaling solution or I guess what did that look like?

We were doing the classic pattern that everyone does, which hopefully is dying, not at GitHub, but I mean, generally as an industry. Well, first of all, we ran our own data centers, which is still kind of a good advantage because we would be able to buy very expensive, fast database servers. And then we had, you know, when I joined, we had 3 database servers and now it's a significant amount more, but it kind of got partitioned. So we would end up taking like the largest tables that were always causing problems. The notifications table was a pain, the stars table was a pain, the statuses table was a pain. Statuses table, at the bottom of a pull request you get all those checks. It's like all of the different statuses like a pull request can be in. We just hammered the database constantly, so we moved them out into their own clusters. They freed up the rest of the database infrastructure in terms of like buffer pool and contention and resources. This is a really common pattern and we see this a lot. A lot of people come to us having kind of split up their main database cluster. Problem is, it's not always better for availability, right? Like, even if you have high availability in those separate clusters, you're kind of, you're pinning a lot on 3 or 4 masters across 3 or 4. So we were doing that, and then eventually, and we made some, built some awesome tooling, right? Like, we continued the maintenance of Orchestrator with Shlomi. We built Ghost. We built lots of really— Skifree, lots of really cool open source projects on top of MySQL. And yeah, like you mentioned, you see the DNA of those are now in Vitess and they're in PlanetScale, the product itself. But we've been fortunate enough to take it way further than we even imagined at GitHub.

Yeah. And what does active work look like on Vitess right now? Is that like, I imagine PlanetScale employees are major contributors. Like, are you sort of the leading contributor? Is that still some YouTube folks? Like, how does that break down?

The project leads are primarily at PlanetScale. You know, we kind of craft the project. We do that. We manage the releases. We maintain Vitess essentially, even though it is a CNCF project. It was, I think, I think it still remains the only graduated database in the CNCF and was a very early project to do so alongside Kubernetes. So yeah, we maintain it. It's a lot of work. We have to manage releases very carefully. When we deploy our software, we know it's going out to billions of users. One of the biggest sites in China runs on Vitess.

Yeah.

It's serious to go and release database software. You can't mess around. You can't just yeet it out to production. It goes through rigorous testing. Every push gets tested against every modern framework. We have all of these acceptance and regression tests, then it's rolled out. But companies like Slack, they have maintainers there, they're still very active maintainers. The Slack, even for a hyperscale backend MySQL scaling solution, we still have a Slack with like 3,000 people in it. We have the monthly maintainer calls, the weekly check-ins with all the maintainers. This is a real big active open source project that gets— has huge impact at a lot of companies. And so we get a great set of folks show up every time to kind of manage the project. Then we have the features in it that get released or setups for new features that are coming for PlanetScale. So it's a big operation to maintain for tests.

Yep, yep, absolutely. Okay, let's talk Vitesse a little bit. Walk me through the high-level architecture. Like, what are the components in a Vitesse cluster?

So there's a few different components of Vitesse. Firstly, VTgate, which is what kind of could be known as a proxy, I guess, is the easiest way of describing it. VTgate is like what terminates your connection and determines the query routing and where it has to go. So Vitess sits on top of MySQL. We run real MySQL under the hood. That has a number of benefits. First of all, MySQL is exceptionally proven and trusted. It's like, what, 26, 27 years old now? Incredibly reliable. Even like a decade ago when I was playing around, I've slammed MySQL servers with writes, ripped the power cord out of the server. and everything that was acknowledged was still there. It's incredibly robust and good and trustworthy. So we build on top of MySQL, but we have a layer on top, a proxying layer, which means we can't support everything that MySQL supports. We're working on the last few things that don't make us 100% compatible. Foreign keys is one of the major ones.

Yeah.

We get a lot of noses turned at us for not having foreign keys, but we've solved it. We just have to keep ramping up towards being production grade, which is a very high bar.

And so that VT gate, is that just emulating like the MySQL parser, or is it using the MySQL parser at all under the hood and hooking into that? Or is it— do you have to basically replicate all that MySQL parsing?

Yeah, fully emulated in Go.

Okay.

Um, so it's not an easy task. Luckily, a lot of time is spent at Google just running down all of those edge cases. And in the majority of cases, unless you're doing something weird, like, your app is pretty much gonna, gonna work. And then certainly for the benefits you get from sharding, or it becomes worth it. It's not like a giant— there's some more on the Postgres side. There's like some really interesting Postgres databases, um, that are kind of Postgres wire compatible but not fully compatible, um, which causes people some like weird edge cases in the long run. We try and avoid that. Like I said, we test all of the frameworks that are kind of out there and popular to make sure they work well, and foreign keys is really one of the last things that we have to fix. So VTgate kind of looks at the query, looks at what has to be done to serve that query, and then breaks it up among shards if necessary to go and retrieve that data. So you can have VTgates horizontally scaled. So that's one reason we can claim unlimited connection scalability. And what we mean by unlimited is if you keep adding BTK nodes, you can keep adding connection resources, basically. We have some people running literally millions and millions of connections to their databases. That itself, you're talking about developer experience, that itself is a subtly extremely difficult problem, right? Like most people have to run proxies in front of their databases to terminate and handle connections. And then if you're using Lambda or various kind of worker architectures, you can really thrash the database with connections, which can be very painful and can just exhaust and waste a load of resources on your database, basically.

Yeah, absolutely. So when I'm in the PlanetScale dashboard and I look at my database and it says like load balancers and it shows my database, Is the load balancer, are those the VTgates essentially?

Yeah.

Or is there even— okay.

Yeah. That's what I'm interested in. And then we have an edge network that kind of routes you to those right, correct places. And you'll see there's a bunch of benchmarks out there that show us to be the fastest or one of the fastest because we have like a fully deployed global edge that then routes you to the right VTgates.

Gotcha. And then what are the storage nodes themselves?

So storage nodes, you have vttablet.

Okay.

It's a sidecar process to MySQL that receives all sorts of events and serves queries and can rewrite queries and can protect and buffer the database, all next to MySQL within a shard. Gotcha.

And okay, and that's running MySQL, and they're running like the latest version of MySQL? Is it sort of pinned to some previous version? Like, what is that?

Oh, so that's one thing we're quite proud of is because of how advanced the architecture is and we can kind of fail over and manipulate the cluster really simply without outages, we keep people very up to date. We manage rollouts on very large customers. to help just make sure that they don't have regressions. But most of the time you're gonna be running on the latest, if not very close to the latest version of MySQL. The only— it's so funny, if you're an RDS user right now, the only online way to migrate from 5.7 to 8 is to move to PlanetScale. Within Amazon, you can't online migrate to the latest version. Without an outage, without downtime, which is—

That's quite amazing to me because, like, you know, how many RDS users they are, I can't believe they couldn't, you know, build some of that migration tooling themselves.

The power of being the incumbent, having everyone trapped. Yeah.

So if I'm like a typical, like, say I'm on like, you know, the Scalar or Scalar Pro plan, I'm not like an enterprise that has, you know, something more managed and custom, but like, are you just automatically upgrading the tests and MySQL for me, or am I clicking buttons to say upgrade to the latest?

We do it all for you.

Okay.

Any PlanetScale user will not have realized that we continually upgrade their software for performance improvements, for all sorts, without them ever noticing the impact. It's really hard to— it's not easy. It's really— it takes a lot of work and a lot of automation to do these kind of upgrades online. We have customers that have come to us purely because they couldn't solve that themselves, right? That's great. That's cool. That's value we provide. And we do tens of thousands of failovers every week, not because the software is failing, but because we use them as a mechanism to upgrade, resize, repack, do all of these various things that we do for our infrastructure. And it just works very well. Gotcha.

Okay. That VT tablet, or sorry, VT Gate, which is the proxy layer, it's parsing that query, and figuring out which shards it needs to go to, does it rewrite that as a MySQL query again, or is it able to pass in like the parsed query to the MySQL instance?

It depends. It sometimes does rewriting because it might have to scatter across a number of shards to go and aggregate the data and pull it together. Sometimes, like often a very common rewrite is it adds limit clauses so that you don't just go and thrash the database with too many rows that you need. It does all these things that help guard against errant application behavior. ORMs, looking at you. We love ORMs, but they, you know, they can get a bit unruly. So there's a whole number of things that it does. Like the team that built the Vitess query engine and the parser, are just geniuses, and I wish I understood with the depth that they do, but the work they do is absolutely outstanding. Writing a query planner is very, very hard.

No kidding. No kidding. I can't even imagine. Okay, so that high-level architecture. So we basically have the stateless proxy layer that, you know, has some config about where these shards are assigned, and then all those shard layers that run the storage nodes. Is that a pretty common setup for, if we're talking about any of these distributed databases, whether it's Cockroach or Amazon Aurora or Dynamo or Cassandra, is that the same high-level architecture? Are there any major differences between Vitess and some of those other ones, or even groupings of patterns there?

I think at a very high level, they're all going to have that same kind of push down to storage node, load balance somewhere. We've chosen for kind of a shared-nothing sharded model. Some folks kind of do an automated sharding model where they kind of KV across around a load of nodes. That has certain trade-offs. We choose to have a single writer node per shard rather than multi-master. That's an interesting trade-off. It's a trade-off made for scale and safety primarily, right? If you have Multiple masters, then you have to do conflict resolution. If you're doing that across like a network with latency, you can dramatically slow down the throughput of your database. And that—

What are the ones that are doing multi-master? Would that be Cassandra? Are there—

Cassandra do it, but then they have like various consistency tweaks. Cockroach do it. Spanner does. But again, like they all have these trade-offs under the hood about really understanding kind of where you're placing your data, how you're doing it. With us, each shard follows a primary machine, basically. And that means you can have very, very fast writes that get sent over. They get sent over semi-synchronously, which means they have to be acknowledged by at least one other node in the cluster so you know the data is safe. you're not trying to gain quorum among 3 or 6 or whatever nodes.

Yep. Gotcha. So what is that? So tell me about that on the flow of a write request. So when it's going to come in, it's going to hit the VT gate, route it to, you know, ideally one VT tablet and—

Straight to the MySQL primary. And another part of the architecture that's really important is VT Orchestrator, which is always watching your cluster. And making sure that it's always in the correct state to serve queries. So it always has a master, it always has at least one replica, or as many as you've configured, and make sure if that master disappears, replica gets promoted, new one is created. Like, it's always making sure that your cluster stays in the correct state to go and do the job you need. And, and, and this is where, like, a simpler— it's much more easy to reason around, we wrote to one node, within a few milliseconds it appears on another node, versus we have 2 copies of our application talking to 2 separate databases and now they are conflicting. That inside a code base, to reason around those types of, um, problems is much, much harder and it's really not needed by many people. If you have a low-volume kind of metadata store that has to be strongly consistent absolutely everywhere, you can choose some of these architectures and they'll probably work for you.

What does the typical topology look like for a, you know, a big service? And especially within a shard, how many replicas are we talking behind that primary and are they Are they same region? Are they same data center? Are they cross-region? What does that look like?

So by default, every PlanetScale database is multi-AZ. We put replicas across each availability zone should an availability zone go. We have a very multi-layered approach. So like, we don't trust EBS fully, right? Like, there's some, like Aurora, for example, they rely on EBS's underlying replication to To replicate the data. We do too, but per node, but then we have multiple nodes that have multiple EBS volumes. So we're not sharing single volumes across the database. We are making sure there is an isolated copy on EBS, or the Google Cloud version, per AZ, meaning you can gain— like, you can lose it completely. You lose an availability zone or an EBS volume inside your cluster. And your data is completely safe and replicated elsewhere.

Yeah. So you mentioned semi-synchronous replication there. So when that write comes in, hits the master, has to be acknowledged by at least one other one. If I have a— like, if all my shards have, say, 10 replicas or something like that, are all of them sort of in that candidate group for semi-synchronous replication? Or is it like, you know, a group of 3 that are sort of You know, the primary and the replicas that are responsible for that, and the other ones are mainly just like read replicas purely.

It's tunable, but really it only needs to get to one other server. Um, normally it's not waiting for all 10 to receive that. And you wouldn't really— this is the beauty of sharding. Most people's sharded, um, setups have like one or two other replicas. Yeah, because you're breaking your data out across so many servers and it's horizontal across these shards, your failure domain gets to be much smaller. So like if you've got hundreds of terabytes of data but it's strung across 100 shards, for example, when a time— when, if, when, if a failover happens, it's a tiny blip for a very small subset of your users and it comes back online rather than sharing Tons of infrastructure, which means you can have global outages. That's bad.

Yeah. When you see someone that has lots of replicas per shard, do you, like, do you say, hey, you should actually be sharding more, like smaller shards? Or is it— is that just different patterns? Like sometimes more replicas is better, sometimes more shards is better?

That's what we'd recommend. But then everyone has their version of the Justin Bieber shard, and the Justin Bieber shard is the shard setup that was at Instagram. He basically was on his own infrastructure because he was the top user until a bunch of issues were solved. And like everyone's got that shard that has that really important customer on it or whatever that they may beef up a little bit. But normally we recommend keeping it kind of uniform and sharding smaller, which is not a difficult operation.

With Vitesse, is it Can I do isolation of shards that small? Like, if I have a major customer, sort of get that— it's not like I have this keyspace that's going to be equally divided. I can sort of make a really narrow one.

You can get very, very granular in terms of how you want to pin and place shards and place data. You can even shard by geographical regions and have data pinned to various regions. So people use that for GDPR and nightmare situations like that.

Gotcha. And then can I make queries that don't use my shard key?

Yep.

And will that just scatter-gather hit every node?

Correct. Not the most ideal, but it's completely possible. Yeah. And for some people, it's not even ever a problem. But We kind of help the larger customers tune their queries to hit shards more uniformly. Yep.

Okay. Tell me about vStream. How does that come in? What is it and how do you all use it?

vStream is an incredible technology. vStream powers our Connect product. It powers Parts of Boost. vStream basically is an enhancement to MySQL replication that allows us to stream data pretty much across shards. Oh, it's not built on top of MySQL, it's itself part of the Go application. But if you imagine a reshard, a massive reshard where you have to completely change your shard key, it's kind of a tough operation because it means you have to stream data across disparate shards into a different cluster and re-shard it. Some people do that routinely for their query patterns. So it was also the message queueing infrastructure at YouTube as well. And it's just a very robust streaming architecture that allows us to be incredibly flexible with the data. We haven't even got close to kind of surfacing the power of things like vStream into our product. We will get there in terms of making even more powerful primitives. There's some extremely, extremely cool things that, you know, Vitesse users even know about that ask us, you know, when is that coming in the product?

When can we get that? Yeah. Okay. So vStream is basically just like streaming out any write updates in my table. Where do I, or where do you hook into vStream? Is that at the VT gate level? Is that at each VT tablet level, or maybe like the primary for a shard? If I want to process that, how am I connecting to that?

So you can connect to it. So if you want to connect to the vStream right now, we actually send it out to you via Connect, which is our, you know, which can give you a— it's an API that can give you constant streaming. So you can put that into Kafka, you can put it anywhere. It's how we do insights as well, which is how we capture queries and we stream them to our data warehouse. So we just expose that for you, and you can listen to the aggregated stream of the whole cluster. That's one thing that's really, really important. It's not even per shard, it's the entire cluster's change stream, which is again really powerful when you have a sharded system.

And so when you're talking about insights, are you doing, I guess, like read queries also end up in the stream? So I could look at Is that right?

No, I don't think reads are part of it. Okay.

It's just inserts.

Inserts. Yeah.

Or updates. Yep. All kind of updates.

Okay.

Very nice. I wanna talk a little bit about just, like, Vitess limitations almost, or just, like, the fact that, you know, now it's a distributed system that has multiple nodes, like, what can and can't be supported or just changes. So, like, tell me about transactions. Are transactions supported in Vitess? Like, what pattern are they supported in? single shard only? Are they cross-shard? What can I do there?

So single shard is easy, well supported. There is a cross-shard transaction implementation that is certainly slower and could do with a bunch of improvements. It's not something we're focusing on heavily right now because most people, if they're at scale, they kind of, they don't need it and they're much better off localizing the transactions within the shard anyway. But it is possible. Although just not completely recommended. I think the limitation that is probably most apparent to most folks is, to the wider or widest group of folks, is you get a tiny bit more latency because of the extra hops that we have to put you through. Like if you are just comparing a straight up like create RDS database on Amazon, with PlanetScale, there's going to be a little extra latency on PlanetScale just because of those extra hops. Now, it's an unfair comparison because with that RDS setup, you don't have anything that's highly available. You don't have a proxy in front of it. You don't have any of those things. You don't have real connection scaling. So when you compare them, it's the same, but you do have this like default extra hop that we can't really take away, but we return for you in, in, in return for that hop, unlimited connection scaling, high availability, all of these things that make it very easy to not think about your database. But it's a thing. It's like if you're doing 1,000 really bad queries or have like a horrible N+1 on your page, like, you might notice it. I think the idea is that you can— should really sort of architect away from that being a problem.

Yeah. And what does that look like? Like, say I'm running a query that, that is just going to hit a single shard. It's not going to be, you know, a cross-shard transaction or a scatter-gather. If it's just hitting a single shard, are we talking like, you know, sub-millisecond addition for that hop there?

Yeah. Yeah. So it's like going to add about a millisecond, I think, to the— so it's not, it's not— and that's just the connection coming in to get to the database server. Everything you do on the database server then will be— isn't taking extra hops. Yep.

Okay. What about what sort of consistency does PlanetScale provide if I'm reading from a primary and also if I'm reading from a read replica?

So, you get repeatable read isolation level. We don't do serialized isolation, obviously, because it's very slow and it wouldn't be necessary for the one way that we're replicating the data. It pretty much mirrors what you get from MySQL.

Yeah. And also just like single-item read consistency, that'd be, you know, pretty strong consistency if I'm hitting the primary, but also if I'm using a replica, then some eventual consistency.

Yeah. If it's been committed anywhere across the cluster, it has been committed. You're not reading anything dirty or that is uncommitted. Yeah, gotcha.

What type of, like, are there certain patterns that you recommend people avoid? And, you know, scatter-gather type queries, also like many-to-many, is that something you try and get people away from? Or any other types of patterns, given that you could hit those cross-shard queries?

Many-to-many is not terrible when you're, again, within the same shard. Scatter-gather is the main one we try and move people away from. Or sometimes people do a lot relying on single rows. Like updating single counts is obviously very, very painful for any database server and lock contention. But normally it's just like helping people kind of get away from some nasty query patterns that can be kind of very detrimental to the database. We make that really easy to find through PlanetScale Insights. You can tag—

Yeah, yeah, let's move on to Insights because I think it's so interesting because I know a lot of other databases are like, hey, only turn on logging for super slow queries because you might have some performance, but y'all have all the queries basically available for that.

Correct.

Yeah, what's going on there? And that's because of vStream?

Yeah, we stream that data out to a data warehouse. We capture every single query, and it means that we can do really solid analysis. We just did a really good blog post about how we store time series data actually in sharded MySQL. And we basically keep a sketch, like a pattern of 8 days trailing of the performance of each individual query. And we've seen trillions of these at this point. And it means we can get, like you said, if you attach Datadog or whatever, it's like sampling your database, right? It's not getting every single query. And anyone who's spent enough time scaling anything has spent time hunting for a specific database query that they can't find. Because like when you have a server that's doing, I don't know, 50,000 queries a second, finding those one or two that are really detrimental, um, is really, really tough. Well, we, we see all of them and we, we, we can then surface the ones that really, really matter. And it's very unique in what we do. And there's actually so much of the current roadmap is about making insights incredible. Part of our strategy as a product is we had to have a number of things in place in terms of primitives. One of the primitives being deploy requests. We had to have a mechanism that we can suggest changes to your database. We needed something that we can get in the loop and that there's a workflow attached.

Yeah.

We didn't just go and say, we're building a database backend. We're building a whole load of workflows that you can only build when you have the database backend. And so Insights is a key way of informing what begins workflows. And it is our most favorite feature, even among giant customers. Like one of our customers, very, very large, runs hundreds and hundreds and hundreds of terabytes on PlanetScale. And then they have one guy that wakes up and have his morning workflow is open the Insight page, Find the slowest query, fix it. What was slow? Now, if you're lucky enough to have a tool that just tells you this is the worst query, you're already winning. And then just having people that just fix them routinely, you constantly keep your app feeling extremely quick. And it's just great. I mean, it's one of the best ways to just continually optimize your database. And that will just get more and more optimized over time. we're nearly done with a bunch of features that will just make database administration just a thing of the past.

Yeah, that's amazing. Is it hard to do sort of aggregation by query signature to say, like, hey, this query is actually the same as this query, even though it has some different parameters?

Yeah, it's not easy. The post talks about that a little bit, actually. Some of it's a bit, you know, it's, it's, you have to take the fingerprint of those queries. And there's queries, even we're even learning similar queries by intent as well in terms of what they're actually trying to gain. Um, it's not easy. Um, it's, it's— but it surfaces really amazing data when, when you have it all. Like showing people when that query showed up. How many times have you debugged something? It's like, oh, now I'm finding out, like looking through code to find out where and why a query showed up. Like we can show what connection ran that query, where it came from. And there's also cool things that we support, like you can use key-value format in query. You know, I don't know if people realize you can add comments to SQL queries.

Yeah.

So we then pass those key values and allow you to just tag inside. So you might find a slow query in Insights and see that it's coming from some worker because it's tagged that. And then people even tag down to like request IDs and things.

Or like, or like method even, just be like, hey, it's this method, it's this request ID, all that stuff.

Yeah, we do exactly that with the Marginalia gem. You can see action and the view and the controller. It's great.

That's, that's amazing. Okay, so that's a good time to switch to PlanetScale from, you know, pure Vitess. So just to be clear, if I come in, I'm using Scalar— if I'm using Hobby or Scalar or Scalar Pro, no matter what, I'm gonna be using Vitess under the hood, even if I don't have Correct. My cluster sharded. Okay. How hard is it if I do want to switch up to sharding later? Now I'm at terabytes of data or something like that and I want to shard. Is that pretty easy to just make this a sharded system now? Is that a button click? Is that like a, hey, we got to call you? What's that look like?

At the moment, it's going to become a button click. But right now, because the majority of customers that need sharding are quite large, we're usually talking to them anyway. And if they're migrating from a legacy architecture, I mean, some people are still moving out of data centers onto things like PlanetScale. We kind of have to help them with some of the edge cases. We've got to a stage where it's pretty much scripted in terms of discovering what the best shard key would be for you by looking at your queries. And then we basically help you. Yeah, it's very cool.

It's like insights again.

Yeah. Yeah. And so We'll get to the point where it will be a button click, but we right now, we want our bar for user experience is very, very high. And because it's a kind of, at the moment, a fairly personal process to go through with based on your application, we spend a bit of time with you sort of figuring it out. A large amount of the kind of proof of concepts we do with customers is just kind of, we deliver them a shard. sharding scheme. This is how your data will be bucketed. And it's not terrible. It's way worse than switching database.

Oh yeah.

And that's usually anyone who's gone and picked— like right now, if you're listening to this and you've gone and picked a database that hasn't run at huge scale, you might be either short on your business or you might have a complete database rewrite. in your future. That's usually the path people go through, and it's hell. And when you're doing that, you're spending a year not shipping, you're having outages. This is why we've done PlanetScale. I mean, that database scaling operation is agony.

Yep. Yep. How hard— so you mentioned earlier, like, you know, moving from RDS MySQL 5.7 to 8 or something like that, you can do with zero downtime. Can you do other migrations very easily? And like, I'm thinking like especially non-Mysql migrations. Can you do a Postgres to Mysql migration, or is that a more custom process that will take a little bit?

It's custom. One of our biggest customers has actually came from Postgres. It's not crazily difficult if they're not using some of the kind of edge case features of Postgres. The data import normally has to just be done by like kind of manually— well, not manually, but like nibbling out the data and putting into the PlanetScale database. If it's from a MySQL database, we can do it, like I said, fully online. So the online migration process from MySQL is really cool. Basically, we connect to the MySQL node. You kind of give us the credentials to the MySQL server. It's all encrypted. We don't kind of Be crazy to allow your data to go over the open internet without encryption. But basically it's a vStream again. So every other database provider you see and you go to their documentation, we didn't want this for our users. The migration process starts with dump your database.

Yeah.

Not everyone, especially a lot of developers that haven't spent their time, don't know the trade-offs with just dumping your database. They do not realize that you have to kind of capture a point in time that you can replicate from. It's a difficult process. It's just not easy to go and do.

Yeah.

And people want their import docs to look as kind of easy and clean as possible, and don't describe some of the dangerous trade-offs that can be made by doing things this way. We didn't want that. We wanted it to be seamless and truly an easy migration. So the way you migrate to PlanetScale is you connect to, what, say, an RDS database. We nibble the data out. We do never dump. There's no dumping of data involved in this process. We just— this is again with vStream— can incrementally build replicas and copies. So it just nibbles the data slowly out of your database until it's completely caught up to date and replicating. And then we tell you Ta-da! PlanetScale's up to date with your previous infrastructure. And this is when it gets really good. You create— we create you some credentials on PlanetScale and you connect your application to those. So you redeploy your app using these new credentials.

This new connection string.

Yeah. And what we're doing is we are proxying writes back to your RDS database. So you're hitting PlanetScale, you're reading from us, But we're proxying writes back to your old database. That's step 2. And then step 3, you just say it's time to fail over. And what we will do is we will swap the roles. So we will make that RDS database a replica of us. We will then, using our proxy, transfer your buffer and transfer your writes to the primary on PlanetScale. And within 3 steps, you have achieved a fully zero downtime migration without doing really anything except having to give us some credentials and change your application's credentials.

That's pretty amazing. And so you talked about the dump process and people don't understand it. Is that mostly a performance issue that they don't understand? Like, hey, if you go try and do a full dump on your production database, you're going to be just really screwing it up, or what other issues are there with that?

Well, yeah, you very often will just lock your database, but you have to make sure you capture the right, um, checkpoint for replication, basically. Like, you have to discover you're in the right place for binlogs, for example. And if you're not and you replay from there, you can— you could lose data. You can have all of these catastrophic problems, or you can get you can get to an issue where it's just arduous and not difficult. Like, if the bin logs aren't fully turned on, like, don't have enough time to replicate, then you're gonna miss certain transactions. Like, it's really, really painful.

Yeah, yeah, absolutely. Okay. Earlier you talked about you have some customers that are moving from on-prem. I also saw, like, in an article you had last year with future.com talking about this cloud-prem Tell me a little bit about that.

So there's multiple reasons cloud-prem is really cool and why people— so people are moving— we'll talk about on-prem first.

Sure.

There's a lot of databases in data centers and we don't want to provide an on-premise version of our product for a myriad of reasons. Number one being we want to provide the absolute best database service we can possibly provide. And without our engineers and people being able to remediate the infrastructure, not possible. Upgrades, for example, if you want the latest PlanetScale feature, you get it because we roll it out. If you have an on-prem PlanetScale, you have to beg your system administrator to go and upgrade your on-prem PlanetScale. We just don't want to get into that world. We want the magic of a SaaS platform. You just log in, it's gotten better. I love to see the tweets where someone's like, oh, PlanetScale added this thing. It's like, Great. Without any user having to mess around. That's one reason we won't do an on-prem product. It also makes you ship very slowly. If you're doing on-prem and cloud at the same time, it's really difficult to match your—

Yeah.

You're doing proper releases here, and then you're doing continuous releases. It just becomes a mess and a bad experience for everybody. Then we see a huge trend of lots of people having databases in the data center and wanting to migrate cloud, there's very few one-hop ways to get into the cloud. So what a lot of people do, and you'll be familiar with this, is they take a data center architecture and they just scoop it up and drop it into the cloud. They're just like, oh, it's just like other people's servers. It's just terrible. When you've got a database server next to you in a rack or just 2 racks over, the latency difference and the availability profile is much, much different. When you're running RAID on machines that you can go and recover and fix problems on, it's very different to like volumes that may never come back.

Just gone.

Or across a network that you don't control. Because Vitesse is built so natively to the kind of hostile environment of the cloud, and because it's compatible with MySQL, and we can just replicate from one to the other, you give this kind of olive branch from the old world and you help them pull into the new world, or bridge, not olive, bridge from the old to the new. Whereas most people who are migrating huge amounts of data out of the data center, they're having to re-architect at the same time to meet the needs of the cloud, or they don't and things just get worse and more expensive and they probably should never have done it in the first place. So that's why we have a lot of customers that move from the data center because We give them a way of just replicating from one into a completely modern architecture by default. And it's obviously very, very attractive for them. But we also have customers in regulated industries or finance, that is regulated, but just generally they would rather data be secured in their own account. But what does that leave us? We could either spend decades of just like constant trust building to get to that stage. Or we can provide a cloud-prem solution. So what the cloud-prem solution means is you give us a subaccount in whichever cloud you use, our backend gets provisioned there. You still use app.planetscale.com to create deploy requests, to kind of delete a branch or whatever you need to do, look at your insights, but the data lives inside your account. So the customers love it because their security teams, they're like, yeah, it's all inside our cloud account. We can monitor access, we can see what's happening. And then the operators and the developers, they love it too because they get a fully managed database service. So it just feels like another part of the AWS ecosystem, but usable by everybody.

Yeah. Is— do you think This cloud-prem idea, is it a particular moment in time where you have these on-prem companies and they either have the regulatory requirements that you're talking about, or maybe just the cultural desire to be able to see it and touch it in their AWS account? Or do you think, in 30 years, cloud-prem is still going to be a thing that people want, where they want it in their account, their VPC?

I think they're going to still want it. They also have massive commitments. Some of our customers have Sure. A billion dollars of commitment to Amazon or whatever, right? Like, this helps, right? The more they can put into their account, the better they can get in terms of discounting. And also just like, when this is possible, there's not gonna need to. Like, if they're getting all the benefits of cloud, it still runs inside their account for like security. You know what I mean? So, Yeah. It's, it's, um, yeah, I don't know, it'd be nice, but I, I can't imagine. Tech moves wonderfully quickly in some areas and incredibly slowly in others. Um, it's gonna be a long time to convince government agencies or banks to store their data in just anyone's. And I quite honestly, I would rather not have my data just shoved in some random database that I was like, Yeah. And there's a lot of them out there. So we go very heavily on trust. We have all of the certifications and we make trusting us really easy with all of these various layers and primitives. And, and we— a lot of the features you don't see us tweeting about that our larger customers experience is we have a lot of features around security and securing people's data. Everything is encrypted at rest, in transit, You can't connect to PlanetScale database without a key. Everything is incredibly tailored towards security as being one of the number one jobs of us.

When you're running it in that cloud-prem model, are you running your own Kubernetes cluster? Will you do it on their Kubernetes cluster if they have one?

We want the account to stay isolated so we don't run on their Kubernetes. It will usually run on EKS. Or whichever cloud equivalent primitive is there. But it's isolated within that account and orchestrated. So they're not really supposed to touch it. It's not theirs to try and administrate. The operator itself handles that. It still remains to be highly available. It's all of the great things about the self-healing nature of Vitesse. But The residency of that data is with them.

Yeah. What about just, I guess, multi-tenancy generally for PlanetScale? So everyone gets their own Vitesse cluster. Are those running on shared Kubernetes clusters or things within your own infrastructure?

Yeah, it depends which. So they're all obviously very isolated. The chances is next to zero, right? There's no way people can arbitrarily Run code or break out of the container isolation. If you are in the kind of just sign up and you run Scaler or Scaler Pro, you're going to be defaulted into multi-tenancy, but again, exceptionally isolated. But you can actually then, when you talk to us through sales, you can have single tenancy where we spin up an entire stack just for you. And then obviously our managed customers that are using the cloud-prem architecture, that is obviously also single tenant. Yeah.

Okay, let's switch to pricing a bit. So I know you have the Scalar model that's based on rows read and written, sort of like a DynamoDB or FaunaDB type model there. And then now you more recently have the Scalar Pro that's more like, I'd say, the traditional here's how much compute and RAM you have. Was this a customer ask? Like, people didn't want to make that leap yet, or how did this come about?

The serverless pricing is very exciting to a certain generation of engineer and horrifying to others. We went out there with serverless pricing. For a number of reasons. It's where modern generations are going, right? The serverless crew, they need a database that can scale to like in crazy heights, unlimited, all the things we do. However, we found when you went to our pricing page and you were coming from a more traditional architecture, say running on RDS, you'd look at that pricing page and be like, oh, there's nothing here for me. Which is not true. There's lots there.

Yeah.

So we wanted to communicate that if you look at your RDS database and it has this many resources, you make this many resources on PlanetScale and it will be fine. And this is roughly how much it would cost. And it doesn't get talked about in serverless much, but truly businesses don't buy that way. Pretty much no businesses buy that way. They have a budget. You budget for your infrastructure at the beginning of the year. And if you're paying per request, budgeting that is so difficult. The pressure that puts back on the engineering team, it's like, well, what do you need us to ship? This thing. Well, I can't plan how many this feature's query outputs ahead of time.

Right.

People want to set a budget, buy that amount, and then that's it. And then normally that's doable. And much easier when you're doing resource-based pricing, which is what Scaler Pro is. And actually, we've been really surprised with— it's only been out for 3 weeks and it's mind-blowing to me how many people have bought Scaler Pro databases.

Yep. Do you think we'll move more towards the usage model, or do you think this is always just going to be the case of just like, hey, people want some predictability, especially about these large infrastructure costs like your database?

I think we may get to usage fully everywhere, but not for a long time. It's where the abstraction layers are, right? Yeah, I mean, this new generation of developer, if you've been a developer for 2 or 3 years, this is not pejorative, by the way. A lot of people say these things negatively. I don't think they're a negative. I think it's progress that this is the case.

Yeah.

People have never racked servers. They don't think in CPUs. Really, what does a vCPU mean when you're buying a managed service? It doesn't always mean something to everybody. So I think as that generation grows and matures and become leaders themselves in software, yeah, we'll see this proliferation of usage-based, but we'll see tooling come alongside it. You'll get tools to help predict your usage. And there's all of these edge cases. Autoscaling is almost mythological in some ways. It's not truly real in a way that people believe. And you can't paper over that with certain amounts of pricing. And that dream of, oh, my infrastructure can burst to 5 million QPS and back to zero, it's not real. No software has achieved that. And the service pricing kind of hints at that being possible. But maybe as services just get more and more abstracted and talk less about the underlying infrastructure being servers, things will get more usage-based. But right now it's just kind of a sensible abstraction that people remember and understand.

Yeah. Is there an architectural difference behind the scenes if When someone's signing up for Scaler versus Scaler Pro, do you have to sort of set up that Envite test cluster differently?

Nope. No, it's just purely— Just how you're paying.

Yeah.

Whichever. And we can help you figure it out. Some people mostly figure it out on their own, which is cheaper, right? Some people have constantly randomly bursty workloads, and it's good for them just to provision a bunch of hardware that's way above that burst. Other people have workloads that don't burst as much, that are lower traffic, or are just very consistent and predictable, and it might just be cheaper to pay the scale away.

Can I switch my database from one to the other?

Yep.

Gotcha. Cool.

Yeah, and we even have seen people switching, right? Just to see how their bill— I guess they're checking how their bill shakes out if they just pay for the hardware to be sat around. It will save some people money. It will be more expensive for others. We've proactively worked with customers to make sure they're on the cheapest and fairer version.

Yeah, sure thing. Okay, I want to switch gears a little bit into some softer stuff, a little less tech. But you've written on the importance of maintaining a high-performance culture. And I think it's hard to understand the culture of a company from the outside, but you can look for clues on whether they're high performance or not. And I think some good clues from PlanetScale would just be You know, the cadence of shipping like major features like boosts and insights, things like that. I would also say just like from what I can see on the outside, really, really good reliability. Like you don't see the Hacker News stories about PlanetScale that you do about some other providers. I would also say just like a really strong and delightful design aesthetic, like around your site and in your social stuff, all that stuff. And then just like really good educational content, whether that's regular blog posts, like Aaron Francis's course, all that stuff. I would say it would be a really good sign. So how did you sort of get that or maintain that, or what principles do you have there for other people looking to do something similar?

If you want a really high-performing culture, it has to be an active choice from the top, because otherwise you can't push it, and no one else can push it inside your organization. It has to just be not a nice-to-have. It has to just be a non-negotiable. part of the culture. Like, we all have a line that people can't cross in terms of how they speak to each other, how they do— like, we all— there's, you know, every company has that. But— and every company also has a talent bar and a performance bar, and you just have to set that. And it's completely arbitrary how you set it. Like, literally, it comes down to just what you're willing to tolerate or enforce. You just set that high and you try not to deviate from it. At any point. Every company I've ever seen that's gone off the rails is because they lower the talent bar. Average people with too much time can just run amok around an organization and ruin it. Everyone's worked on a team with that lazy person. It undermines you. You don't then follow your passion and work as hard as you wish you could or want to, because it's really hard to know that someone's going to just share in the win. One of our company values is no passengers. We mean that. We're here to build a software company and a business and be wildly successful. I want everyone who works at PlanetScale to be wildly successful. And that means you can't tolerate passengers and people sitting around. And so you have to be very careful and it has to be part of the culture. And we get this feedback routinely from employees, they say, yeah, I mean, it just feels good knowing that we really try and address any problems that come up and we don't tolerate messing around. It's not what we're here to do. It doesn't serve our users. Talked about our reliability. We take things like that extremely seriously. And to do that, you have to have exceptional people. And we work really hard in keeping And retaining successful and exceptional people.

Maintaining that talent bar, is that hard? Or just the fact that you're working in enormous scale, hard technical problems, is that easier than maybe other companies have it?

It's hard to start, but if you have problems, these types of problems interest smart people usually, right? They gravitate to difficult infrastructure problems, and databases are very, very difficult. They're extra difficult when you try and— you're very complimentary about our taste and design and our style. We see most database companies, they do the backend piece. I'm not diminishing how difficult that is, but then they pat themselves on the back and don't go far enough to make it— we try both. We want both. We have to have that rock-solid foundation.

Yeah.

It has to be delivered beautifully. And so it makes the challenge even harder. People love that challenge. It's tough, it's fun. So that attracts good people. And when good people work somewhere and they enjoy it, they tell other good people who come along as well. Every time we open a role, we just get an incredible amount of people come and apply to work here. And because you get that kind of reputation. But in the long run, I think it's the laziest and best way to run a company.

Right.

What could be easier than just a small group of incredibly talented people? I see companies that have nowhere near the size of customer we have that have got engineering teams 4 or 5 times the size of ours. They must just be a flabby wasteland of nonsense to have that many people. What are they doing?

How many people work for PlanetScale now?

We have 85 people at the whole company.

Under 100. That's, given some of the workloads that you're running, and just the shipping cadence, that's pretty incredible.

Most people would be, like, I think most people want to run giant, big organizations and puff their chest up. And I am more proud by every dollar we make, Per head, right? Like I want the best ratio of dollars per head of any company in the world is my would be my my mission. It's just it's the best way of doing it. Like you you produce better products and you you you produce better experiences. Yeah, the company will grow and it will continue to grow, but very very deliberately. And at the end of the day, people have to judge us by our service and the quality of the work that we do. And I think the easiest way to get that done is have fewer people, fewer heads in the room.

Yeah. You've published some notes on how you do management at PlanetScale. You've also mentioned that managers ruin companies. How does management ruin companies? Is it by pulling in average people or tolerating average results, or what does that look like?

A manager can wage a war of attrition against a a company, no matter how large the size is. They're normally not busy enough. They're normally not experts or craft experts at what they do. They don't understand the work. They try and measure everything by impact. And I know people are going to be like, what? It's not about— it's definitely about the impact of the work. But another one of our company values is experts leading experts. You have to be an expert in what you do. All of our managers are outstanding, and the reason they are managers is because they love the work that their team does, and they know by being a really good manager, more of that work gets done. Just a proxy for doing more of the really cool thing. When you get non-practitioner managers and they're just professional man— they're just the manager class, they just do management. Your org just goes into disarray because they can't produce the work. So then they have to control people that do produce the work. So they have to make up arbitrary evaluation criteria for those people, which leaves those people feeling disenfranchised. If you work for someone that doesn't truly understand how you reasoned around something or made the trade-offs you make, you just get a little pat on the head and then asked what impact you had. I've seen some incredible engineers at companies just get like— Completely railroaded because didn't their their work couldn't be quantified by their boss, and that's tragic. And you just see a lot of companies just completely fail. I always upset people when I say when I share my opinions of management on Twitter, and people get really grumpy. But a lot of people, but also a lot of people agree. And yeah, I have my opinion.

Yep. Yep. When you talk about experts leading experts, like as you've continued to climb up the executive ladder, like VP Eng and then, you know, chief product chief, chief executive, like how do you still stay sharp and technical? Do you do some hands-on stuff, even if it's in your spare time? Do you just keep up with design docs? Like, or do you just know, know your, know your spots more and where you aren't leading the experts?

So I would— yeah, I would hate to horrify our engineering team by submitting any of my work to— I'm not— right. So here's the model ways. So I know I try and acknowledge the things I don't know, right? Like, can I— I do not spend time in the weeds technically as much as I could do at PlanetScale because I have other things I should do, right? There's— we're here to build a business. There's unique things about my role that I should do. Yeah. But I gain a technical appreciation. I love technology. I still code in my free time for little personal projects or build silly things or websites or just automation for my home is one thing I mess around with. So I still write code at least once a week. But there's a— you got to be careful. There's another profile of like, I was technical once style leaders that were like, I was an engineer once, and they try and like write code for their team, and it just becomes like this embarrassing— I try not to do that. I also just hold technical conversations. Like, our VP of engineering is outstanding, right? He communicates the changes we're making, and you can just track along. Once you know this stuff once, you kind of track along with the changes, you understand the trade-offs. And the other thing is the role I have to play with the organization is understanding the very technical things we have to do for a database. Then we have to translate that into design and visual design and marketing and brand. And because I'm always in the kind of middle of that, I very much appreciate brand.

Yeah.

I very much appreciate our marketing and beautiful design. And again, that only comes from it being cared about at the top because otherwise, are very technical companies, you can tell that their designers are getting steamrolled. And so you have to hold all of those things in balance and appreciate all of those things, which means you think about them a lot and you learn about them a lot.

Yep. Is that something you've always had natural? I agree, PlanetScale has really good design for an infrastructure company. And if you told me, hey, former MySQL DBA is like their CEO. I just like wouldn't expect great design, or like, no offense to you, you know, but like, you really have a great design aesthetic. And like, how much is that something you've had a talent for that you've picked up over the years, or you just like are able to find great designers that can, that can help pull that out? Because I've worked at some technical companies that did not have that sort of, that sort of feeling.

I've always appreciated design and style. ever since I was young. I did best at art at school. I love art. My house is absolutely teeming with art. I've always loved and appreciated aesthetics and style and fashion, even though nowadays all I wear is black t-shirts and a database hat. But I do— I've always had a strong—

Even that is great fashion, right?

Yeah, maybe, maybe. But I've always appreciated— I love bags, for example. I've got a ridiculous collection of bags and shoes and all of these various things. I just love aesthetic things, things that— there's things like the iPhone, which is just like a technical masterpiece presented so simply. I've got this obsession with power and simplicity and how hard it is to present incredibly powerful things simply. And it's always just fascinated me. And then GitHub, fully solidified this for me, that it is possible to build a thriving and exciting company while caring about aesthetic. The early engineers and designers of GitHub had the most perfect taste. I don't think they could quantify what that magic was, because you don't have to when you have taste. And, and seeing the lengths that we went to to not ship things that were ugly or shoddy was truly inspiring. And we carried that over to PlanetScale. But it's something we all really care about. And it's also, again, why you have to be careful with the talent bar because you can't democratize taste. You just— that's why arguing with people on the internet is pointless because they just don't see it the way you see it, right? You just have to hold it in front of their faces like, is this ugly or bad? Yes or no?

Right.

And some people want to hide their product in— I mean, I'm just very proud at PlanetScale, we can take you to petabytes of data and you'll never have to write YAML. I mean, that's some— the things people put their users through is abysmal. And we just try really hard not to do that. And over the long run, that really pays off. And it's put MySQL in places that it historically never would've been. I see, well, first of all, I see our website. Very often you see these top 10 best websites or technical websites. It's like Linear, us, Notion, Stripe, GitHub, and you're just immensely proud. And we have a technology that was started off as being most relevant to giant or enterprises with giant scaling problems. And the fact now we've got hundreds of thousands of developers using the product for the most ripping hot cool projects is because we delivered incredible developer experience and taste and style. And that's all changed. It's how things are done nowadays.

Yeah, yeah, I love it. Oh, that's super inspiring. Cool. I love this. I want to close off with a fun segment. So I like your Twitter persona. It's very much like a blunt, say it to the point. It's been on this podcast as well. But what What I want to do is I'm going to give a competitor. I want you to say one nice thing about this competitor, each one as I say it. So let's start off with an easy one. Say one nice thing about Postgres.

Incredible community.

Why do we— we talked about this a little bit before we got on, but why do we see so many more managed Postgres providers or forks or things? Postgres-compatible things compared to MySQL, even though MySQL runs so much more of the internet, ranks higher in DB Engines rankings and things like that?

I think there's a lot of— well, you see, I think it's because people want to consolidate around a protocol. I think you're seeing a lot of Postgres-compatible databases. I think that's the difference. There's very few that are running pure hosted community Postgres like Supabase does, right? A lot of it is a lot of new generation databases thinking that, well, let's go where it's like most applicable to have the kind of compatibility, and they mostly choose Postgres, right? So like they'll then implement the parser or they'll implement the wire protocol. The thing is, it's heavily fractured. And it makes choice really, really difficult. And I think it's something that is probably something to be concerned about in the long run for the community, because the Postgres community is outstanding. But if it's just getting torn in millions of directions because every company cynically just wants to use the protocol because it's popular, I don't know what that makes Postgres in a decade's time, right? Especially as we shift into the cloud. So I think it's, it's very interesting, but they have a great community, the plugins community. is fantastic, but they've just made trade-offs that I wouldn't make building a database.

Yeah. Okay. All right, next one, you have to say something nice about Amazon RDS.

The payments always work.

What about Amazon Aurora?

Good swing at a fundamentally bad architecture. That wasn't a compliment, sorry. It's a good swing. It's a deep—

It's a good swing, yeah.

At a certain level, I think it's got a number of very nice features. When you get to the upper limits, it's horrible though.

Can you tell briefly what's the swing?

Well, that architecture, that kind of shared everything across the file system, is great when you want an instant read replica. When you want that 16th instant read replica, then you're in agony because it doesn't scale. past like 16 nodes or whatever, or when you want to scale writers, it's single writer, or when you want the latest version of MySQL, it takes forever for them to update it because they're not running real MySQL. You know what I mean? It's just like these little things, like they do a lot for you. It's like a lot of really cool stuff, but then the edge cases are brutal. Like, like we want a customer that found out the wrong way that Aurora cross-region failovers are one way. It's like the documentation, they bury that. You know what I mean? It's like, does Aurora have multi-region? Yes. And everyone's like, surely that would be two-way, right? And then they find out it's not. You know what I'm trying to say? It's just all these weird edge cases you get with the file systems like that. It's not like we don't have edge cases. We just try and communicate them a bit more loudly. Yep.

Yep. Sure thing. Okay. Another sort of shared everything infrastructure, at least shared storage, Neon, which is serverless Postgres.

Again, I think it's an interesting swing at an approach I wouldn't take.

What about, last one, the sort of like edge-based SQLite databases? So Turso, LightFS, OnflyIO.

I think these are really exciting. Again, wouldn't use them for a big serious application, but there's a lot of smaller applications where people should just be using SQLite. So I think it could be really interesting. There's a lot of database abuse that happens to people wedging models into poor traditional databases where we have all of our constraints of high availability and data consistency, all these things, and probably don't need to. You can just use SQLite.

Yep. Yep. Absolutely true. Well, Sam, I appreciate you coming on. It's been a great conversation. If people want to find out more about PlanetScale, about you, where can they find you?

@isamlambert on Twitter or planetscale.com's blog is probably the best way to see what I have to say.

Awesome. Yeah, I highly recommend a follow for Sam on Twitter if you want some, you know, just like hot takes and real talk on databases and also just like good news from seeing the cool stuff that comes out from PlanetScale. So Sam, thanks for joining today. I really, really enjoyed the conversation.

Thanks, Alex. Thanks for having me.