Databases and Engineering with @PlanetScale CEO - Sam Lambert

Hey everyone. Welcome back to another fantastic episode of the Geek Narrator Podcast. Today we have the amazing Sam Lambert, CEO and co-founder of PlanetScale, known to be the world's fastest and most scalable cloud database. We have talked about databases, operational excellence, avoiding complexity, also known as simple engineering, and many more interesting ideas and takeaways. I captured some of Sam's interesting, sometimes controversial takes on databases, engineering, scalability, Postgres versus MySQL, replication, Vitesse, and so on, and asked Sam to clarify, expand, and elaborate with examples, which is definitely super interesting. So do watch the entire episode because you are going to love it. And, and, and if you do, please hit the like button, share it with your network, and subscribe to the channel. If you want to support this channel, you can also join memberships using the link below in the description and get access to member-only videos. Now enjoy the episode. Thanks. Hey Sam, welcome to the show. I'm really excited to have you today. There's just plenty of things to talk about, but before we get started, let's start with an introduction about yourself, about your background, about what you have been up to. And then we can dive right into it.

First of all, thank you for having me. It's great to be here. My background, I've really just spent my career working in databases, mostly at scale at various different companies. Probably the most notable 2, I did a short amount of time at Meta, obviously the company behind Facebook and WhatsApp and Instagram, which is at a very large scale. And then I spent most of my time At GitHub, which is I think the 40th largest website on the internet. It's pretty decent scale and obviously the home of all software development. And then now I'm here at PlanetScale where we have a cloud service, cloud database service for Vitesse and Postgres that aims to scale some of the like world's largest relational databases. Most the largest ones are Vitesse databases. We have Neat. Niki coming, which is like Vitesse for Postgres. Coming next year and we'll start taking on some of the very large Postgres databases.

Awesome. Yeah. I mean, that, that's all really interesting and I want to learn more about all the things that you mentioned. But I want to start with a, let's say, casual question, not really related to databases question. How in this world do you get so much time to write blogs, run a company like PlanetScale, You are pretty much solidly active on Twitter. You have, you tweet about really these short takes, but they make a lot of sense. So how do you manage your time and how does it all work for you?

I wouldn't say I manage my time in a very routine way in the sense that like, I know some people have very set routines. I don't. I love working, so I'm always working practically or doing something, just like checking in on my phone. And there's always stuff to do. I like to keep my time actually very open. I think that's like the time you can have the most creativity. PlanetScale is in certain ways, like very chaotic in positive ways, in the sense that we work really fast. We very often ship things ahead of deadline. And that means we try and keep our time very fluid and can respond to things that we want to do or customer demands and build really quickly. So that's a long way of answering that we don't, I don't have a ton of structure in my time, but that's a good thing. Like, we just did 2 announcements in the last week. We knew we were building those things and they're mostly done, but we didn't realize, we didn't think we would do the announcement right away. You know, we woke up on Monday and thought we could announce these this week. Let's just do it and it'll move us along quicker. So we just did it quickly. We don't, we're not like, it's incredibly regimented in very fluid ways. We don't plan too far ahead. We just flow with what our customers want, what goes on, and we respond to issues that we see. When it comes to like Twitter, to me it's just a release. I'm always working, I'm always, or doing stuff at home with family and it's just like a little fun thing to shoot off a thought or whatever and respond to people. And I think it's fun in the sense that we live in this, we're in this industry that is incredibly high stakes. There's lots of sort of money involved. It's very serious. And then there's this town square on Twitter where you get to interact with people that are also building companies or partaking in companies or building consequential open source projects or blogging. There's like famous accounts. That are folks that you barely even know who they are, but they've put out years of good technical content and they've become known for certain things in the industry. It's a very special place for that. So I just enjoy dipping in and out of there every now and then when I have random thoughts. Some are coherent, some are just ideas. Sometimes I tweet things I don't fully believe, even just to see what the reaction is, to see whether people vehemently disagree. Or if it just evolves your thinking. I'm always very curious about what other people have to say and their positions, and it's a good way of getting them.

Yeah, absolutely. And looks like it is working for you. So that's great.

We'll see.

Cool. So I want to start with simplicity, as in you, I think you tweeted about simple engineering and you mentioned that simple engineering is better engineering. And everyone is overcomplicating everything, which I think on the surface, I think it makes a lot of sense and who wouldn't agree? But then practically speaking, there are situations when there are complicated use cases and for those we need some complications. So I want to ask you, like, what is your framework to decide on where is the line between necessary complexity and unnecessary complexity when Designing solutions.

So first of all, I don't think people, I agree, people would not disagree that engineering should be simple.

Yeah.

But most people do, but don't, they don't act that way in practice. They know, it's like people who call themselves rational, rationalists or whatever, right? Like it's a self-assigned title, which essentially says everyone else is irrational with their choices. Same with simple, right? What I might call simple, Someone might view as complicated, and then I've self-assigned myself as simple. So it's very, it's first of all, like kind of very subjective. However, complexity can really creep in. I will answer your question about the framework, which is to say that I don't really have one, but I think the industry suffers from chronic boredom. I think a lot of people are very bored. in their jobs. A lot of engineering and the things that have to be done in engineering is pre-wrote. Like a lot of it is done. Websites, CRUD apps, you get a Postgres database, you get a MySQL database, use a framework, and then you can just pretty much build. You can host it pretty simply. And I don't think most people want their jobs to be reduced to things like that all the time. So then they, invent things to do, which is where complexity happens. So people talk about the overuse of Kubernetes, for example, and most people are overusing Kubernetes and trying to host simple websites and do all this kind of thing and waste their time. We run a lot of Kubernetes, everything in Kubernetes actually, at very large scale, huge amounts of state in Kubernetes, and it's complicated. It's very hard. to do, but you can take simple approaches to going doing this and know when things should be made simply and solutions should be simple. And when complex tools are giving you what you need to achieve your simple goals. And so I think if you view engineering holistically, if you view it as achieving an outcome and iterating, you can build things very simply. by just being smart about the approaches you choose and just try and driving out things that are complex. I think you get unnecessary complexity when you try and bite off very large projects rather than building very incrementally. We build extremely incrementally, like the first scope of, like, we think of like roughly what we want a user to be able to do. And then we just see how fast we can ship the earliest version of this. Like we say, we have little mantras, like one of them is make it possible. So you just make it possible to do the thing. I'll give you an example, like online upgrades from major Postgres versions right now is a real pain. Postgres does not make it easy. We have not made it easy yet for major versions, but we've made it possible, right? There's nothing stopping a user using one of our migration guides to go from one to the other on site in our platform. And now we're gonna iterate. Towards making that easier until the whole story is automated and done. And that just comes through breaking it down, breaking problems down very simply, and just figuring it out as you go. And you get, see, you tend to get simpler solutions as you build rather than saying, okay, it has to be this hugely architected system that is going to have all of these caveats and is going to leave us optionality here and there. And you just, you build something too big and you just don't ship anything.

I think that makes sense. And I think that boils down to the core of software engineering where people recommend thinking about problems and solving them incrementally. And then most of the complexity, at least in my experience, comes when you try to solve a problem that do not exist for you or that you have seen that someone else is trying to solve.

Yep.

So would you agree that if probably following these 2 things, solve a problem incrementally and don't solve a problem that you don't have, you pretty much reach a pretty good state in terms of simple engineering.

You can. It certainly allows you to do simple engineering if you take that approach.

Yeah.

Then it comes down to taste, right? There's a lot of it is about taste. Something that really resonates with me is the Rails way, which talks about giving you really empowering tooling. that can be very simple.

Yeah.

But powerful. At GitHub, we used to have this interview as part of the interview process, which was like essentially a test as to whether you would use a shell script to build something or not. Like we had, I can't remember what it was, like a very simple, we asked you to do something and it was essentially pass/fail very quickly. If someone started building this like massively abstract Java, architecture to just do something very simple, they were just not going to be a good culture fit. If they cracked open a shell script to get something done, then we knew that they were smart. Yeah. And that's like the truth of it. Now, I tried to find this and maybe one of your audience could find it. I'd be so happy if someone did. There was a story about an old magazine, a computer science magazine, maybe from the '90s, where there was like a kind of, someone posed a question of how something would be done. And that literal thing happened. Someone wrote this kind of 8-page, really well-factored app in whatever the language was at the time. Maybe it was Perl or something. Someone else just wrote a shell script and they both did exactly the same thing. They both output exactly the same thing. And it just shows the approaches. Some people just overcomplicated things and designs, and you meet architects that wanna draw really big SlideShare presentations and microservices and This is not for me, this doesn't really get anything done.

Yeah, I think it makes sense. You talked about tooling and I've also seen that. So when you have tooling and when you have like standard ways of solving a problem without having like n ways to solve a problem, I think that also leads to simplicity because then let's say as a team, you know that, okay, if you have this problem, this is the way to solve it or this is the tool you have to use, unless it's a, like a really new problem that no one has ever encountered. I think that also helps. Would you agree?

Yes. First of all, there's very few problems no one's like that no one around out there has encountered. It's not like we're— computer science changes much, right? A lot of the things we're dealing with are still just the facts of sort of physics and the limits of computers. And I would agree that if you have More standard tools, like trying to implement something to get going in those tools and then proving that they're not the right thing. You can always change later, right? At least when you found out, you have to start. What's that? Who says there's one, some CEO out there has says like action creates information. I think that's like very true. Just getting going on whatever language or framework that you know to like Because actually, really, you're just— most of the exploration should go into understanding the problem, right? That's the implementation and the fix. We get really strung up on that as engineers. No one's looking at the Golden Gate Bridge, certain types of nerdy people are looking at the Golden Gate Bridge and thinking like, wow, I mean, they used this sort of riveting technique or type of steel or whatever. Most people are like, wow, bridge, a huge gap for people to connect. Like, it's an important piece of infrastructure.

Yeah.

Not enough engineers think about what they're doing for their user or the actual problem they're solving. And they care a lot more about how they're solving it. And that just leads to messing around versus if you're really obsessed with getting the user an outcome, you'll probably get it done with something quite simple. And if not, you learn about the problem and then you can adjust how you solve those problems.

Makes sense. Since we are talking about simplicity and complexity, I want to talk about the slotted counter pattern that you used at GitHub, and there's a nice blog. I also tweeted about it today. It's an interesting thing because I also got to implement something similar while working at HubSpot for solving the same problem, which is when there is a high contention on updating a single row, then you can split the number of rows that are here that you're going to update and solve the problem of contention. While this is a very good pattern, there's still, when you reach a stage or a scale where you actually have this problem, right? Some people would argue that this is again slightly complicated solution. So where is this line where we figure out? So this is like taking an example of what we were talking about earlier. How do we figure out that, okay, this is the time where I should be using Slotted Counter? Because if I use starting— use— start using the— this pattern right from the beginning, it might be too complicated and I may not reach that stage ever. So how do you decide on that? Probably taking this example, probably if you can share some examples working at GitHub, like what was the problem? I think it would help.

As soon as you start hitting deadlocks is when you should start using it. It's very simple. It's a very simple solution to what is a painful problem. And some people would try and architect other services that do key generation, like to do ID generation and do all this sort of stuff where you can just kick it into the long grass by just making— you could easily build just a tiny library that extract, like a gem that would just manage a counter table for you in probably like 40 lines of code or something to just to go and do this. And it'd be really easy. I think you just, the general rule, I can't even remember what was causing, or maybe it was, it could have been counting anything. It was, it would've been just like, It could have been just counting downloads or counting stars, or it could have just been, so I can't remember exactly what it was, but it was as soon as you hit issues. GitHub is a great case for this. GitHub was a Rails app, still is a Rails app. Things would go like hit scale limits eventually, and then you would like re-architect piece by piece. But it was incredible how well things lasted and ran into a certain scale. It just depends what your app's doing, right? Isn't it? That's the thing about software engineering. That's the reason. people get so lost and things become so complicated is because a lot of people are making up, making their own problems by making their own products, by choosing their own users. And it can just go far off into the weeds. You must have seen lots of good simple engineering at HubSpot. They're a great company, Vitesse users actually as well.

Yeah.

But I can't remember the exact number, but I remember when I was told the size of HubSpot's engineering team, I was shocked because HubSpot runs at enormous scale. I know the size of some of their data stores, not just for tests. They are enormous. And the team is very, I think the company's only 1,500 people, isn't it? It's a pretty small company.

Yeah, I think that's pretty—

But it's obviously a huge company in terms of revenue and being a public company. HubSpot is a tech company that I massively admire. Everyone we work with at HubSpot is just outstanding. I love that company. So you must have seen very good pragmatic engineering. You know this, I can just tell. Give me a product, show me a product, the size of that and the size of a person's engineering team. I can just tell you whether they have good engineers or not. I can also tell you based on a lot of, based on what services they use, what they've chosen. You can also tell if an engineering team is good or pragmatic or even running at any scale based on what they're actually using. When you look at the size of what HubSpot's doing for the size of engineers, they just have to be good engineers, which is. Pretty cool.

I totally agree. I think one of the strongest engineering cultures, really disciplined and strong engineering mindset that I've seen in my career. So totally agree there. Cool. Talking about Postgres, right? Like you like recently tweeted that you basically, while launching PlanetScale Postgres, you didn't find any critical bugs, you, but you found some design flaws. So I wanted to understand what these design flaws refer to and how does they, how do they work in MySQL? So a comparison between what kind of design flaw did you find and how does it work in MySQL?

First of all, I'll say I'm open to the idea that they're not even flaws. So we can call them philosophical differences.

Okay.

Like I could just be wrong. I'm coming from the angle I'm coming at it from. I would say these things are flawed, but if they weren't designed to do what I'm trying to do, then me, it's me holding it wrong. But generally PlanetScale thinks very hard about high availability. It's something that we try, like we really push for. So in our Vitesse product, We build things that are highly available, and that means we can roll nodes, we can kill nodes, we can do all these things and not interrupt customers' queries. Like, we could just press a button now and restart millions of containers and not cause really any interruptions to any customers. That's real cool. And that's achieved through Vitess plus MySQL and the way MySQL does replication. We are trying to get to the same level of high availability for Postgres. And this is made to be very difficult by Postgres's replication design. There's a number of things that feed into that design, or just the state of Postgres that are difficult. So really upgrading from major version to major version is very painful. Without the same libc version on the source and the replica, you can corrupt data. All these things are very And let me wind back. Philosophically, Postgres seems to bind the state of the primary node and the replica node too tightly. Postgres replication is designed to create a kind of physically matched, physical, a physical copy essentially of the primary node so that when you do things like physical replication, You get just a bit-by-bit copy of what's on the primary, which means the primary needs to know about the state of the replica and things and manage things like replication slots that have to be replicated themselves. And all of these things are very fiddly and very difficult to deal, to do. And deal with. And because it's aiming to be like a, just a full copy, things like parameter changes have to be done on the primary node first, which makes high availability very difficult. With MySQL, the replication is less correct in the sense that it does not guarantee that the replica itself is a full copy. Now it can, it will be, but it's not guaranteeing that. Like when Postgres tells you, when you replicate, you attach a replica, it gives you a full guaranteed consistent copy of the primary, which is great. That's really good and is really simple. MySQL is less militant about that goal, but give you all the tools to achieve the same thing, but more flexibly. So you have GTIDs in MySQL, and as long as the logs are on a node, you can just repoint MySQL nodes, change one primary to another primary, and have the replicas switch. And as long as they can just resume from that GTID, it doesn't matter. The primary doesn't care whether even the table, you could like black hole the tables on the replicas. You can filter replication so you don't get it from You only get certain tables. The primary just doesn't care. The primary's not gonna freak out if it, it's not asking even, it just, it's just waiting for you to acknowledge and sends you the next position and you just keep streaming from there. So it's like very asynchronous. There is semi-synchronous replication, which means like they guarantee that it gets into the logs of another box, and that's very good for durability. But at the end of the day, you can do things like if you want to just change, say, connection count, you can do it on all of the replicas and then just promote one of the replicas to be the primary and it just works. That's just not possible with Postgres. Same with CDC, right? If you wanna reconnect CDC clients after a long-time disconnection, if the replication slot is, and a failover's happened, if the replication slot's not there, it just doesn't work anymore. And so it's very brittle by design. And it makes doing high availability and things harder. They are improving this story, but I would say replication and just cluster management has been one of the hardest parts of working with Postgres.

Okay. I think that's fair because I was, I've been part of a couple of version migrations and upgrades in Postgres, and it was a pain. Even being on RDS, it was still a Pretty painful process also because of the replication slots and things like that.

Exactly.

So I think I get that. And as you said, it may not be necessarily a flaw. It's just that moving one problem to another problem, like it's a trade-off where the replication is simple, but then it creates some problems when you are doing this operational stuff. And what is the problem that may occur with The MySQL way of doing replication though.

You could, you couldn't do things like if you disconnect a replica for long enough and the binlogs expire on the primary, you have to start again from your full backup. Whereas in Postgres, you can just like connect and it will start shipping the data over. Things like that are very nice. There's nothing that inherently bad about it in the sense that it's much more mature and much more flexible. But you could, I guess, if you were trying to be fancy and didn't know how it works, you could, there's like various footguns. This is the same amount as Postgres though.

Yeah.

And I think it's like less likely to cause an outage. Like we've seen even try, there's one customer trying to migrate right now. They're on Postgres 13 and it's very busy server. It's got the upper limits of what you can get on the hosting platform they're on. Just connecting a replica that's far behind basically brings the primary down. Things like that, that just doesn't really happen in the MySQL world when you do it that way. So yeah, it's just like little things like that. Like it just, you can replicate a new MySQL version off of an old version super easily. You can't really do that very easily in Postgres, especially if a lot of change, there's a lot of change, like they've, they're doing changes towards making collations and things better and not handled. In such a brittle way. Yeah, I can't think of that many ways that MySQL's worse at replication. So it's worse at other things that there's tons wrong with it, but that's not one of them.

Yeah. Okay. I think it makes sense. On Twitter and on some of the blogs, there are also some weird takes about like when to choose MySQL versus when to choose Postgres. And it's probably a subjective discussion, pretty nuanced on your application, your team size, your expertise, and a lot of it. What are your thoughts on that if you have to choose one database? And I know you mentioned a couple of design flaws, but let's say I, as an application user, I do not have to care or worry about a lot of those operational stuff. Let's say I use some tooling. What would be your design, or let's say decision-making model figure out, okay, what's the right way to choose MySQL versus Postgres?

A lot of it comes down to your goals. Really does. Like, it truly does come down to what you want. If you're really not having to operate it and you're picking now in 2025, you probably want to pick Postgres because that's what most people use now, right? It's, you know, you're going to be able to hire people that have expertise. You're gonna, the ecosystem is bigger.

Yeah.

Most of the tools work with it. MySQL's more legacy in that world. If you're scaling up rapidly as a business, like a lot of the company, AI companies that run on us and use Vitesse are doing, none of that matters. Like at a certain scale, none of this stuff matters. All the, like all the fancy stuff, all the little extensions in Postgres, they just goes, it just all goes out the window at a certain scale. And then it's whether it's, I, it's very hard. I don't know. I never really nowadays view Vitesse and MySQL as independently. I never just, I have not gone and just set up a standard MySQL server for a long time. I just do it on us with Vitesse. If you are growing really quickly and you've got like a write, high write-bound workload with hundreds of terabytes or even tens of terabytes of data, it's gonna just be much easier managed in Vitesse. Just, it's just so much more mature. And you're getting SQL semantics. You're not getting all of them that you would in Postgres. You're not getting extensions. You're not getting like a lot of the stuff that you get with Postgres, but a lot of those things go out the window anyway. And we see a lot of people migrate from Postgres to Vitesse for getting to scale, but it really just depends on your goals. If you're a small startup and you're gonna use, just on the PlanetScale create database page, and you don't think you're gonna go to millions of users instantly, then probably just use Postgres cuz we'll take care of The annoying bits, you'll be part of a thriving ecosystem. Your new developers that join the company will be able to use it and will be happy. That's how.

That makes sense. And yeah, at the end, think about your goals. What do you want to achieve? If you're more focused on the community, on the compatibility with external tooling, the way Postgres is growing, probably it is a better choice. But then, as you said, there are some other goals where MySQL would make more sense and probably also Vitess. Talking about Vitess, you mentioned, I guess it was some blog that I read, but it was about like people have only seen about 10% of Vitess and there are some other features of Vitess that probably PlanetScale is working on and they are pretty mind-blowing. So what are these capabilities of Vitess that probably are not known commonly?

So I said that quite a while ago, we're probably a fair bit past 10%, but people don't realize that geo, like we don't allow in our current pro— in the cloud product, we don't allow people to geo-shard. You can do that. There's messages in Vitess, there's like message queuing in Vitess that's very robust and very strong. Materialization is very good in Vitess. We don't really expose an easy story for that yet. The various, like, flexible ways to do sharding, most people don't end up using a lot of those, but like, they're there and they're possible. There's just all of these, like, loads of things that have been thought of for tons of our customers to help them migrate that people just don't realize. Like, I had a customer ask us the other day, we want to use Vitess for, like, key generation sequences, which is something Vitess is very good at because if you're sharding, You like primary, you don't generate primary keys in the shard anymore. You generate them globally. And they wanted to generate primary keys for 2 tables at the same time. And they thought they'd have to do that in the app and then write that through. And then we were like, no, Vitesse has already thought, we've already thought of that. And this is how you would do that. There's like all these just like long tail of things that are really nice. Like most people have never experienced a reshard. A lot of people do their own self-sharding, which is Hard, right, to do. What's even harder is resharding. Like, resharding is a really difficult exercise. Going through that process with people is really fun to watch 'cause they can, you see us coordinate that as like an online operation. It's really nice. And then there's move tables workflows where you can split tables out of certain clusters and do that online. There's just like lots of very flexible tools that allow you to manage and move around hundreds of terabytes or petabytes of data. Very flexibly. And I just, it's a shame most people don't get to see that in action. It's really fun to do it with customers. And maybe we should do a better job of showing that stuff. Like we've done like cross-country failovers for 600, 700 terabyte clusters. Just watching that, watching hundreds of replication streams just spin up and build clusters across the globe and then just do that. It's really fun.

Yeah. I can imagine. Talking about sharding and where, as you said, some of the users implemented sharding without knowing that Viteess already provides them. I still see that somewhere this is like a leaky abstraction where users still have to think about sharding. Do you think that there's a better way to model this where people who are using these services, they don't even have to think about sharding at all and Probably they just think of it as a one, one database and then just work with it.

So it's both complicated and not complicated in the sense that if you get your sharding key right and your app is scoped the right way for that sharding key, it's not too diff— it's not crazily difficult, right? Like you just have to, There's companies that build auto-sharding tools. We would say we do explicit sharding, which is you explicitly state how the workload gets sharded, and then you build your application that way. You need to know how to do this, or you need to be aware of this to avoid things like scatter-gather queries. So a lot of these auto-sharding data stores, they auto-shard by essentially randomly placing data and scatter gathering. So you can go from one query that goes to one machine, and that's really good. And without knowing it, change or your data grows enough that you can go to hit every machine in the cluster to do a single query. That's gonna be very slow. That's gonna be exceptionally slow. And that's what we'd call like a scatter gather query. You scatter across the whole cluster and you gather it together. You can do those in Vitess. They are just slow. If you do a select count against your whole cluster, it's gonna have to hit every shard and count the rows and consolidate that count for you at the Vitess level. And we do that, right? Like, like Vitess is very good at that. Vitess rewrites your queries to go to the right shards and pulls the data back and gives you the data you want as you expect it based on your sharding scheme. So there's been many attempts at people not having to worry about sharding, and they all end up sounding really good. And for the first, 2 days or whatever when you're building it. Yeah, cool. This is super easy. It's magic. Why can't this be magic? And then you start to find out very painfully why it is not magic and why you can't just randomly put data on machines and just gather it whenever you need it. It just doesn't really work that way. So instead you explicitly shard, you tell Vitess where to place the data. You're a bit more mindful of that. And at the end of the day, by the time you've got to sharding scale, this is nowhere near the worst thing you are doing on this system or other systems. to achieve scalability. You just have to be aware. You just have to be aware of the underlying, that you're like, it looks like one database, but it could actually be thousands or hundreds of different servers. You have to know that you don't want to hit all of them at once.

Yeah, I think it makes sense because it can get really complicated, but I probably, it also depends on the use case. For example, like talking about the transactional use case, like Postgres and MySQL, probably it will be probably latency sensitive. And since you're running transactions, how about the analytical use cases? Probably not even real-time, but analytical use cases. Would it make sense there or have people succeeded?

Some people have done pretty OLAP workloads on Vitesse, but a lot of the time we're telling people not to do that. There is much better databases for doing this. And you should go and use those. They have different trade-offs. They are not designed to be online 24/7 and operated 24/7, and they're not designed to be as fast as we are. But they're designed for different queries. You can append, you can like signal to Vitess as part of the connection that you want to go OLAP mode, which means we will less aggressively kill connections. We'll let you to, like, we strictly enforce row limits and query times for scalability, right? OLTP queries just should not take that long. OLAP mode allows you to run much longer queries. But it's just still not a good idea. It's just not efficient. It's just not efficient. You just should use Data stores that are more optimized. People try and do hybrids. That's nonsense too. That doesn't work. It's different storage formats, different types of query parameters, different types of query planners go into OLAP. And that's like, fine. That's good. It's just a different workload. And we've had people come to us thinking they have a large OLTP workload and we, instead of trying to convince them into it to use Vitesse, we just tell them, go and use an OLAP database. And then they're usually like much happier.

Yeah, I think it makes sense given the use case. It's probably like OLAP use case and there are so many options available. It's just that people sometimes feel that if I have one database and why can't I solve both the problems with that? But then at a certain scale, you create problems for both OLAP and the transactional workload, which is not great. Yeah.

You've got to unfortunately most of the time pick the best tool for the job. Yeah. Not too early. Like we talked about earlier, there's like a bunch of people that are just doing their reporting and stuff off of their Postgres nodes.

Yeah.

And it's kind of working for them until they get so much data that it's not working. Do it until it stops working and then do something else.

Yeah, absolutely. I think it makes sense. One of the things which is also very interesting in also PlanetScale and some other databases is branching, right? You can create a branch of your database as in you're working with your code and then work with it. Once you're ready, you can open up a PR and get it reviewed and stuff like that.

Yeah.

But the obvious problem is probably there are 2 problems, right? One is the conflicts where multiple people are working on different versions and they try to merge it to the main branch of the database. And I was reading about it that the 3-way merge algorithm using Vitess schema diff It's an interesting way, but I was trying to imagine how does it work without this algorithm? Are there other algorithms that people are using? And have you seen any nasty conflict problems that was really difficult to fix or even resolve?

No, only it's because we really only do it at the schema level. We don't allow like data merging. People have tried. attempted this. It's like a really painful prob thing. It's really hard to fix and no one really needs it. It's like a nice, it's like a, it's a nice to have, but the trade-offs are not good. There's other types of branch. So I would say our branching is very good for schema changes and like you restore data from a backup. If you need backup data in a branch, but the data never gets pushed into production. So it's not that hard to do. We do it this way for multiple reasons. Most of the time people just wanna make schema changes. That's easy. You shouldn't test with that. With people do it in certain jurisdictions, it is illegal and at best it's risky, right? To test with production data. So branching immediately from production is not really a good idea. I understand why people do this though. This is, I basically, I think every startup does this.

Yeah.

And I think it's fine. I think it's just a trade-off that people make. A lot of the time people have seed scripts where it downloads data from prod so they can test locally. They just do, this just happens. It'd be silly and idealistic to pretend that people don't do this.

Yeah.

But we didn't really spend much time going into a constant time branching system like Aurora would be. Those things are very good. They're very flexible. You can break off other, you can create branches and get your full dataset in there very quickly, like extremely quickly. You can snapshot, rollback, all this stuff is Very intelligent. It just comes with real trade-offs. If you're separating storage and compute to do this, that's slow. We just don't do that. No, we don't, haven't really had many difficult things to do there, but we haven't taken on solving the whole problem the way other people have because we just don't believe in the fundamental trade-offs. It's not worth making every production query slower so that you can have this functionality in dev. It's just not worth it.

Yeah, I want to get to that, but before this, I think I've seen some use cases where people need this kind of production data, or at least similar to production data, because let's say I'm running some query and that query is really slow and it's timing out and also some deadlocks or whatever issues. I want to test this query and understand what's going on and trying to test it with similar set of dataset or similar type of distribution of data. I think it's really difficult. So that's why I guess people try to either create like a different snapshot on a different machine, try to do their experimentation there. And I get it's, I guess it's easier when you just can branch, create a branch and test it out and then do something about it. Of course, there are issues like you are working with production data. There might be some sensitive data and whatever, like things like that.

Well, sensitive, it's, I believe in the EU it is not legal do this, to test with. You cannot use user data for development because it's against what— it's, you essentially have not— it's assumed that the user has not consented to this, whatever GDPR, all this sort of stuff. If you're in Europe, you deal with these things.

Yeah.

Some of them are very silly. Some I understand. But also when you see these kind of mess-ups, I don't know how often they, like, you see them come around now and then, they probably happen more than is reported, but Not to like an insane degree, obviously. When a test email goes out to everybody, that's very often because they're testing on production data, right? They enqueue jobs on someone's laptop or whatever with real emails.

Yeah.

That like happens. I mean, it's just silly. I mean, it's really bad. There was, this got debated on Twitter a while back where one company that does this kind of data branching was like very proud of themselves for it and was talking about this. And then loads of people replied with, you just shouldn't do this. This is just not a good idea. I fall somewhere in the middle on that argument, which is people will do it. People do want to be able to do it. It's fine to build something that allows people to do that. Yeah. Not worth the trade-offs. And the majority of our customers don't do it.

Yeah.

And some do, and I'm not judging them. We're really not. Really, I'm not. Yeah. Yeah.

I think it's a bit more nuanced. And as you mentioned, sometimes these trade-offs are not worth it. Having this branching feature is still really useful and where you get test environment without the data probably, and having to, having a tool that can actually merge your changes without conflicts. I think that's really powerful. Do you see that? And sorry, I'm, if I'm misquoting you. But I think I remember some statements around that, like probably we'll get, be able to get rid of the staging environment because we have these branches on databases. Am I misquoting you or there was some discussion?

No, I think, and a lot of people do use it that way. A lot of people have staging environments, including us. Yeah. But for certain applications that are probably started on PlanetScale, a lot of them don't use staging environments. They just spin up dev branches whenever they need it. And there's good, we have GitHub Actions, people have built them for that in Vercel so you can pair your environments with like a new database.

Yep.

Again, you might have some bootstrapping scripts to put some seed data in there or whatever, but it's a kind of good way of not managing dev environments. That said, we use for like local development, MySQL and Postgres to do those things. And that works just as well. It's our aim that you can use just the standard MySQL and Postgres servers locally and then put PlanetScale for production so that you can use it however you want. It's, this stuff is, it's good when you're net new product, you can build, like when someone's building a new app and they use that and that's their flow and that's staging databases. Most companies that are already established bring a ton of their own processes. And so it doesn't fit as elegantly into that system.

But then I also think that like stage environments are also about applications, right? And not just about databases. And that's where it gets really tricky when you use, let's say, production database with a staging environment, like your branch of your application and not the database. It gets tricky to keep that sort of separation.

Oh, you're saying when people use a branch of their app and this and the, and keep it connected to the prod data? Yeah. Not a terrible idea either. You've got a chance of deploying a bug, but by the time it's hit prod, you've had many chances to stop that anyway. Like at that point, it should be considered prod. You know what I mean? Like, it really is prod. It's just like a different SHA of an app in prod. But if it's talking to the database, it's prod still. If you can log in or 1% of traffic is hitting it or whatever, it's prod. So you better have done it. Yeah. Like, I wouldn't call it staging. I wouldn't call it development. I would say like, it's a flag that is like a subset. It's almost like a feature flag, right? Like it's a different code path.

Right.

set of code paths that are in prod, if they're connecting to a prod database, they better have removed the DROP TABLES or whatever that are in. Just like, it's the same. Yeah.

Okay. Coming back to the point that you mentioned about, and I guess PlanetScale Metal uses NVMe drives and it is at a high level, it feels like it is going against the idea of like cloud-native databases where They are pushing a lot on disaggregation of compute and storage and leverage the storage-level replication on databases. So what are your thoughts? Like, what are the trade-offs here? Like, of course, one is the, like, the latency. If you're using NVMe, you get amazing latencies. And I guess you mentioned that for some of the use cases, why slow down, like, all the queries that you have? But what are the downsides of using it?

First of all, I would not say our approach is any less cloud native. It runs, we only run in GCP and AWS purely on Kubernetes. So it's mostly more cloud native than the disaggregated storage and compute ones, 'cause they're not rolling nodes the way we are or whatever. Like they're not, our cloud nativeness is Proven by our reliability, right? Because the cloud is inherently failure-prone and we are very robust and resilient throughout that. So I think that's more how cloud native is designed. You can separate storage and compute in a data center. You just build services on top of those and it's doable. So I wouldn't say it's more cloud native, but it's definitely showed up alongside the cloud, right? And it was originated by major cloud vendors. The ones that kind of, you could really just, and OLAP had this a lot sooner, but Aurora is probably the OG kind of separated storage and compute database engine. So the downsides of that, first of all, no, I'll do our downsides first, which is it's less flexible. That's a fact. Right. Like a machine failure because you've bound The disk, the storage, and the CPU and memory. Node failure takes the whole thing down. So you have to bring up another node from a backup. Again, that is not hard, right? That's, we do this. It's guaranteed to happen while we're on this call. We have so many machines out there that like someone's nodes are getting replaced. Users don't notice. It's handled by us as a cloud provider. So it's a downside, right? When you wanna bring up new replicas, it's not instant 'cause you're not just attaching replication compute onto a storage engine and having it work that way. You can't just run one compute node and have a bank of storage 'cause in a disaggregated storage and compute architecture, hopefully the replication is managed by the storage system. So you can just have one compute. It's not highly available, but who cares? Like for some people, they don't care. So you get durability without that. So that's a real benefit. For all those downsides of our system, which are hidden extremely well to the customer, you get to talk to your disks inside the same machine, which is an order of magnitude faster than going out of the machine. It just is. It just simply is so much quicker. to do that, that you can see it in all of our benchmarks. If you look at, if you go on our blog, you'll find PlanetScale Metal benchmarks for either Postgres or MySQL. We absolutely demolish every single one of the storage and compute separation, storage and compute databases in terms of performance. We picked pretty much every single one of the major, we picked all the major ones and then the small, and even the smaller ones when it came to Benchmarking and absolutely nailed them internally. They just, they had nowhere near in performance. For some of them, we gave them twice the compute and they were still slower. So that's the benefit. At that point, I've lost interest in why any of the flexibility is worth it. But in their architecture, yeah, you can scale up compute dynamically from storage. That's fun. When you do that once a month or whatever, at most, that's, it's nice. That's fast. You can attach read replicas instantly. Again, when do you need it to be instant? I guess during an incident or an issue where you need more compute. That said, with Metal, the compute stretches way further and you are less likely to be having that incident in the first place because of how performant it is. I had a customer that went from, they probably, they hit us with probably about a consistent 300, about 3 mil Million QPS on Metal, and they accidentally sent us an additional 1.5 million QPS when another service came online after being offline and hit their database. None of us noticed till 2 days later when we looked back at the graphs because Metal just took it. Yeah. I mean, there's all these really nice benefits of separating storage and compute. Like, it's more, it'd be insincere for me to say it's not easy, more flexible. It doesn't help me get these Scale to zero, you can branch your data easier with a consistent time branching. Spinning up read replicas is quicker. Having one node is quicker. It's not worth slowing every single thing down. It just, it truly isn't. It really, every query is potentially twice as slow as it should be for the flexibility of adding read replicas instantly. That's just not worth it. It really isn't worth it. At the smaller scale, small-scale Auroras or whatever, it's really good. Like, it's great for people because they don't care that much. They want that flexibility and they don't even notice the performance. Yeah, it's good. But after a certain scale, absolutely crazy optimization, a crazy trade-off to make. It just really is. Yeah.

With the separation of cloud, like compute and storage, They have, they also have to cache a lot of like hot data. And then that also creates some sort of bottleneck. When things go down, either your latency spikes or you have to warm up your cache and all the strategy of what data to cache. And that becomes a little complicated and it's really hard to get it right.

It's hard. And you see a lot, you really see the downsides of separating storage and compute in your P99s. 'Cause you get these really errant, that is not at all predictable in terms of performance because you're going out of the network, you're going across some sort of balancing. Once that file system is doing some level of rebalancing or a node failure has happened within, it's, you can get these really crit— and then the P99s are the worst because it's when your biggest customers, they're experiencing problems. It's just tough. It's just, it's not, people will say it's, Completely trade-offless. That is obviously not true. If you believe that, you should question how well you understand how computers work. And if you're listening to it, you should question why someone is saying this to you. They're obviously like trying to shill something that's not good. Yeah.

I think it makes sense. What about replication though? I guess you use like semi-sync replication as in As soon as one replica has acknowledged a write, you acknowledge back it to the client. What are the drawbacks of this, or what are the trade-offs really?

It means you have to have 3 nodes because if you have 2, it's actually a higher chance of write unavailability. Because if you lose your rep, you're now, you double the chance of failure when you have 2 machines, right? You have 2 sets of power supplies. That are not supposed to go bang to maintain write availability. So you have to have 3. So if one goes down, you reduce your chances. You are obviously doing it like a network hop to land a write, but it's still, it's just an acknowledgement. So it's significantly faster still to do that, but you're still depending on another machine. But that's where you get your durability. You've made sure that the write has gone off to another node. It doesn't do conflict resolution, but you shouldn't be doing that anyway because you should always assume that there's no, like, that's a whole other field of databases that is way more niche and way more or less relevant to OLTP or web-style workloads. But you're not doing conflict resolution. That's fine. You shouldn't need to. If you need it, then you're doing something even sillier. Yeah. I mean, it has downsides, but it's just not as severe as, like, you're still doing that type of Machine coordination and replication at a disaggregated file system layer. With this, we're using the real replication protocols that are first party in the storage engine.

Yeah.

Meaning it beh— oh, that's actually, how bad of me. That's one benefit of disaggregated storage and compute is no replication delay. Now, of course, the queries going in were slower, so you've paid the penalty in some way. Right? You've, it's still slow, but once a write is acknowledged, a replica will be able to get that view of the data immediately 'cause it's been committed to the file system. So that's a nice benefit. So sometimes we've seen people come off of disaggregated storage and compute architectures. They've gained massive performance, but then they've uncovered that their app does some crazy thing, like at night, re— like some people just, these are the kind of things that work in the early days when you've got thousands of rows in the table, they do some, they read something, do some batch job that kind of modifies the data and writes it all back to the database in one big flurry. And that creates replication delay. And then they just have to, it's easy to mitigate. You just farm it into some worker process that just rate limits it a bit based on replication delay and you're fine. But it's an, it's a dynamic that people forget exists when they've moved to separation storage and compute. Again, how do you wanna spend your time? Do you want your app to be twice, 3 times, 4 times, 5 times faster and have to make sure you don't delay your replicas?

Yeah.

Which is harder to do on faster storage anyway. Still a trade-off that's not worth it. Yeah.

Again, I guess, depending on your goals.

And it really is. That's the worst part. Believe it or not, I'd be even less, I'd be even more insincere on Twitter if I was willing to not acknowledge that. Ultimately, everyone should just do whatever they want, whatever works for them or their business is the truth.

Yeah.

And truthfully, most people are at such a small scale that everything works. Use SQLite, just write to a file on disk. Who cares? That's the thing. That's where most arguments end up. Most of the people who are working on very large-scale systems are busy and don't care. You've seen the thing, you've seen the sort of scale HubSpot, HubSpot, think in thousands and tens of thousands of servers, not 15, like most people on Twitter who are arguing have a Hertz and a VPS. You're just like, cool, I'm not gonna convince you that you can't run your 5-user product on Redis. You can, and you should enjoy that. That's the point, really. That's the major issue with all of it, which is everyone's right bends based on their lens. And then when you start talking to people, you know, we have customers that have millions of servers. Yeah. That's a whole other thing. You shouldn't behave like them. You shouldn't make the optimizations they make. And you also shouldn't pretend that your little technology that works for you will work for them.

Yeah, I think it makes total sense. And there's a lot of trade-offs and decisions you have to make. And the best is to align those decisions with your use case and what problem you're trying to solve for the customer and the user. Cool. I think you also tweeted about an amazing deploy request. I think it was around 200 terabytes or something. Tell us a bit more about that. Like how long did the migration took and how, what was it that made it work? Right. How did it not impact other workloads? What could have gone wrong and stuff like that?

So I don't know exactly how long it took. Maybe a day, maybe 2 days. It depends. It's obviously a lot of data to move. While you're doing a deploy request into the test product, you're doubling the size of it. You're like, you have a shadow table of the old and the new versions of the table. So that's a lot of storage.

Yeah.

The thing that's impressive about that is one, sharding makes it very concurrent. So you're doing, that was probably like a 256-shard cluster. So, You're doing it in parallel across 256 shards.

Yeah.

Meaning that you're per machine, per shard, it's much less data being moved around. So you get really nice parallelism. So yeah, probably did take less than a day. It's very mature. The Vitesse system in PlanetScale is very mature. We rate limit. So if we see spikes of connections, spikes of queries, replication delay, The whole process backs off and can, and be, and can basically be paused. The copying operation can be paused and we just replicate from one table to the other. So it, you can tune up the aggressiveness so that it's very aggressive when it's copying the data or that it stays pretty chill while. So the cool thing is if it's like a lot of data, like hundreds of petabytes to terabytes to move, or petabytes, it's done. You'll see during people's peak, it slows right down. In the off period, it'll spin back, speed back up again. And then it just will complete a fully online schema change across all of those tables. Yeah, it could go wrong. It could create rare operations, like it could create high load and contention on every query that causes a thundering herd that slows everything down. It could, you could run out of disk space while doing it. You could just do, like, many things could go wrong. Through careful engineering and how robust and how many times these things have happened, it works really nicely. Like you could be doing it on 5 gigs of data and it works very smoothly and feels almost instant. Or you can do it on 200 terabytes. The other cool thing is it could have been, you could have rolled it back in 4 seconds if you'd have been, if it had gone into prod and caused a problem, 4 seconds and you'd have been rolled back out. We're using our rewind feature, which is just extremely cool.

Is it also based off of branching or?

You branch to do the schema change and then you're merging that branch, which deploying that branch, which then will go and do that.

So next thing I wanted to talk about is the uptime and like, how do you manage to get such an amazing uptime? Because a lot of these applications, like a lot of simple applications that they do a lot of, the database does a lot of heavy lifting for them, right? And if the database availability is down, it impacts like the overall availability. I think you also mentioned some, I don't know, some mathematical formula around If your database is just 99% available, your application and the load balancer needs to be 101% available to meet your SLOs. So your database availability is the core, right? And if it gets bad, then you don't have any control basically. So I think availability is really important when it comes to the database. So what are some practices that you follow? And given that your team is really small in size. I don't know, 55, maybe more now. I read some number 55 some day ago, I guess.

60, 65 now. But yes, it's still very small in comparison.

Still very small. Absolutely. Even if it's 100, I think it's really small. So what are the practices you follow to ensure that the availability guarantees or the SLOs are met that probably others could learn from?

So it's a two-sided coin. I'm mostly, most proud of our availability in a world where we don't control the clients. Like we, we are a service and at the end of the day, we don't write people's apps for them. So their apps can still do ridiculously bad things and we have to defend ourselves. You can't always do that. Like you just can't stop People spinning up 10 times more front ends capacity than they should, and then just smashing the database. Like, you can just take a database down, just exhaust the connections on it. Just deploy more like web workers from your platform than you have database connections. That's the database finished. What can we do there? We don't know which of your connections are the priority ones that you just gonna overwhelm your database.

Yeah.

So we can't do everything for customers, but we can defend ourselves the best we can. And then we can also build software that is resilient. So we have what's called a shared-nothing architecture, which is not possible with separation of storage and compute, by the way, because you are sharing layers like the storage system. So by a shared-nothing architecture, each shard has 3 nodes that do not share a unified storage layer. They each have a storage layer inside the computer called the disks. So if one server explodes, that's fine. Like, it's completely fine, right? Like, the whole stack has gone inside that server, but that doesn't matter. You have other versions of that. So you don't share this consolidated layer. So a bug that impacts one of the nodes in that system just takes down one of the nodes. Where if you have a shared storage layer, a bug in the storage layer A bug in the load balancer, a bug in the network can cause those things. So you are sharing the fate of those things. So you strive to have a shared nothing system, meaning you have these kind of overlapping Swiss cheeses, which means that at any point it's possible to get through those to get a request out the door. We then default to high, like we default to multi-AZ deployments. Every 3-node cluster has a node in every availability zone of the region that you're in. You can do multi-region if you wish. Most people shouldn't do that, but you can, and it's totally doable. We are very sensible about how we roll out software. We don't use niche and silly architectures. We just, we use the storage engines the way they're meant to be used. We follow the best practices and we learn. That is the most key thing is we learn. We treat our organization like an immune system, and if we get an infection, we deal with it really quickly. And those infections are things like outages. We've never caused a full, like on a multi-sharded, on a sharded database, we've never knocked out a full database. Like you can, there's been times where you say the primary node of a shard has had something happen to it, but that's partial availability. That's the best you can hope for in a system. But when we have incidents and issues, and we do, we have caused people problems. It's a database service, so you can't, nothing has 100% uptime. It's impossible.

Yeah.

You react, it's the reaction to that is the most key. So I'm on call 24/7. If it's a customer-impacting thing, I'll get paged. I'm not gonna engineer, like the engineers do that, but I'm there like to respond to customers, to status the website and do all these things. and make sure we work to do follow-ups. But I don't even need to do that. I've never had to tell an engineer at Planescape to follow up on an outage, ever. They just do.

Yeah.

It halts all work until we make sure that can't happen again. And now we've gotten to the point where the operators are so mature, the Kubernetes operators, when we do see novel failures, we get extreme interest. Like everyone stops to go and figure out why. And make sure that can't happen again. If you keep doing that, like an immune system, keep making the antibodies, you build something that's very mature. In a lot of cultures, people don't do that. They, and this incident happens and they think that won't happen again, or that's a lot of work. We'll do some other thing if that happens or whatever. I don't know. We just take it very seriously because we're a database company. It's like job number one.

Yeah.

This is why when other database companies say things like, we have this feature, we have this thing, as a, some sort of response, as if it's some Valid counter to our availability. Silliness. I really, you would, if you buy an air, if you're like buy an airline ticket, you'd like it to be comfortable and whatever, but I hope they're focusing on the plane staying in the air to a much higher degree than whether the in-flight entertainment is good. That stuff's good. You can have that stuff when the planes aren't dropping out of the sky. So it's like this hierarchy of needs for us. Never lose data, never go down, never let customers down. And then it comes, even it comes down to decisions you make before anything has ever gone wrong. Like, how do you want things to go wrong? If you are gonna fail safe, like how should that present itself to the user so they can do good things? If you try and fill your nut, if you try and like completely exhaust the storage in a PlanetScale node, we put you in read-only mode first. That's an issue, right? If someone hasn't scaled up, but we've chosen to make writes unavailable before reads, right? So that you can respond quicker and serve some users. Like, you just have to make these trade-offs essentially of how, when things do go wrong, how do they go wrong sensibly? How, and how do you deal with that? And then just be highly adaptive to things that change and go wrong. It's hire smart people that will think about the edge cases. They'll think through scenarios and they'll talk about it and It's just a lot of it comes with experience. It's, I'm convinced unless your leadership team and your engineers have done this before and learned the rules and screwed up many times, you can't just retrofit this into your culture. It has to be learned by doing things at scale with consequence. Otherwise, it's just not possible. Yeah.

And as you said, like learning from it is extremely important. So I guess it boils down to Customer obsession, doing the right things as in standard things, not doing silly things, prioritizing the right things as in like inverting the equations. For some products, maybe cool features and APIs and some other stuff might be table stakes, but actually reliability, availability, and customer support must be table stakes and the others.

Yeah.

Okay. That makes sense. I am, I'm always a bit paranoid about things whenever we're talking about databases and backups. And so it's good that we follow standard practices, but unless we regularly test these things, and because there have been many incidents where data loss happened, and then after the incident had happened, they realized that, oh, the backups were also not working. How do you make sure, or what kind of testing do you do to ensure these things are tested regularly? Right? Because these things can be missed.

Our backup system, which by the way, this was a recent thing that got debated on Twitter, has to be an independent system that makes a logical backup of the system. You can't trust a single system to back itself up. That is not how that works. It is not immune from bugs. You have to have a separate isolated system that takes a logical copy of the dataset. Into a separate location and cannot replicate issues or bugs. So that's one, the system is that, so it's a real backup system, first of all. And second of all, it validates its backups every single day. So it's very smart how this happens. So the way backups are taken at PlanetScale is that we, first of all, reinstate yesterday's backup, or not even yesterday's, The last one, right? You can back up every 15 minutes if you want on PlanetScale. Depends on how much data you have, obviously. Otherwise you get to overlapping. The default is a 1-day backups, and then we store bin logs as well, so we can get you back to the previous state of where you are by rebuilding. But the way this is done is we restore the logical backup and then we replay the logs from current world back to that backup and apply them over the top. So that gives, that fully runs, that fully ensures that the backup system is complete and not corrupt. And so we'll actually restore and then applies all of the changes over the top, and then we back that up. So you get this constant testing of the entire system and the data. To ensure constant delivery of backups, basically, meaning you will always know you have a restorable, non-corrupt copy of the data from that moment in time. Okay.

And yeah, I was also following that debate and that's an interesting one.

But terrifying.

Cool. I think these are really important insights and thanks for clarifying all the doubts I had and elaborating on all the tweets and takes you had on databases. I have tons of more questions, but unfortunately we have limited time. I think it's best to wrap up. So, because we have covered a lot of ground, different types of scenarios and questions, I think it was interesting and it is going to be very interesting for our viewers. I hope you also enjoyed it because I did.

This was really fun. Thank you for doing all the research and having me on. I'm sure I'll come back and we can At some point and we can do, we can go through even more stuff, but I hope your audience enjoyed it. I hope they check out PlanetScale and if you disagree, tell me, I would love to hear.

Yeah, I think Sam is pretty active on Twitter and if you just follow him, you'll try to figure out what he's talking about. You can debate, you can argue, whatever. But yeah, do check out PlanetScale. It's amazing. And also, if you like this episode, please don't forget to hit the like button. share it with the network and subscribe to the channel. Thanks a lot for watching. See you in the next episode.

Thank you.