PlanetScale - Sam Lambert, Open Source Serverless Database

This is Floss Weekly. I'm Doc Searls. Jonathan Bennett and I talk this week with Sam Lambert of PlanetScale, which does MySQL and far more based on Vitess. That is coming up next.

Podcasts you love.

From people you trust.

This is TWiT.

This is Floss Weekly, episode 684. Recorded Wednesday, June 8th, 2022. PlanetScale. This episode of FLOSS Weekly is brought to you by New Relic. That next 9 PM call is just waiting to happen. Get New Relic before it does. And you can get access to the whole New Relic platform and 100 gigabytes of data free per month forever. No credit card required. Sign up at newrelic.com/floss. And by Kolide. That's Kolide with a K. Get endpoint management that puts the user first. Visit kolide.com/floss. That's kolide.com/floss to learn more and activate a free 14-day trial today. No credit card required. Hello again, everybody, everywhere you are. I am Doc Searls, and this is Floss Weekly, where I'm joined this week By Jonathan Bennett himself. There he is. In his hideout in Oklahoma someplace.

Yes, it's kind of a stormy hideout today. I got woken up first thing this morning with my wife saying, it's really dark outside, and kaboom! So I may get interrupted by thunder, but that's all right.

But you also live in a part of the country where things get sucked into oblivion by tornadoes, right?

I mean—

I mean, they call it Tornado Alley for a reason. Yes.

Yeah. It's a thing. And you have earthquakes there now too.

Very, very minor earthquakes. The meme about that is it's 2 lawn chairs with one of them fallen over and we will rebuild. It's about how serious it was.

So, so how are you doing?

I'm good. You know, it's been kind of a scramble for the show today because at first we thought we didn't have a guest and then, oh no, we have a guest. And then Simon had something come up. So I've been off and on and off and on and here I am.

You've come off the bench, even though you do start on a lot of shows. Yeah, it's great having you here. Are you familiar with Sam Lambert, our guest that we have on today?

Not him personally. I did spend a few minutes looking at his company and the things that they provide. And I gotta say, I am fascinated by this idea of big distributed databases. And, you know, there's kind of a law, a theorem, the CAP theorem, they call it. Essentially, it says, There are 3 things that you really want, but you can only ever have 2 of them at once. And I always find it interesting to ask, you know, when we have somebody that's a distributed database, how do you tackle that problem?

Yeah, and I'm sort of interested in following up on some of the stuff you brought up on another of our shows, This Week in Enterprise Tech, which was last summer. So if you're into this, you might wanna go back and listen to that one as well. Wanna get into it, got off to a little bit of a late start today, by letting everybody know that this episode of Floss Weekly is brought to you by by New Relic. If you're a software engineer, you've been here, it's 9 PM, you're finally unwinding from work, your phone buzzes with an alert, something's broken, and your mind's already racing at what could be wrong. Is it the backend or the frontend? Is it global? Is it the servers? Is it the network? Is it the cloud provider? Do we have slow-running queries? Did I introduce a bug in my last deploy? Now the whole team's scrambling from tool to tool and messaging person after person to find out and fix the issue. According to a New Relic report, Only half of all organizations are implementing observability for their networks and systems. The report showed how maintaining network observability continues to be an issue for companies around the world. That won't happen if you get New Relic. New Relic combines 16 different monitoring products that you'd normally buy separately so engineering teams can see across their entire software stack in one place. You'll get application monitoring, that's APM, unified monitoring for all your apps and microservices. Kubernetes and Pixie, instant Kubernetes observability with Pixie. Distributed tracing, see all your traces without management headaches so you can find and fix issues fast. Network performance monitoring, stop guessing where performance issues start and ditch data silos for a system-wide correlated view. And so much more. More importantly, you can pinpoint issues down to the line of code so you know exactly why the problem happened and can resolve it quickly. That's why the dev and ops teams at DoorDash, GitHub, Epic Games, and more than 14,000 other companies use New Relic to debug and improve their software. Whether you run a cloud-native startup or a Fortune 500 company, it takes just 5 minutes to set up New Relic in your environment. That next 9 PM call is just waiting to happen. Get New Relic before it does, and you can get access to the whole New Relic platform and 100 gigabytes of data free per month forever. No credit card required. Sign up at newrelic.com/floss. That's newrelic.com/floss. newrelic.com/floss. Okay. So our guest today is Sam Lambert of PlanetScale. He's an engineer, angel investor, a CEO at the company building a next-generation cloud database. He was VP of engineering at GitHub. He was responsible for creating GitHub Actions, a popular workflow automation tool. Before that, he led the traffic and video infrastructure teams at Facebook. He also sat on the Cloud Native Computing Foundation Technical Oversight Committee and is an active scout at Sequoia Capital where he invests in early-stage startups. And the list goes on. Welcome to the show, Sam. It's great to have you.

Thank you. Thank you for being here. It's really awesome to be back.

So are you new to the title? I think I saw a different title for you when you did This Week in Enterprise Tech last summer.

Yeah, I've been CEO for about 8 months now.

And where are you located? Your accent suggests to me a different time zone than we're in.

Yeah. So originally I'm from the UK, but I'm actually based in San Francisco now. I've lived here since about 2015.

Yeah. Well, originally I'm from New Jersey. I was based around San Francisco. Now I'm in Bloomington, Indiana. So we get all over the place. So tell us a little bit about PlanetScale and what makes it different. And we're especially interested in the open source angle of running a company that depends on open source and is deeply involved in open source development and the community and how that differs from the kind that isn't.

Yeah, I'll start with the open source angle actually. So we are the— PlanetScale are the maintainers of Vitesse, which is an open source database orchestration and sharding platform. It's built on top of MySQL, another open-source platform. We maintain the project and we— it's a project that gets contributions from a number of very large companies that run it at scale. It's the backend for a bunch of large websites and services that we interact with today. PlanetScale is the cloud offering on top of Vitesse. So we allow developers and engineering teams to use Vitesse and leverage all of Vitesse's power in an incredibly easy-to-use, simple, developer-oriented system. You'll see that there's like a bunch of features and kind of our mental model looks very similar to the Git mental model, and that comes from our experience at GitHub and our focus on developers. And so it's kind of the pairing of this really powerful commercially available kind of— the most powerful commercially available database, uh, that is open source, uh, as the underpinning for this, this developer platform that we have.

Hey Sam, I want to jump in, and I, I saw something just announced on your blog over at PlanetScale, and that's PlanetScale Insights, which looks really interesting. Um, it also reminds me a lot of— I believe the project is called Netdata. I've been recently fiddling with because of something else I'm working on. Uh, tell us a little bit about Insights, and then I'm curious whether that was actually built on top of NetData.

It's not built on top of NetData. It's actually built on top of some of the primitives we have in Vitess. But yeah, it's, um, it's a really exciting, uh, new feature. You know, people want to be able to look at what their database is doing, and we all— like, everyone is guilty of writing a bad query now and then, and Often you find out by taking your website down. And Insights is the kind of tool that gives people the real-time ability to see what's going on. And the reason it's very, very different to traditional kind of database performance monitoring tools is that they usually sit on the outside and kind of look at the database. So they'll do like show process list, or they'll try and look at what queries are running. But they can't catch every query. And if you try, you will cause such an overhead on a system that it becomes self-fulfilling that the system is having issues because of the monitoring. So the really cool thing about Insights is when we see the query, when Vitesse gets the query, we immediately send it through a data pipeline to a data warehouse without— before it hits the database. And then we can actually monitor the results that come back from the database, and it will— we can then look at those results. So we can actually show you every single query that happens against your database, as opposed to what usually happens, which is sampling. Most other tools that are like third-party bolt-ons to the database are usually sampling. And that means finding random, errant, problematic queries is very, very difficult. And I've spent weeks of my life hunting certain queries that were causing problems in production. We had a really weird issue at GitHub once. And because you can't log every query, you have to just look at a sample. And it's much more difficult. Now we can show you down to literally the exact extent instantiation of exactly the query that caused you a problem. And that is extremely exciting. It also means that we now have a bank of your query data that we can run analyses on and start to give you better insights. And that's obviously why the product is named how it is. You can also see when you've done a deploy request on PlanetScale, you can actually see that overlaid against your performance metrics. So say you add an index and it has a really great result on query latency, you're actually able to see that directly inside the Insights product.

Yeah, very cool. We've got a question coming from the chat room, and these are some of the things that I wanted to ask about too. JoJo Dancer is asking, the serverless databases are interesting, but how are they really implemented? Let's start with that one. Serverless is kind of a lie, isn't it? There's actually a server somewhere, right?

There are servers. It would be great for margins if there were no servers, but there are definitely servers. So the way to think about serverless is, honestly, it's a movement. And it's kind of a movement when it comes to rejecting how the cloud has been so far, right? So it's been very, very complicated, right? You go and use a cloud tool right now. And yeah, they're not like— you're not having to rack servers anymore. You're not having to configure things yourself. But you still have to know way too much about actually configuring the database. And that's not the full promise of the cloud, in my opinion. The full promise of the cloud is taking that unit of scale, the server, building an abstraction on top of it, and meaning that people can just come to your service to do fundamentally what they're there to do, which is store data and query it. So there's definitely servers, but we really take on the goal of managing it. So you get that end goal, you get that end kind of value, which is just exactly how your database should behave without having to understand replication zones, CPUs. Like, you know, the concept of a vCPU, it, you know, You see this in products, you sign up for these databases, you have to select how many you need. You don't know how the database performs. It's too much. It's too much administration. It's too much messing around. So we chose to go serverless because we wanna build something that has an immense amount of end-user value without having to mess around with backend infrastructure.

And so what— this is something else that JoJo asked about. What is the process of kind of plugging an existing app into this? If I have something that supports a MySQL backend, can I simply give it, say, a DNS address that points up to either, either a Vitess install or to you guys and it happily runs and works?

Yeah. So you can, you can actually, you can actually import a MySQL database into PlanetScale on Vitess very, very, very simply. In fact, if it's cloud hosted on, say, Amazon, for example, or Google Cloud SQL, the process for migration is actually very, very cool. And I have not seen anyone doing this. Usually you have to like dump the data, restore it into the database, kind of get the database to catch up. That's cumbersome and very annoying.

Right.

Due to some of the really cool primitives inside Vitess, we've been able to make that really simple. So the way you do it on PlanetScale is you give us the credentials of your current database. And we start to harvest the data in, in the background. So we don't dump the data. We just start to nibble it in. Once we're up to date, then you get the option to redeploy your application with PlanetScale's credentials. So your main database in Amazon is still running. Your data is live replicating into PlanetScale from that database. And then you redeploy your app to talk to PlanetScale. And we will proxy back through your original database. And then when you decide to do a cutover, we will literally swap the roles. So we will make the old database a replica of PlanetScale, and PlanetScale will become the primary. And so you actually can do a fully online migration into PlanetScale thanks to Vitesse without having to like plan downtime and do all those cumbersome things you usually have to do.

Very cool. And so I assume the application sees PlanetScale as just another MySQL server. There's no, there's no magic that the application has to hook into or know about, right?

Right. There's a few caveats. There's a few things that we don't fully support. But yeah, everyone who, you know, is using MySQL can pretty much come in and start to leverage some of the things about the platform, which is great for us. You know, MySQL community is absolutely huge and there's so many applications out there that can benefit from our platform. And so, you know, it's really exciting to have that compatibility.

So we're talking about MySQL and there is kind of this rift in the MySQL community. And I'm curious whether this has impacted you at all. And that is the MySQL MariaDB fork. Where do you guys sit on that?

No, I think, I think the wonderful thing about open source is that people can go in whatever direction they see fit. And MariaDB are doing fantastic work. And so Oracle, honestly, I know people are going to hate me on an open source podcast talking about Oracle, but they do know databases and the MySQL 8 release is just fantastic. And we see patches from Facebook, from Google, from all of these companies that are still contributing to MySQL, making it better and better and better. So open source is alive and well in either fork. But there was a big concern for a long time that Oracle were going to cause some real damage. And it doesn't seem to have happened. It seems to keep going, keep getting better.

Well, I think the MariaDB fork is essentially the stick that keeps Oracle in line. Wouldn't you say?

Yeah, it's great. That's the beauty of open source, right? Like, you know, it could go any— it could go anywhere, and, and, uh, we can build upon those things and people can create their own services and carry the torch forward. We actually have— we run our own fork now internally to power some of the cool stuff around Insights and some of the features that are coming over the years. So we are able to own our own destiny there too.

Uh, are there aspirations for some of those MySQL features eventually getting pushed upstream?

Yeah, I think we talk about it. I think if it's something that's very generally usable, I think the right thing to do is to send those back and contribute to the community alongside a lot of other folks, as well as doing it with Vitesse. It's something that we're very tuned to do is contribute to open source.

All right. Now there's a term that has been used in some of the documentation. I think you mentioned it too. And I'm still trying to wrap my mind around exactly what do you mean by sharding when it comes to PlanetScale? What does that refer to?

Yeah. So sharding, it's a funny term. Sharding is essentially breaking your logical database into many databases across many machines and segmenting. You can do it by user, you could do it by product, you could do it like— there's very many ways you choose a sharding key that essentially allows you to distribute your data into these buckets or shards. And it means you can gain huge horizontal scalability. So you can just keep adding machines horizontally and scale writes and reads in a very flexible way, which is obviously something that's quite hard to do at the database layer, especially when you've kind of built your application to run on a single host and you run out of space or the performance of a single machine— you've outgrown the performance of a single machine or a set of machines. Sharding becomes a solution to actually take it to a very, very large scale. And it's a Very common pattern used by large-scale database deployments.

It's sort of a— to borrow a C paradigm, it's kind of a hashed key lookup table where you have some method of breaking your data into a repeatable way to break your data into different groups, and then you assign a server to each of those different groups.

Correct. And it makes your failure domain smaller if you have a failover. within one shard, it's some smaller set of customers that are affected. It gives you just a bunch of flexibility across your database layer.

And so I'm curious how this works with, and I mentioned it in the kind of the start, the CAP theorem. That's the idea that you can have consistency, availability, partition tolerance, but pick only 2. You can't have all 3 at once. It almost sounds like this is a workaround to be able to have all 3.

It's not quite all 3. It's the way of thinking about CAP theorem as it applies to Vitesse is you can actually satisfy any of them as long as you understand the trade-offs. And you kind of satisfy on a more per-request basis. So consistent, you can be consistent, you can be available. If you partition your data into shards, it means that partition tolerance is less.

Yeah.

Problematic because it's just by— it's harder to partition the data. So it can satisfy all, it depends what you need for your application.

All right. And then I'm curious about kind of the relationship between PlanetScale and Vitess. You said you guys are kind of the benevolent dictators of that project. What does that community look like?

It's an awesome community. If you look on our website, you'll see a list of logos of folks that are in our community. We have Slack, Square, Roblox, all of these companies that are contributing, working on Vitess ourselves.

We—

it's a very, very large project. It's actually one of the oldest Go projects in existence. It started on Go 0.1. If you look at the kind of Go docs, they call Vitess out as a— they thank Vitess and the Vitess team for giving feedback to the language. very, very early on. And when it became an open source project and part of the CNCF, it took— people just picked it up and ran with it. That's how I got my first kind of exposure to Vitess was GitHub. We needed to scale our database, same thing everyone has to do. We took Vitess and absolutely fell in love with it. And, you know, it'd been a lot— most companies before then were home rolling kind of their own sharding. And that's really, really painful and difficult to do. Google tech come out that has solved this at colossal scale. Vitess started at YouTube and was YouTube's main database with billions of active users. It was pretty awesome to get that to come out and become a really generally usable open source project. So it's seen really great success.

And so there's quite a collection of people that use Vitess directly and don't go through PlanetScale. What do you think the kind of the breakdown, the percentage is of people that use Vitess versus people that use PlanetScale?

There's more people who use Vitesse through PlanetScale, although they may not directly know it, if that makes sense. They know they're using PlanetScale. Vitesse is the kind of engine under the hood, and there's a lot of people using it on PlanetScale. And then there's obviously our larger customers that are running it on-prem themselves. In terms of traffic, I'd say there's more traffic going through non-PlanetScale Vitesse right now, but that will change over the years.

But you guys, you guys offered support contracts, I'm sure, for big companies that still want to do it on-premise.

We do work with very large companies. Yeah, because we, you know, they, they are contributors to the project as well. We work with them on features and, and bugs and generally just making it better in terms of managing the project. You know, when it, when it, when it scales and manages such large projects like JD.com, it's like you see on screen now, it's just a massive, massive deployment. It works better that we all contribute together and manage the project well. And it is a very well-managed project. It's one of the very few CNCF-graduated projects and has been for a long time.

All right. Hey, we have one question that is interesting to think about, again from Jojo Dancer. He is one of our most prolific commenters. And he wants to know what the command line interface is like. And this is an interesting question because This is one of these things that you shouldn't ever have to do with MySQL, and yet you find yourself doing it all the time with MySQL, and that is pulling up the MySQL command line and manually fiddling something to fix something. How does that work with Vitess and then also with PlanetScale? How do you do a command line interface when you've got things spread out to the moon like this works?

Well, the great thing about Vitess is that it can transparently shard for the application, so you don't need your application to necessarily know or understand that it's sharded under the hood. So you can actually just connect with a MySQL CLI and it works. Now, there's caveats there that on certain profiles of application with various types of performance, you might need to structure your data differently, but it pretty much behaves quite similarly to MySQL and is certainly compatible. Like your client, for example, would be the same MySQL client you would be using with vanilla MySQL. When it comes to the command line for PlanetScale, look, it's very easy and flexible to use. You can create databases, you can create PlanetScale branches, you can do all of these various things very, very simply from the command line. And people use the command line to script PlanetScale into their CI/CD process, which is a really neat way of doing it. And it's super flexible, super usable. You don't have to do any real administration from the command line. There's no messing around having to do that. We handle that for you on the platform.

So I have a question coming up about Rewind, which I know is something you care about, and schema migrations. But first, I have to let everybody know that this episode of FLOSS Weekly is brought to you by Kolide. Kolide is a new take on endpoint management that asks the question, how can we get end users more involved? This is in contrast to old-school device management tools like MDM, which lock down your employees' devices. Without considering their needs or even attempting to educate them about the security of their laptop. Kolide is built by like-minded security practitioners who in the past saw just how much MDM was disrupting their end users, often frustrating them so much that they would throw up their hands and just switch to using their personal laptops without telling anyone. In that scenario, everyone loses. Kolide, on the other hand, is different. Instead of locking down a device, Kolide takes a user-focused approach that communicates security recommendations to your employees directly on Slack. After Kolide is set up, device security turns from a black and white state into a dynamic conversation. This conversation starts with the end users installing the endpoint agent on their own through a guided process that happens right inside their first Slack message. From there, Kolide regularly sends employees recommendations when their device is in an insecure state. This can range from simple problems like the screen lock not being set correctly to hard-to-solve and nuanced issues like asking people to secure 2-factor backup codes sitting in their Downloads folder properly. And because it's talking directly to employees, Kolide is educating them about the company's policies and how to best keep their devices secure using real, tangible examples, not theoretical scenarios. Kolide, cross-platform endpoint management for Linux, Mac, and Windows devices that puts end users first for teams at Slack. Get endpoint management that puts the user first. Visit kolide.com/floss to learn more and activate a free 14-day trial today, no credit card required. Visit kolide.com/floss today. And right now, you get a goodie bag of Collide swag after signing up for a new trial as their way of saying thank you. Okay, so Rewind and schema changes and schema migrations, I know you talked about that in the prior show, and I'm wondering, you know, what is it that Rewind does?

Yeah, so Rewind is just awesome. I love— like, Rewind is So I get very— you have to excuse me, I get very excited by our own product just because I spent years in databases. And it was— I have so much of the brain damage that, uh, caused by having to admin databases at scale for a long time that now we're making it simple.

That's where you're the CEO.

Yeah, right. Um, and so Rewind is, uh, first of all, it's a world like no one's done this before. We're really, really excited to actually bring this to folks. This is leveraging Actually leveraging some methodology we built from GitHub in another open source project we open sourced from GitHub, which was called Ghost, G-H-O-S-T. We've combined the methodology and the guys that built that are at PlanetScale. We built that into Vitess and then we, the final loop was building Rewind. Essentially Rewind is an undo button for schema migrations. Schema migrations are tough as it is. Most people have to use open-source command line tools to do schema migrations against MySQL and Postgres and other tools like that. And it's cumbersome and difficult. And you don't get an undo button. So let's go through a scenario that actually happened to me in my first week at GitHub and took the website offline, which was very unfortunate and very scary. You say you think you've cleaned up the models in your codebase. You're no longer— using certain columns. You want to tidy things up and refactor things. So like, we're going to get rid of some columns in this scenario. And you've got a large application. You think you've managed to refactor it. So you go to the database, and you go and drop these columns. You run whatever CLI command to do it, and the columns disappear. And suddenly, your application crashes because it was actually in some background worker process. It was still using those columns, and they now disappeared. All of the application's queries are failing.

Now everything's done.

This happened all the time. This happens constantly. And this happened in my first week. And you get this terrible feeling. It's like, OK, so we either just add an empty column, roll forward, and then hope for the best— not good— restore from a backup on another machine, add the column back, move the data over. We're in an incredibly high-stress situation right now. And there's been some very, over the years, extremely public cases of— like dropping data, losing it, and it going wrong. But in this world, in the PlanetScale world, once you've deployed your schema change, which you can do like code, we have a primitive that is like the pull request called a deploy request, which means you can collaboratively work on your schema and deploy your new schema and it rolls into production online. Once when you realize you've screwed that up and something has gone wrong, you can literally press the undo button and it will revert the deploy request. It will bring the column you dropped back with the data, and the intermediate data that got written to that table will still be there as well. So it is a true rewind and undo for your database schema deployment. And so in the case of GitHub, instead of being down for 2 hours, we would've hit that button and 3 seconds later the site would've bounced back with absolutely no side effects.

I just have to say, that is beautiful. As someone who has crashed their own services doing similar things, that is a gorgeous idea.

I love it. Thank you. We're very excited. In fact, in our first week, people were writing in and being like, you just saved me. And I was like, yes, this is exactly what we want. And this kind of builds into the overall PlanetScale philosophy, which is we have a very good backend database. That's proven. I mean, you've seen the companies that are running it. People are probably chatting, listening to this and chatting in Slack right now. Every message you're sending is hitting Vitesse directly and working there. We know that. We know we have a great database. But at PlanetScale, we don't— we take that a step further to obsess over what developers do with it.

Yeah. Right?

No other database platform is thinking about, like, what does the life of a developer look like against the database? Why is the database scary? Why are schema changes difficult? Why can't you deploy it like code? We're going there, and we're trying to solve all those problems for people too.

You must have so many stories. I mean, it seems to me you're taking a fairly dry subject and making it— and really putting it on the ground in a strong way. And I'm wondering, I mean, you know, how you're constantly reconciling what users are doing, not just developers, you know, and your customers, but how does usage get back? I mean, obviously Rewind's a handy thing for sort of intermediaries there, but, you know, for ordinary folks that are just trying to keep up, how does that look to them? Or maybe it's so transparent they don't even know. Just sort of like thinking about the end user here.

Well, so we try and make it extremely simple for the end user. We really, really think about what their experience is. And for databases, you've got to be careful. There's a lot of gotchas and things you can run into. And it— like, databases are almost like a cake mold. Like, the constraints that you create, your— the applications kind of get baked into that mold. So we, we try and be really careful with the constraints we, we give to users. And we try not to make it— it has to feel magic, but it ultimately shouldn't be magic, right? People should know where the— like, You know, you're on the Tower of Terror ride and whatever, you should at least know where the fire exits are. Um, you want to believe, you want to have fun, but you also, at the same time, it's a database, it has to do the things it's supposed to do. So we, we constantly ride this fine line of beautiful, easy to use, but feels powerful and has heft. So it's almost like Vitesse is a really powerful engine, and now we have to really focus on the interior and the handling and all the other things that make that super usable.

So I'm wondering, this is a separate thing. Recently, there's a— at the end of May, there was news about IPv4 and IPv6, over 3.6 million exposed on MySQL servers around that. Are you up on that? Or do you have anything to say about that one?

It's a very common thing that happens with a lot of products and platforms. I think this is actually why the cloud— I mean, we weren't vulnerable to this, but I'm— I think it's, it's a, it's actually why the cloud may have some benefits, right? Because when you're running a SaaS product, you have dedicated security teams, you get pen tested. I think often if people are installing things and self-managing, they get, they get these foot guns of defaults that are usually— defaults are usually there to enable the adoption of the product, but then, or the platform or the tool, and then you run into the foot all the nasty foot cannons that come with, with this, and you end up with security vulnerabilities. And I'm sure there's like security researchers out there that have a very terrible picture of the world and what and how insecure the internet actually is.

It was not a vulnerability in our code. It was a vulnerability in the example configuration that we shipped with our code. It's not our fault.

I mean, I think you touched on something there, and not only on how the cloud has expanded as a resource, but how SaaS itself has helped open source. Because now you have a layer there that's actually accountable. I mean, one of the arguments against open source 1,000 years ago was, I don't have a throat to choke if something goes wrong. But with SaaS, you do.

Yes.

Is that a fair assessment, do you think?

Yeah, it's true. And we also then have corporate sponsorship behind projects that can make them even better. As long as they are good stewards of that, and it's a fair exchange of value, then I think it's fine. Open source has these challenges. We learned this at GitHub. Without corporate backing, some open source projects are left to incredible people that maintain critical pieces of the internet, essentially unfunded. That's a very fragile place to be. I think it's the guy that maintains libssl. It's like 1 or 2 people. And they do it themselves and get a bit of sponsorship from companies. These people do incredible work, and it's thankless work. Being an open-source maintainer is incredibly tough. When we take it on as a company, it's people's jobs. They take vacation— vacation, sorry, braces— vacation. They have coverage. They have a healthcare plan with us. They have all these things, and we can continue to Contribute to open source and make it better without it being kind of a burdensome task on the maintainers.

I just want to add for the listeners and viewers, and you mentioned braces, you're talking about ones that are in your face, right? Rather than some software thing. Yeah.

I had braces yesterday, Monday, so I'm still getting used to these.

Oh my gosh. That's tough.

Yeah. You're a brave man to do the interview that soon afterwards. I was very excited to be here.

Very excited.

I'm excited to be here.

Yeah. So I kind of wanna dig into that a little bit more because, you know, that is quite the challenge around open source because you have things like NTPD, which everything uses, and it's literally one guy who would like to have more money for it. Then you have obscure things that everything uses, like the list of time zones around the world, or the list of the way computer terminals work, and I'm forgetting the technical name for that, but there's this really obscure database that describes all of the different ways a terminal could be set up. And if something breaks in that, your command line breaks altogether.

Yep.

And there's not— let me put it this way. As open source, we've not necessarily figured out how do we fund these things, how do we make sure that the next generation of developers is gonna work on some of these obscure but important projects. I don't know, I'm curious, what do you think the answer is there? You've obviously put something together that works well for Vitesse. Do we just need a company for every one of these projects? Or, you know, there's some other things that people are trying to make work like Tidelift and other projects like that.

I'm curious your thoughts. I think people— I think corporations need to be responsible and go back and fund the projects that they rely on, both for their own, like, business continuity, because they'll find out eventually, right? If a project gets maintained, I mean, the risk is there, right? And it's still not simple. I mean, like, during the GitHub acquisition, we spent a bunch of time going through and auditing all of our open source use, because there's risk, you know, there's risk to using certain open source, and there's licensing problems.

Sure.

Licensing is a really hot topic when it comes to open source and security and compliance. And so I think One way to make this sustainable is, you know, if you're a developer at an organization and you're using a great open source tool, nag your bosses to give you some money to sponsor the projects that you use and help them become sustainable and allow people to build sustainable lives around maintaining open source. And then I think we could— it could go a very, very long way. And companies and corporations are starting to look at doing this. But then there's always got to be some other angle, right? Like, you don't— we— it's a generational thing, right? People now appreciate open source more than they ever have. And I think companies are starting to donate and sponsor and fund open source for the good of open source. And organizations like the CNCF and the Linux Foundation do great work helping move this along and defending open source. But there's some companies that expect something in return, right? They expect a fundamental change to the project if they're the ones paying for it, or they want this thing that only they will use that requires all this work. So it's still in its infancy in terms of how corporations relate, but it's getting there. And then we had the open-core business model, which it works for some people, doesn't work for everyone. I think this new generation of SaaS and open-source frameworks that work best on the SaaS that produces them, that kind of thing could be really good for sustainability.

Sure.

Speaking about that, bringing it back, there's something I like to ask projects. I don't know if you'll have an answer for it, but if you do, I anticipate it'll be really interesting. And that is, what is the weirdest or most novel use you've seen someone make, either of PlanetScale or Vitess? What's the strangest thing someone has done with it?

Hmm.

Novel use. I'm not sure really, because it's a back— it's a database, right? So it's kind of hard to use it in weird— I'm sure someone's up to something strange. And if you are, I would love to— if you're listening and you are, I would love to hear it. But I'm sure there is some very crazy use for it. But I'm not aware of many. We skip through GitHub in extremely crazy ways. That was— stories there.

Yeah, that's for sure. I'm sure there is somewhere, you know, a video hosting company where they break their videos down into chunks and store them as discrete entries into the database. Uh, someone has built a Turing complete inside the database, uh, using MySQL. I'm sure there's all kinds of stuff, but unfortunately some of those things you never hear about. Uh, there's, there's an M— there's an XKCD comic that says that, you know, someone has solved the P=NP hard problem, but it's tied up in an eggbeater initialization routine.

Yeah, yeah, yeah.

All right. Doc, you wanna take it and get into CNCF and some of those?

Yeah, I was just— I see here that you're on the Cloud Native Computing Foundation, the CNCF Technical Oversight Committee. Back when I was an editor with Linux Journal, and I visited and talked with people working on the CNCF, it was very focused on the cellular system. and 5G, especially rolling out 5G and the side of 5G that wanted to get databases as close to where they're used as possible with low latency. Is that still there? I mean, what is the CNCF about at this point? And does that matter either? Because I remember one of the things that I heard, and it may just be a rumor, not even a rumor, it's just something. of interest was that the phone companies wanted to get into 5G for the database business. They wanted to be as close to the customer as possible with the least latency as possible. In other words, just to provide the facilities there inside of which SaaS companies could come in and then pay them for rent or whatever in their former central offices or whatever. So I'm wondering if you could shed some light on that. never had anybody write, really talk about it.

We're never gonna make the speed of light quicker. So, you know, we can't— if your, if your database is in Seattle, but your users are in Japan, I mean, there's, there's, there's some latency there that's gonna go across the ocean to serve that query. So having your data nearer the user is always beneficial. This is obviously why we have people use CDNs. And, but it gets really difficult in the database world when it comes to consistency, because to achieve cross-node consistency, databases need to talk to each other. And then if they're going across various environments, it can become really, really slow. That's why we have PlanetScale portals, which means you can put replicas of your data simply and easily anywhere in the world. So you can query those to get the latest up-to-date information in a read-only fashion nearer the user and improve the speed and latency of your application. Now, some people take that very far and make it the fundamental primitive of their database. And that has, again, its own caveats and gotchas that become very, very difficult. I think it's an interesting conversation that's going on about this. There's also just a lot of applications that don't need to do it. Building globally this way is the new microservices. It's the new hot thing. It's like, let's have this completely geo-distributed application. And then you find that like 90% of people's users are in continental US or in Europe, and it's a pointless endeavor. And you've gained— you've taken on a load of trade-offs that you just didn't need to take on. So it's very complicated. You need to think about it for your application, for your traffic patterns, or for your user base. But the other thing is that most of the major websites on the internet run out of very, very few locations and don't go for this fully globally replicated So this may or may not be involved, but I think you probably have some insights on it.

I've just spent the last month and a half being all over the place. I was in San Francisco and up in Marin and in Los Angeles and then places in North Carolina, towns in North Carolina that are busy going through a transition from an older industrial model to a new one. And I see commercial space for rent everywhere. And I have a, I have a son who works in recruiting and staffing and all that. And we have a generation that doesn't want to go into work. They want to work, but they don't want jobs inside buildings. They don't want to be in cubicles. They want to be in their own spaces. And this— what it seems to me underneath all of this, where I'm going with this, is that you take digital technology and plus the internet, plus The low latency we were just talking about. And you have an entirely new framework for work of all kinds. Now, your work with GitHub's involved with this and a bunch of other things. I'm wondering if, especially everything involves a database, of course, now at this point. So you must have some insights on where this goes over the next few years, because I think we're going to see fewer and fewer people in these big buildings and more and more residences maybe putting into those buildings, whatever else. Retailing is the same thing.

Yeah.

No more giant department stores. Those are turning into something else.

It's very interesting from a company cultural kind of point of companies going remote. It's interesting technologically as well, because it's a systems problem. We— one of the reasons we built PlanetScale branches and have them solely in the cloud is because we want people to have easy development environments that don't require like their local machine being made to look like production, which is an impossible task, especially in this SaaS-based, cloud-based future. And that means that people can do development from anywhere, and we allow branches to be provisioned wherever you are. So local to you, so you don't feel that latency. Culturally, I think it's very interesting. Remote work is very interesting. GitHub was a very early company to go fully remote. Well, I say fully remote, we had an office, but the company acted remote. It was more like a coworking space for the company. And it's an awesome future. PlanetScale is extremely remote. We don't really have a central office anymore. We're— majority of us work from home. And it's something that's really, really amazing. And it's definitely gonna be demanded by this new generation. They want to travel while they work, they want to live a different lifestyle. And it's really amazing. And it's amazing that technology can, can Hey, I wanna jump back in.

I saw something on your bio that really fascinated me, and that is you do also some work at Sequoia Capital, which sounds like maybe one of the better venture capital funds. One of the things that we talk about here on Floss Weekly sometimes is some of these open-source projects and the changes that they go through, and sometimes the weird things that they do as a part of trying to get venture capital funding. I'm curious your thoughts on how VC changes FLOSS and maybe the way that startups can do that better.

Yeah. So, yeah, and I haven't been so active recently, but I was doing scouting for Sequoia, which is like finding interesting early-stage companies to invest in, and something, you know, that I'm passionate about. Yeah, I think venture capital has a really large role to play in making open source sustainable. There's this great exchange of, you know, we can fund you, um, and VCs can fund you to, uh, not think about necessarily the business model just yet, but to build and then develop an open source— develop a business model on top of that. And it means that open source has a sustainable and healthy, uh, pattern to be developed. And so you see so many open source projects that are still going that are doing really, really well, gain open source funding, and it means it just builds, makes it much more healthy without someone having to build a profitable business from day one on top of open sources. It's, you know, open source any project, it's really, really tough to do.

Yeah, we seem to get, we seem to get some guests that come in and, you know, they have like an open core model or some, you know, strange licensing thing. And we kind of always think to ourselves, is that something that their venture capital guys made them do? And so, just, I don't know, it's just kind of this odd, sometimes an odd relationship between VC and an open-source project. Yeah, it's weird sometimes.

I'm not sure the VCs make many people do anything. I think the VCs have to invest in You have to pitch VCs on what you think will turn into a sustainable business. Especially in this current environment, it can't be an experiment. You have to know. Open Core was interesting. To me, well, I think now most companies founding are probably trying to think about how to have their open source project maintained by you being a SaaS company. That's certainly how we want to do it. Open Core has a lot of challenges. You end up building a massive support model to support companies on-prem, It's harder to grow the business. You can't. It's harder to model how your revenue is going to go. It's, but some companies have made it work fantastically. I just think you know it's not a model that's gonna persist forever. I think it's more about can you build a SaaS service on top of the open source project that you have built, and in databases that's obviously a very natural thing to do. In some some sense, for other projects, it's a much much harder thing to do. Yeah.

So I'm curious, let's say somebody's involved in a project, maybe the main developer for a project doesn't have a business model yet, but, you know, they're starting to see the writing on the wall of it not working as a hobby project anymore. But maybe it's something that's important enough they would like to keep it alive. And so kind of, is there any advice that you would give somebody in that state where they're saying, I'm trying to figure out how this could be a business, but I'm not seeing it immediately? Like, where, where should someone go to start, start thinking more about that process?

I think there's many things you could do. You can reach out to venture capitalists and say, look, I have this idea. This is what the adoption looks like. Is there a business in this? Is one, one thing to think about. The other is, if, if there isn't a business in it, or you don't have the heart to go and do a decade-long journey of building a company, you could reach out to your biggest users, right? You could reach out to companies that depend on it and say, look, you know, I'd like a job, I want to keep working on this project, you benefit from it. Is that some clear, is that a good exchange of value here in terms of doing that? At GitHub, we used to hire the maintainers of projects that we depended on. We didn't necessarily require much from them other than keep the project healthy because we need it. We did that with Rails, and Shopify does that with Rails. We did it with Git. There's so many companies that do the same thing. That's one way of doing it. Or maybe it just becomes a side project where you get a bit of sponsorship to do it. But there's that that kind of value that you can get stuck in, which is somewhat successful, that it's a demand on your time, but there's no commercial way of getting— way of making it commercial. That's a really tough spot to be in.

Yes.

Yeah.

You probably get sworn at on GitHub a lot if that happens. Hard life being a maintainer. Seriously, people maintain it.

It is sometimes.

They're saints.

Yes, absolutely. So there's one other thing in your bio that really fascinated me, and that is that you worked at Facebook doing some of their video integration stuff. I have to say, 95% of the time, video on Facebook and live video is just awesome. And 5% of the time, there are terrible sync issues and it's horrible. I'm curious if you have any tales from the trenches or maybe an open-source angle on how Facebook did, most of the time, so well with video.

So I had a short stint at Facebook, but the time I was there was absolutely fantastic. incredible engineers, absolutely incredible people that go so deep into certain subjects, which is the great thing about being an organization with like 40,000 engineers and at such massive scale. It's beneficial to have someone who sits there working on network card drivers and making them a little bit better because it can save you a massive amount of money. Tiny improvements in performance can make things much better. Facebook processes a lot of video, I believe the most in the world by volume, in terms of small videos, lots of people going live, people going constantly. And it was a— there was so much work that goes into it at Facebook, including having devices that sit within the networks of carriers, building those, making sure they work autonomously and securely. It's a really, really hard problem. And they did a lot of work on it. working with QUIC, doing a lot of optimization, running their own video stack inside the applications. It just was constant work from incredibly smart and diligent people that worked really, really hard on it. And you're right, it is very fast. There's lots of intelligence, even in deciding what CDN or data center your application— your app or you get routed to. It's just huge amounts of engineering that goes into it, and they're phenomenal at it.

Yeah, I was— one of the things I've often thought about or noticed, especially early on with Facebook, is that while Twitter gave you the fail whale over and over and over again, and other services like Flickr, which I depend on a lot doing photography, they have some, you know, a little image of a girl hauling a panda around saying, there's something wrong, it's our panda, you know. But Facebook, I think Facebook has had one kind of like big public failure, and it was temporary, but for the most part has been remarkably well-run engineering-wise. And I guess that's what you're saying in a way. And it's not just a matter of having more engineers. I think it's a matter of having better ones or better engineering. So hats off on that.

It's taken so seriously. Some of the people that I would really love to grow up to be like, the engineering leadership at Facebook, they are just outstanding. They just truly take being a massive large-scale operation really sensibly. They also built pragmatism into their culture very, very, very early on. It was like, we're PHP and MySQL to start. They took that very, very far. Opposed to some engineering cultures, Facebook is very much about, let's ship things quickly, see if it's good, deliver value to users, and we'll make it work when we make it work versus Other companies that just spend forever kind of just like perfectly architecting everything, never get it into the hands of humans, and then it doesn't work. So there is a set of very pragmatic, very intelligent folks working super hard to make that happen. And I'm glad that it's noticed. I mean, the reliability is very, very good. Things do go wrong. I mean, it's a huge operation, but it's— I think it's the pragmatic approach and doing sensible approaches to engineering took them very, very far.

Well, this has been a very fast hour, and we always close it with a few questions that begin with, is there anything we haven't asked you yet that you'd like us to have asked, or that your team has wanted you to ask?

No, I don't think so. I think it's pretty comprehensive. We went through a lot.

Yeah, yeah, we have. So, well, One we've actually dropped from the list just because it's getting to be old hat in a way, but we haven't brought it up here is, do you have anything to say about blockchain?

You know, actually, I was at a dinner with a bunch of CEOs a few months ago. And there was this really— one of the people at table, obviously crypto came up. And one of the people said, why don't we not— let's not say crypto. Let's not say blockchain. Let's just say like, extremely consistent, very slow database and talk about it that way. And I thought it was a really, really interesting way of framing it. Because that's, it's, that's a technology.

Right?

Like, that's a— there's a— and then what use cases do we have for that technology? We struggled. We were like— it's like when you just frame it as like, here's the specs, we really struggled to find a usage for that because like most state applications want faster, a little less consistent, right? Um, and so I think it's very— crypto to me is fascinating. It, it catches this loop in my brain that I have, which is I know I'm getting old now. I know there's a whole new generation of developers that are building and there's energy and there's something inside them. And you don't want to become irrelevant. You don't want your company to become irrelevant. You want to build things for the next generation. So I still can't abandon it. I can't say no because there's clearly something there. There's also a lot of scams and pump and dump and whatever going on. So it's very hard to extract what's real and what's good there. But I still keep trying because I don't want to like dismiss a new technology that could, could be transformational. So it's— I'm between 2 worlds right now.

So our last 2 questions are, what are your favorite text editor and scripting language?

So I, when I script, I would prefer like either Ruby or Bash. And text editor, I use VS Code because it's just become so good now. In fact, I use Codespaces as well, which I really love. And they pair very nicely with PlanetScale. Branches, but I just love writing a, you know, a nice bash script now and then can take you very far. I also have to program in Python a fair bit for like Home Assistant automation and stuff like that.

Well, fantastic. Well, it has been great having you on the show, Sam. This is a—

thank you for having me.

It's a quick fave. You've been on, on, on, on TWiT twice so far, and I'm sure you'll be back because this is not a— this is not a concluded science or subject. It's a It's interesting to me how far you've gone since the last time you were on, which was less than a year ago. So thanks so much for being here. It's been great.

Thank you for having me. I love the Twit Network. It's awesome to be here.

So, Jonathan, how was that for you, man?

I love when somebody that's working both on an open-source project and a company, when they're excited about what they do. And when they see the possibilities of the technology that they're working on, right? And so, this idea of, it's so easy, especially when you're doing something like a schema change, it's so easy to be the intern that trips over the figurative power cable and brings everything down. And then he said, well, let's just do something about that. Let's fix that. Let's make it so that, you know, there isn't another intern that trips over that particular power cable.

Right.

I love that, and the fact that he's so excited about it. Somebody talked about the idea of ratcheting when it comes to technology. It's like you should learn from the things that have gone wrong in the past and actually fix them so that they're not going to continue to haunt you. So just that idea about being able to rewind and not do a backup restore, but a schema restore so that you instantly fix the thing that you broke. I just think that's great. And then, you know, it sounds like they're working really hard to have the right kind of relationship with their community and with Vitess. And just the fact that, you know, the majority of this cool stuff that we talked about, if you really want to, you can host it yourself, and you don't have to pay them anything to do it. But maybe you want to because it's a hard problem to solve doing it on yourself. I just think it sounds like they've got the right mindset to be involved with an open-source project and doing open-source resource as a service, as it were.

Yeah, I love, and especially going back and listening and watching his last appearance on TWiT, it was really interesting not only to see the delta between the two, but to sense how deeply involved in the world that he and PlanetScale are. And a huge part of it is just being deeply involved in how open source gets made and what's the best way to deploy this? What's the best way to use it? How does SaaS work here? How do you work with the cloud? How do you spread that out as close to the customer as possible? How do you help them recover, rewind when things go wrong, as you were saying? And knowing what's going on in the world, I think that matters. I mean, I know from my early days with Linux, I mean, especially some of the kernel developers, it's like, We're never in user space, we're only in kernel space. And there's this huge contextual gap between the people who made the stuff that everybody runs and the people using the stuff on really particular applications, right? And it's a huge difference. And I actually think it's one reason why Linux on the laptop or the desktop took so long to happen. And that's not a knock, it's just kind of a, An assumption because I think if something like Linux, it really is under everything. But part of what's going on in the world today is everything runs on databases. Databases are everything. They really are. And that's, you know, it seems to me that there's probably nobody with his or her fingers in more pies where things get done than these guys. So hats off. Yeah.

The other angle to this that is a juxtaposition of being incredibly novel and boring is that it's all based on MySQL.

Mm-hmm.

So he mentioned the way Facebook will take something like PHP and just push it and push it and push it to where it's based on the same idea, but it's just been pushed all the way out to the limit. If you're familiar with some of the crazy things Facebook has done to make PHP work better, they're doing the same thing with with MySQL, which was never designed to be used globally. It's one of the old databases. It's the simplest one. Your LAMP stack that everybody kind of turns their nose up now.

Yeah.

Linux, Apache, MySQL, PHP. Well, here's one of those, one of the old players, one of the boring things that none of the cool kids use anymore, being pushed out to the extremes by these guys. I find that fascinating. To take the boring and kind of make it revolutionary again. It's fun.

And how it's maybe keeping, as you were pointing out during the show, keeping Oracle straight, and the Oracles of the world as well. It's not just that one company, it's everybody using this stuff.

Yeah. Well, I mean, the threat of forking, the threat of if you mess this up badly enough, we're just gonna take our toys and go play in the other room, with our fork, that does— that's a very powerful motivator for Oracle particularly to straighten up and treat them right.

Yeah, yeah. So what do you have to plug before I get into our plug?

Okay, so the 2 things I've got. First off, hackaday.com, and of course I run the security column over there every Friday morning. Cover a lot of really interesting stuff. Make sure to keep an eye out there. I've got We've got some other things in the pipeline. I may or may not have my Starlink mounted inside of a tire on the top of my minivan. We might see some things about that over on Hackaday here pretty soon. And then the other thing, of course, is on Club Twit, make sure and check out the Untitled Linux Show. We go live every Saturday afternoon. And then there's the Club Twit exclusive feed that everyone should be subscribed to.

Yes, indeed. I want to plug that one because joining Club Twit's a great way to support our network. A member gets access to ad-free versions of all the shows on Twit, as well as other benefits. There's a bonus Twit Plus feed that includes footage and discussions that don't make the final show edit, and bonus shows like the Giz Fizz, Ask Me Anything, fireside chats with some of your favorite Twit guests and co-hosts. There's a There's a community aspect to it. We have a fun Discord server that's available only for Club Twit members. You want to know when Jonathan's Untitled Linux Show goes live? There it is. Join the Discord, catch it there. Have questions about a career in tech or IT? Join that Discord. I'm sure you have plenty of advice and valuable information from passionate Club Twit members. So sign up, join Club Twit for just $7 a month. Head over to clubtwit.com. Let's see, twit.tv/clubtwit. That's twit.tv/clubtwit. Join today, and we thank you for your support, and we thank you for being on the show. We'll have another good one next week, and until then, I'm Doc Searls. This is Floss Weekly. We'll see you then.

The world is changing rapidly, so rapidly in fact that it's hard to keep up. That's why Micah Sargent and I, Jason Howell, talk with the people People making and breaking the tech news on Tech News Weekly every Thursday. They know these stories better than anyone, so why not get them to talk about it in their own words? Subscribe to Tech News Weekly and you won't miss a beat every Thursday at twit.tv.