Max Levchin recounts a near-catastrophic security incident at PayPal roughly one year before its IPO. PayPal stored about 100 million rows of sensitive financial credentials encrypted under a master key; Levchin implemented a Shamir-style shared key ("sharded key") to replace the single master key. After re-encrypting the database, the site failed to decrypt correctly for about five and a half hours because of a subtle portability bug: getpass() on Solaris accepted up to 256 characters whereas on Linux it truncated input to 8 characters, so the key shards were truncated. Levchin discovered the discrepancy by checking the man page for getpass and fixed the issue, restoring service in about 10 minutes. He reflects on engineering attention to detail, crisis behavior, and the importance of code review and pairing.
Questions this video answers
Roughly 100 million rows.
About five and a half hours.
On Solaris getpass() could capture up to 256 characters; on Linux it was cut off after 8 characters.
About 10 minutes.
Chapters
The sacredest thing at PayPal, certainly during my time there was the database of credit cards and bank account credentials. That was at the time, roughly 100 million rows of what now is known as PII. The acronym wasn't made there yet, but you know, people's financial data. Very, very, very scary to lose it. We encrypted everything. I could talk to you for an hour about how secure we were. It was my obsession, my background. Before all the current dark times began, I was in computer security. So it was all very well encrypted. The one big vulnerability, we had a master key that would decrypt database. And that just bugged the shit out of me. So one day I read this paper, strongly recommended called Sharing secrets among Friends. It's a perfect way to share secrets. And so I implemented sharing your secret sharing over undoubtedly two or three days of no sleep, and very proudly got everybody together to replace the one master key into what's called a sharded key, where each one of the people has a shard. That was really great. We took down the site, we decrypted with a single master key the big PayPal database. We re encrypted it with the sharded key, we brought the site back up and people typed in their shards and it didn't decrypt correctly for the next five and a half hours. We were basically beating our heads against the wall because we were so secure. There was not a backup. So Peter recalled me at like 5:00am like, hey, why is the site still down? And now there's a CNN running a headline saying PayPal went down last night. Nobody knows when it's going to come back. Like total pandemonium. But the long and short of it is this will be only funny to some of you. I coded this thing on Solaris, but I ran it on Linux. Get pass on Solaris could Capture up to 256 characters on Linux cuts it off after 8 characters. So as we try to reassemble it, it would take in the 24 character long passwords and reassemble them only the first 8 characters at a time. It was actually one of the greatest puzzles of my life. I went to my cubicle, which I remember like it was yesterday, and just stared into the black CRT for 30 minutes trying to figure out what could have possibly gone wrong. And the very next thing I did is typed in man, get pass. Which was the only part of this whole thing that I didn't implement myself on both the production machine and my desktop, where I coded It. And I was like, oh, 8 to 56. Fuck me. And 10 minutes later, we're back up. That was.
So this was catastrophic enough, I believe you didn't tell Peter this, the actual story, for over a decade.
So first of all, the first five years, I told no one. And then after that, like, this is too good of a educational story to keep it entirely to myself. So on occasion at like YC startup school or small groups, I would be like, all right, let me tell you the one time where I really fucked it up. And over time, I became less and less embarrassed and more and more interested in telling people the story here, obviously, details always matter. If you're going to be the CTO who writes code, you better be at least as good at details as the people you trust to write code for you. The embarrassing wasn't that I had a bug. It's that it took me so damn long to figure out what it was because I was the one who coded the thing. There was not enough people in the room who could proofread it for me or pair program with me. But two is, how do you behave through a crisis? One of our engineers who was in the room literally screamed, we're going to have to start brute forcing this key. It'll only take a few billion years, but no problems. And so you get to watch how people behave in a moment of extreme pressure. And my proudest part of it was that I had never faced a now, this is a company destroying moment. Like, up until that point, my company's failed in a slow motion way. Like, we're out of money or nobody wants my product. This was like we were, you know, a year from IPO. We had 100 million credentials. This was like a real thing. We won ebay. Everything was going our way, and I was just about to pull the pin out of the hand grenade and sort of gently put it on our collective desks. And the fact that I sort of like the only right thing to do here is to actually think like an engineer, put out the fear of total failure and just try to reason what could have possibly gone wrong and maybe there's a fix.
