Yash Nelapati recounts scaling Pinterest through its early years, covering database migrations, architecture failures with Cassandra and Membase, and the painful live-database jump that nearly broke the product. He describes the team's lack of tests, reliance on third-party search, and lessons learned about keeping systems simple while chasing growth.
Chapters
This is where we were. And between this era, it's like, God, I don't even know. I have all these white hairs from it. That's all I can tell. But at this point, we just pretty much put everything we could with the least effort, because there were only 2 engineers and 2 more engineers starting. So every day I would go on Hacker News and like, hey, what could I do with this problem? Like, hey, you should use my database that I wrote over the weekend. It's amazing. It just auto-splits the data. You don't need to do anything. That's awesome. And that's the pitch. And I'm like, that's beautiful. I'm going to use it. I put Cassandra because why not? There's a company behind it. And somebody said, you should put Membase. Zynga scales— what's that game? FarmVille— on Membase. I'm like, that's awesome. Let's just put that in. Memcache, Redis, everything. Elasticsearch. And Mongo. 3 engineers at this point. And when you put all these technologies in flight and hope everything works, Nothing works. So that's our first lesson that we learned. Like, everything that could fail will fail. So the simplest architecture you could have, the simplest solution you can have, and like keep the system running. So at this point we like proposed like a complete rearchitecture and like feature freeze, because Evan always, every day he wants to do something like, hey, how about we like build wall messages because that's cool on Facebook. Maybe we should bring it. I'm like, sure. And we keep on building and piling up. And one day I was like, we're done. We're just gonna bug fix for the next 6 months, and we need to re-architect the entire crap that we built that is blowing up in our face every day. Super hard lesson that we learned, again, was you just can't migrate a live database, because people are constantly pushing new data, and we're building brand new machines and architectures and, like, code. We're writing code every day and, like, who's gonna test? No time to test. Like, at this point we have zero tests. And, like, somebody mentioned it in, like, a Stanford class and, like, every student freaked out. Like, what? You didn't have a single test? I'm like, yep. So that was another lesson is that we just didn't have the time to do any of this. It's not like we are anti-testing or, like, it's just like something like no time. Just gotta like, so we just gotta build the product. We understand what works, what doesn't work. We pick the what works and make it better every day. You just like constantly like make it, keep making it better. So with very few engineers, it gets very challenging and like no testing and like pushing it to live is a very scary thing. So fast forward another year like of brand new architecture and moving over the data. This is how we simplified it. Like, basically all the crap is gone and everything that worked, we just retained it back in our stack. It was just Memcache, Redis, third-party Solr, because we didn't know how to do search. So we just like offloaded it to another company and like we just gave them the data, they like provided the index. So pretty much MySQL, Redis, and Memcache, that's how, that's all I know how to run efficiently and like we just retained it in the stack. And this is how the team looked. We moved into a brand new office, moved out of the apartment finally. So yeah. One more lesson is, like, you can never keep 2 different data sources in sync. There's always, like, every time I flipped the switch and, like, said, like, hey, we're just gonna, like, read from the new architecture, the Twitter would just blow off. Like, I lost all my boards. I lost all my pins. Can't you get your shit together? And we're like, okay, sorry, we revert it back. And they're like, oh, I'm happy, I can see all my shit. I'm like, okay. Hey, what's going on here? Why is the data not transferred, Marty? And like, hey, let me check. Like, yeah, somehow that one user, we missed it. I'm like, yeah, can we just do the copy over? And like, yeah, copy over. Again, I flip the switch, and again, like a bunch of news is like, I lost my shit. I'm like, oh, whoops, sorry. So it was just like this. My hand was just doing this for for like the next 5 months. And like every time I do the flip, people just like freak out. And I'm like, guys, that thing is gonna blow up. We better make the jump forward and we don't look back. So finally one day I was like, whatever happened happened. Just tell the community like, hey, sorry, we lost a little bit of data, but we'll try to regain it once everything stabilizes. And then that's what we call the Indiana Jones jump because once we, We flipped. There was no going back. That database was just like done. That hardware wouldn't even boot up. And we just like had a backup. So for the next 2 months, it was just like bring up a new master and just like see like whatever pieces of data we could recover and tell sorry. Another 6 months down the line, there was another startup we were sharing this office with who were behind this wall. It's called AeroFS. So we were just like at this point growing so fast that we had to kick them out, take down the wall, take over the entire office. So this is what like a hackathon looks like. Pretty much everybody just doing their own thing. Super unorganized, but trying to keep the site up. But yeah, fun days. Another year goes by. More growth. Everything is just retained the same, just more numbers, like 230 servers of memcache, 10 Redis, 100 Redis, 70 MySQL. At this point, the architecture is solid. We just, we just throw more money at the problem. We're not gonna re-architect. We're not gonna use any engineering time. We're just gonna, like, keep the site running, add more features if we can. At this point, we were growing so fast. And this is the point where, like, All the new engineers were pretty much doing growth hacking, which I'll be talking about more in a little bit. This is all ran by Pinterest and everything else is pretty much third party because we just didn't have the time to architect or maintain these solutions. So a bit of caching for frontend and HBase and Kafka, so search and stuff. This is how the team looked like. At this point, we couldn't find a big enough space in Palo Alto because Palantir was eating all of Palo Alto. So we had to move to the city and like this is the size of the team at that time. So I'm gonna fast forward another 3 years and this is how it looks like today. Whoa. And that's only 50 or 60%. There's like another office in New York, São Paulo, and a bunch of stuff. But there's a lot more people here and there and yeah, so pretty cool. Pretty soon we'll need a drone to shoot the team, probably. But yeah, great lesson we learned. I mean, pretty much what I explained, there's one word I can describe for it, which is growth. Thank you.
