Jocelyn Goldfein: Lessons Learned from Breaking Things

This is when I was in boot camp. It was week five and so I was brand new to the company. Nobody knew me from Adam. And I noticed that my first few weeks in boot camp, I was assigned a bunch of bugs to fix. And it turned out that like three out of the five were obsolete. Someone had already fixed them, or the code had been refactored and just wasn't an issue anymore. And so I realized that there were actually a lot of bugs in the bug database that were kind of, that were cruft, basically. And if you like. Bugs don't sound like a very sexy thing, but if you are determined to ship high quality software, one of the things you need is information about what is your state? Like, what is the state of my software? And a bug database can be a faithful representation of the state of the world if you are really scrupulous about your bug hygiene and about closing out bugs, about triaging your bug, it's about looking at every bug and about closing out the ones that aren't relevant, including ones that exist but you're never going to do anything about because they don't matter. At VMware, I had built a name for myself by being this monstrous triager of bugs. I triaged probably 1,000 bugs in my first month at VMware. At Facebook, I was walking into a much more mature environment that already had bugs, a lot of bugs open. And I also had reflection in hindsight that it's just not what I did at VMware didn't really work for me, but also wasn't incredibly scalable. So I wanted to try something new. I wanted to try automating bug triage in a way and writing a script that would essentially, if a bug had been untouched for three months. At Facebook, things move so fast. If something's been untouched for three months, it's a pretty good sign that it's irrelevant. And so after three months, it would sort of just post an update to the bug saying, hey, is this thing still relevant or can you close it out? And that would trigger an email to everybody cc'd on the bug. And then if another three months went by and nobody responded to that or updated the issue, then it would just auto close it. So I was building this, I called it the Task Reaper. And I was writing this script and I was testing it and I wasn't careful enough in my testing and I accidentally just was moving some code around, it was copy paste error and I moved something out of an if block and you're probably guessing where this is Going, I accidentally ran the test on live data and it pinged 14,000 bugs. Because the first time you run this, there's a lot of untouched bugs. It pinged 14,000 bugs, and that means it generated email to Everybody on the CC list of all 14,000 bugs, which meant I basically launched a denial of service attack on our email infrastructure. So this brought the bug system to its knees, but it also brought email to its knees for the whole company, which is like a pretty big deal. I mean, it doesn't take the site down for users, but it means, like, our sales team can't interact with customers, engineers can't do code reviews. Email is kind of the lifeblood of the company, even more so in those days. We use Facebook itself more for communication now. And so I was just kind of blankly terrified, like, what have I done? And I really expected to be tarred and feathered for this. And there was definitely a vocal company response to it. What just happened to me? Why do I have 200 emails from the task reaper? But what was really visible to me, two things were really striking to me. One was the exchange team and the bug tools team. The people I expected to be most mad at me actually just sort of rolled up their sleeves, waded in, and started fixing the problem. They spun up another process to process the emails faster. They threw extra capacity onto the server. They just got in the trenches, and they had my back, and I was a stranger. They didn't know me. They didn't have no right to be messing with their systems. They just had it. And then on the flip side, the communication from the company, there was no one saying, how dare you, You're a noob. Why didn't you ask permission? Nobody said that. People were saying, what was the point of this? And they were saying, could you maybe send a digest email instead of an email for each one? And they would be like. But no one acted like I didn't have the right to try. They clearly didn't like the result that their email was down, but they didn't act like I shouldn't try. And. And it was like a light bulb went on in my head. I realized, oh, my God, this is what lets this company still innovate, even when it's already. At that time, it had just passed half a billion monthly users. And if any company, it was just successful, I'm like, why does Facebook still have appetite to mess with this thing that is so successful? Why do we launch complete redesigns of the homepage when the homepage is already the most trafficked piece of property on the web. And I sort of got it. In that moment, it was because we are willing to take risks. We are willing to face up to the consequences of failure if it was in the spirit of trying for something, of trying to innovate. So that was just, like, an amazing experience for me. And I think it's not that you can come to Facebook and be incompetent or do things wrong all the time. I think there's a lot of feedback and actions if that's happening. But we view it as every employee's right to. To try things, to take risks. And we expect you to deal with the consequences of your failures, but we also rally and help you and have your back when you fail, and we expect you to rally and have our backs when we fail. So it was kind of a magical introduction to the company.