Team Ownership - Who Owns Production - Chris Slowe, Reddit-UXDX Europe 2019

Hi everybody. Thanks for joining me. Top of the morning on the first day. I think that makes me the opening band. So as the opening band, I hope you find me entertaining but ultimately forgettable compared to the main event. Talking about production ownership, which you know, you can really think of as production ownership in some ways a tragedy, but that's kind of a depressing way to start the day. So I'm going to instead go with production ownership, a comedy as a as how we've kind of done things. This is a topic that's really near and dear to my heart. Been at Reddit for most of its history. Kind of started three months in and so quick about me Founding engineer at Reddit, already introduced, left and came back, had a chance to undo and then redo a bunch of mistakes, which is a delightful experience. And when I say I have a lot of opinions, it means I've seen a lot of really broken things, most of which I broke. And you know, breaking things builds character. And building character is expensive. So my goal for today is to say we're all in this together. This is true as much as a developer group, as in this talk, that we're all in here at 9:30 in the morning on a Monday. So first, really brief history, start off with a what is Reddit opener? I'm kind of curious. Who here uses Reddit? Very cool. Who here considers himself a Reddit user? It's always a different group. I don't know what that is. It's like a bunch of lurkers. But so you know, Reddit, Reddit is a network of communities and it's basically a platform where people can discuss their hobbies and interests with other passionate users. And really the cornerstone of Reddit is about conversation. So, you know, the thing about Reddit is we're kind of a big deal. We're kind of a big site. I usually refer to us as the biggest little site no one's ever heard of, or at least pretend not to. We've got, you know, 330 million monthly active users. We do millions of votes per day. It's kind of a big deal. But let's talk a little bit about how we got there. So how do we get started, you know, really far back in the day? The thing about Reddit that took us the entire history to realize is that Reddit is an example of a product that kind of out of the box, just had this wonderful product market fit. And so we opened up with having, you know, there was always this kind of three month doubling cycle during the early years, which really meant that it kind of focused the mind on what we were doing at any given time, which is to survive the next doubling period. We got acquired really early and we were basically a holding subsidiary for most of the first bunch of years. And this left sort of interesting mark on the company. So here is our employee growth as a function of time. I actually started the access in 2008, where we started actually keeping records. This line at 4 goes all the way back over to about here from 2005. So you can see it's a pretty gentle ramp up. If you recast this in terms of reach per developer, it looks like this. This is the page views per month per developer. You can see that basically everyone on this team is taking on a fairly significant load when it comes to what they're doing. The side effect of this is that everyone has to know what they're doing. If you don't know what you're doing, you're kind of screwed. And so early years, it's like managers, what are managers? You just kind of keep the site going. That's the kind of point. We actually didn't have a PM until about 2014, 2015, just because it was mostly developer focused. And this is also, I think this is at a time when there's this Mythos of the 10x engineer and really fundamentally 10x engineers and startups are very compatible. You have to kind of know what you're doing, be able to do it, or you don't really get a chance to make any headway. So for the more recent stuff, let's talk a little bit about. Org charts, which is kind of the important bit when you get a bigger team. We're now up to about 550 people in the company, about 200 engineers. Now. The hard part about this talk is being retrospective. To start with, I put myself in my shoes from 2008 and realized that if I'd seen this slide 2008, I'd feel a bit like this. It's like you kind of have to take it with a grain of salt. So the utility is like the anatomy of a startup. So this is production. In a startup, production is a block and who owns production is the team. You can actually break that down into individual developers. Everyone knows their turf. So you end up with this situation where if you go from back end to front end, and this is kind of like the products that run across it, well, it's only kind of one product and you get a bunch of devs and they're mostly overlapping and you'll notice it's mostly pairwise, so you have an ability to at any given time work with somebody else. But usually you're kind of working alone, kind of drilling on your own area. Everyone knows what they're doing, all the gaps are filled, or at least the gaps you care about are filled. And everyone in this picture probably has a pretty complete idea of what production looks like and what everyone else is doing, which is kind of an important part. And last thing is, for a startup, there really only needs to be one product. I mean, maybe you at some point during the startup, decide to pivot into a different product, but it's still kind of one product. As you grow, the picture starts to get complicated, right, because you start getting redundancies like this, you start filling in the gaps, you start adding more people. You also have people who are less experienced, little smaller circles, people adjust in their position. And you end up in this position where you've got this kind of overlapping web. And this is about the point at which you have to start thinking about teams because, you know, you can't just have a monolithic team running at this point. And so usually the canonical first thing you have is the split. So kind of this front and back end kind of shift that happens in place and in the middle, we can put some scary air quotes around the API in the middle. But now when you actually have teams, you really need to start thinking about an API, like actual way that your front end and backend communicate. That's official. It's like a contract. You have a dedicated front end team. It's mostly working in mobile clients or web or chooser technology, backend teams generally working in whatever your stack is built in. In our case it was Python. And this also to some extent is a function of just you need to hire a manager. And it's easier to hire a manager to run an entire section of the stack right, rather than to basically choose them to run a product as time goes on. The next thing that usually happens is you can start splitting up your products, you build new products, you start building out teams. Each one of these verticals here has its own bit of ownership. And oftentimes you'll see the same thing on the back end, but it usually happens at the front end first. Where you have the canonical split that used to happen was you'd have the mobile team that would be in charge of all the mobile stuff. And now of course, the mobile team is really your main group. So things get more complicated, of course, you start getting more boxes, you Know, things start to split up and so let's talk about where the problems start to come into play because this is actually, I think the core of production ownership is what problems are you trying to solve at any given time? Well, okay, so let's take this simple setup. This by the way, is usually called the layer cake and candles. It's kind of this mix of vertical and horizontal teams. Well, first of all, you kind of have this API in the middle and it's really kind of unclear who owns it, right? Because you've got overlapping considerations of the client teams use it, but the back end teams kind of make it. So they all kind of have an opinion on it. So that's where the first kind of like interface that requires a little bit of support kind of comes into play. You also get this kind of contention between product teams, right? You have a, you know, what did you do? Kind of kind of argument that happens between two product teams that have overlapping code bases. The one that I think happens more often is the nut. It like there's some sort of bug that happens, right? But it's an interface between two teams. It's like those guys did it, we don't have time to do it. It's not on our roadmap right now. So mind's the gap. And then last of all there is the full stack drilling that happens of like no, really, I know how to write back in code that comes into play. And of course front end teams and client teams want to have control over their destiny and therefore want to get involved in the back end stack. So the point that I want to get across first off is this is actually all kind of the same problem. This kind of drilling that happens with full stack development is really about this kind of horizontal contention that's similar to two product teams that are fighting over what part of the product they run. And so to some extent all of these problems about who owns production is really a question of how do we get teams to interface efficiently. Oh, just that a bunch of people. How do you get them to actually coordinate? Well, here's some ideas and some solutions that we've seen work. Let's start off with bugs. Most of what happens when you have arguments about ownership production is it's really where do you land the bugs? There are always bugs. In some cases the bugs are really clear cut, like P0 backend bugs probably mean you're down or you're in some degraded state. And it's very clear that, that in that situation there needs to be somebody who's going to be paged and held accountable. What's a little less clear is that there's an entire class of front end bugs that fall into the same category. If your checkout flow is broken because somebody pushed a bug that now your JavaScript doesn't make the button work, that's probably a P0. That probably means somebody should get paged. It's still a situation where you actually need escalation. So the important part here is to start off with establishing SLAs around bugs. If you all agree on the level of what a bug is, you can all agree that actually somebody should get paged. What we started doing is we actually centralize all of our bug triage. So this is across all of the product teams and across all the backend teams. We make sure that everyone's part of a rotation to triage bugs. Which means that every group starts to get familiar with other teams bugs and kind of knows where they should land. Or at least you can start to draw some lines that are more clear. Another thing you can do is actually start adding teams that act as explicit glue in here. So you, if you think about some of the things they said that were wrong, you've got these teams that don't necessarily have to coordinate on the client side, whereas in the back end it's much more of like a single block, at least early on for many of the early years. So you can just add a team in here. So this is a kind of a. You can think of this as a support team that starts to build out what is the underlying infrastructure and the surface upon which people build products. We have this team actually called Web Infrastructure in, you know, kind of. We call our backend team the infrastructure team, as most teams do. And the idea here is that even on a front end stack, there's kind of a back of the front end. There's like, there's the set of common libraries and modules that everyone uses. And you need a team that's able to kind of go in and start supporting that and make sure that you can build products on top of it efficiently. Another thing we can do is actually look at the individual composition of a team. So let's kind of do our zoom and enhance magic. So here's what do one of these stacks look like? Well, generally what you have is you have a team which consists of a single director who's kind of running a bunch of engineering pods almost. And these engineering prods are naturally kind of these interstack groups, right? So you have an EM who's probably running the team. You have a web engineer, you have an iOS engineer, an Android engineer and kind of back end. And of course you know, you probably have a product manager and a designer and kind of the whole set of things. But I'm kind of keeping it right now to kind of engineering just to be, just to be simple. So the hard part with this kind of setup is now what you've got is you have people distributed across teams of the same discipline who can't really naturally coordinate. So there's no way to necessarily keep synced. And so the trick we've employed is one that we actually got from Spotify, which is this notion of guilds. So a guild is kind of, you can think of it as like effectively a cross cutting V team, like a virtualized team. And they're not so much teams as more notional ways to coordinate. And so in this case we can actually say there's like a backend guild which is kind of this kind of semi, voluntary, semi, just kind of like show up and talk to your fellow backend people and coordinate. And we have a front end guild and a mobile guild. And this means that everybody gets to kind of complain and complain about the stuff that bothers them the most about other teams. You know, for example, backend guild universally is like, oh my God, you won't believe the failure we just saw in production last week. Front end guild. The one I hear the most is two more JavaScript frameworks came up during the time of this meeting. Actually, wait, no, three. And you know, mobile guild. I have to say I'm kind of a backend chauvinist. I'll actually warn you, all those backend guys are blaming us again for retries is the one that I think I've heard the most. But the question is, does this now work for production? So we solved some of the product contention. How does this work for the actual drilling into the backend? Well, you know, first off, bugs are everyone's problem. That's a pretty easy one. If you can't find an owner of a bug, it's literally whichever team seems to be the least loaded or whoever basically gets selected randomly from a group. When you talk about clients and servers, here's where things get complicated. Right, because we can't actually employ guilds. Guilds are intended to be single discipline. So you're starting mixing kind of back end and front end. There's different tooling, there's different languages, different. There's some contention about exactly what normal behaviors are. And so it's kind of hard to do that. And so the thing that we thought about was, okay, so what about this language gap? Can we do something about the language gap? In our case, you've got generally Python on the back end, JavaScript on the front end. We have Objective C in the clients. So what about cross stack languages? So this is generally where Node comes into play. Node's becoming much more common, especially in backend stacks, as time goes on. So here's a short story of our foray into using Node. We just recently did a giant redesign of Reddit. This is a small change to the way the site works. Of course it launched perfectly, everything was totally fine, nothing ever went wrong. But here, hypothetically, something that could have gone wrong. The idea was, as we're doing a front end rebuild, let's see if we can actually increase developer velocity of the client engineers. In this case, you have this problem of you're basically building out a new API on top of an existing API. And so, well, what about in our stack? Let's just add a shim. So what we did is we built a shim sitting on top of our API. And the idea here is not so much that we want to develop an entirely new API, but we want to add flexibility to the client engineers to actually develop their own API as they need it, and to kind of figure out with our current API what the right interplay is. And then the intention is long term, we rebuild our API around this new paradigm. So a shim is born. We decided to do it in Node because again, client engineers already working in JavaScript who might as well give them a language they're used to dealing with. So, you know, developer velocity. In this case, we have a monolith kind of sitting at our core and we also have other services that are kind of running around. So the idea is that when a request comes in. Oh yeah. So as we were actually launching this, we want to make sure that as we're rolling this out, it's kind of done in a very deliberate fashion. So we were doing a whole bunch of beta testing with our new stack. We were making sure that everything could actually load up. The tricky part, of course, to building a new thing at scale means that any small problem is going to be magnified pretty rapidly. So we decided to do a slow ramp up on this one and we launched at 2%. Here's what our request graph looked like on our backend. That is not 2% growth, that's about 100% growth. So this is one of those what on Earth is happening. This is in the axis, if you can't see it, is in thousands of operations per second. So 2% causes about a doubling in our backend, which seems to be a problem. Well, so the way that we set this up was we kind of put this on top of all our backend services and, you know, a request comes in and it goes to this shim layer. And the first thing the shim says is, okay, who has to handle this? And it has a whole bunch of choices. You can think of it as the newer version of this that we have in place is we started building out GraphQL. And the underlying idea of GraphQL is I have a request. I don't really care about the response, but I have to figure out who should actually handle it downstream. In this case, we had a whole bunch of decision points. It's like we could actually ask the monolith. We could ask this other service called Gateway. We could start to kind of fiddle with it. And Gateway actually had the same kind of process going on. And there's this extra little arrow. So what's happening here? We have one request coming in. It turns into multiple requests going to our backend that goes to our monolith and basically dosses ourself. So we were seeing cases where one request could turn into as many as 15 or 20 requests going to our backend. This is by definition bad, especially when we're talking about scaling up. So this is where we got this extra factor of 25 roughly in the traffic load. So much cursing ensues at this point, we discover this is a problem. So, hey, so what do we actually do in this situation? We've got a team that is. Is not really used to thinking in these ways. Well, fortunately, we actually had a whole bunch of systems in place. We hadn't actually just universally deployed it. One of my favorite quotes is that the future is here. It's just not universally distributed. In this case, we actually had tooling in place. We just never actually pushed it up into the node stack. So there's this great tool called zipkin. Zipkin does tracing. Tracing means for a given request, you can actually look through the entire stack and figure out what sub requests come from it, how it gets routed across all of your services. And of course, part of this is adding communication. So we actually had a chance to collaborate across all of our teams. Seems like a very basic idea of making sure your teams actually talk to one another. So in this case, the team had a goal and a way to measure it. And we could actually prioritize whether or not this is the right thing to do. We could actually, you know, this communication meant that we could surface to front end teams like, hey, look, you're actually slamming the backend and actually have them trace it down with some tooling that was very straightforward to use. So a couple deploys took away most of this. This is one of those when you actually figure out what's going on, it goes away very rapidly. So we spent a whole bunch of iteration time on that. So what did we learn here is where I'm trying to go. Okay, so what I'm saying right now is that as you're kind of drilling, you can think of this as you kind of squish the backend teams. So what we've done is we've kind of increased the total surface area where you have collaboration between front end and back end engineers. And you also have, by virtue of the fact that we now start adding into production technologies that client engineers are used to developing it naturally means that back end engineers aren't as familiar with developing it. And so you kind of jam into the stack this weird technology that no one really understands. So the back end teams in this way become sort of a giant new single point of failure. This generally also kills motivation on both sides. You end up in a situation where you're kind of constantly firefighting and now on top of it all, you're firefighting against technologies you don't understand as well. There's also lots of, not lots of stuff going into production right now. You have not just client libraries shipping up to the client, you also have client libraries you're shipping to production servers. And you have to figure out how to scale that stuff up with tooling that might not be as to close. And don't get me started on JavaScript bundling, which I think is the tool of the devil right now, or at least the tools of the devil, as there is a new one every two weeks. So the important part here is to really, for the back end teams and for any team in a situation where you're constantly firefighting, stop the bleeding first is the important part, and then scale up the teams. Quick plug for this. There's a great book by Will Larson called An Elegant Puzzle, which is really a good description of how to scale up infrastructure teams at scale. And he has his basic tenets of, you know, when you have a problem where you're hitting scale, sometimes the correct solution is in fact too higher. If you can't get ahead of the curve, you're probably stuck and hiring is the only possible solution. All right, so what we have done is the next iteration of this whole process, besides reassessing our desire to use Node in the deep stack is actually at a layer. So when you have an unclear owner in this situation, add some owners. In this case we were adding an SRE layer. So the intention here is that we have a set of SREs whose entire job is to be embedded in the actual product teams and to act as the interface. This also adds a nice ability of us forming a guild effectively across the entire company where they're able to coordinate, they're able to do things like share an entire on call rotation and actually figure out how to best use the tooling that they have and build out new tooling at the same time. This is currently in kind of a slow roll and so I'll say I'll let you know how that breaks probably in a talk at this forum next year. But let's just go through kind of overall. Overall takeaways I want you to have from this talk is a set of ones apparently so do QA on your slides an important thing. But in this case, let's say that communication is definitely key and specialization is no excuse to stop coordinating. Part of the reason why we were actually able to deal with the mess that we had made with our shim was because we just had teams coordinate and actually talk about what was going on and drill into it. I will also say that all of the pictures I drew of production ownership were these kind of weird horizontal vertical flips that happen. That's normal and healthy. In fact, there should be some tension in the stack. I've heard of companies who will decide to periodically decide to switch from horizontal to vertical. And the nice thing about doing that is you get a chance to kind of have this implicit to explicit cross pollination happening. So first of all, it's like you have your production teams and your product stacks and you switch to oh yeah, you know what everybody's doing. JavaScript flips into a team and then you work on that for a while. So it gives you a chance to reconsolidate. The last thing here really is that vertical teams need glue. No matter what you're doing, you really can't have completely separated units across this chart. There has to be overlap. In the early startup days you're talking about overlap between people and actual developer life cycles. In the later days you actually need to have either explicit people acting as that glue or you need to have ways to collaborate or you need a team that kind of catches them all together. So with that, I'd like to say thank you.