Scaling Tech Organizations by Chris Slowe | CTO at Reddit

Thank you for joining. I'm going to give a talk about scaling tech orgs. Wayne already did a quick intro about me, but really fast sweep down. It's about me. Start off my career in experimental physics, so kind of non traditional background. I am not a classically trained computer scientist, mostly spending time on the science is messy part in that part of my career and then got started at Reddit very early on, mostly doing platform and infrastructure work. I discovered very early on that I don't have much in the way of tolerance for fiddly client work and the work that we've been doing on Reddit that we're currently replacing is actually a lot of the work that I originally did as a lot younger and engineer. So kind of came full circle, came back as CTO and am undoing my own work which is always exciting. Talking about scaling a tech org and this is about basically where Reddit is in the journey as well as all of the work. It's gotten to scale Reddit up to current size and from experience other startups just for scale. Reddit currently is at about 1200 employees, which is always a terrifying number for me to say because the number is always bigger than the last time I said it. Also I would like to point out that all rants unless otherwise noted, are based on live experience either from me or from colleagues who are forgotten that I've learned the story from. So if you're in the audience colleague, I'm sorry that I stole your story. So a couple learnings up front as I was writing this talk. This is actually my learnings first learning is that I actually have strong opinions about org structure. Considering I got my career started doing technology, the fact that I have strong opinions about management and org structure is a little bit shocking. If I had a time machine, I'd like to go back and tell myself this to see the shock on my face. I'd like to probably pass on learnings here which are pitfalls that I have myself viewed from the bottom and try to come up with some preventative measures to avoid those pitfalls and soften the blow. Not all pitfalls are avoidable. Sometimes that's just part of learning and experience. But the point of these kinds of talks are to avoid mistakes of other people if possible. Okay, so I'm going to do this as a plain 4X, as I like to call it. Four different phases of a company, at least as seen from my the tech org. The first one is the unsurprising one which is the startup phase. This is the Company can be covered by a reasonable number of pizzas and this is usually scaling up from about 0 to 25, give or take. The nice thing about the size is that that's probably about as many pizzas as you can afford during the shoestream part of your company. Next up, I like to call the next stage the dawn of managers. This is when the company gets to be complicated enough that you have to think about structure and making sure not to in the process prematurely optimize. This is kind of around up to about 100, where a little bit of ad hoc management definitely helps out a lot. Phase three is the director phase. This is when you start getting into you can't keep the plan in your head and the you here is probably me. So you actually need people to start running the strategy and the plan. This is in the kind of 100 to 500 phase. And then last up, where we're currently at right now at Reddit, which is VPs and GMs, which is where things get very complicated and very fractal and also very exciting. A lot of interfaces there. Okay, so first up, startup phase. For those of you who are doing startups, none of this should be a surprise, but I think what I'd like to convey is we've all been in this before, so I'm going to use this visual representation a lot during this talk. So I've got this two dimensional graph of a team and the team is going to be broken up into blocks and across the horizontal axis I'm going to talk about different product units. And during the vertical axis I'm going to talk about server to client. Now I should mention that I'm a bit of a back end chauvinist. As I mentioned early on I was an infrastructure person, so I probably will be a little bit heavily biased towards talking about servers. I might actually start talking about websites. That's just a sign that I'm old generally. I mean platforms and apps and clients and all those other rich things on the client side. Okay, so startup side, totally easy. This is the team. You don't really have much of an org chart. You have a bunch of people who are working on the same project. This is a block of humans. Maybe you have some domain experts. Initially, these are some initial blobby startup people. We've all worked with them, we've all been them probably in some cases. Generally the pattern I usually see on the tech side especially is I usually have somebody who's working kind of heavily more in the back end and then you Have a couple of more client generalists to work on the product. This is if you're lucky enough to have more than one engineer to start with. If you start with one engineer, then it's kind of a one stop shop. You're working on everything. So again, no surprises here. The key takeaways on this org chart is that we're talking about like at most two dozen people. Everybody knows what everybody else is doing. And this is because ultimately N squared is very small. There's no coordination overhead. There is. If you don't know what you're working on, you probably should be working on a startup. And there, there's also no backups and no backstops. You're it. You might be the only person who knows that technology because you're the one who probably built it and are keeping it alive. And so, and you know, problems that come up in here are the fact there are no backstops, no redundancy. Somebody has a bad day, your bus number is probably one. So probably stay away from buses. And you know, a bad hire at this stage can actually be fatal to the company because you're Talking about potentially 10% of your company being a problem. So that is the, as they say, easy part. This is also the part where ultimately this is where the 10x engineer becomes really helpful if you can find them. So if you can get those blobs, those people and their domain expertise to be large, you can actually cover what you need very efficiently. It cuts down on coordination overhead, it gives you autonomy, and it gives you a chance to make sure that speed is covered. And everyone is already going to have too much surface area. So you have to hire people who have a lot of appetite for surface area and can manage complexity. So the result of this is that generally this is the point. It's kind of the one spot in a company where a handful of 10x engineers, if you can find them, can really cover everything and it's actually efficient. And again, the key point here is because they don't have to coordinate, they just go off and do their thing and hopefully lovely technology and product flows out from it. Now you should, those of you who've done a startup know that it's never quite this simple. I should also be telling you about unicorns and ponies, but this is the dream of the startup is small team and ultimately autonomy. Okay, so you've gotten through the startup, you've survived well. What's happening? Well, in the startup phase, you've got this blobby kind of chart where you have People who are autonomous, you start to hire in people to fill in these gaps. And that's not to say that these people you've hired in the next phase are going to be smaller impact. In fact, you're still very early in the company. Everyone here is a huge impact, but ultimately they're coming in a little later. They've got some, you know, some of the turfs already covered, and they're going to start to fill in places and act as redundancy and backstops for people you already have. What's going to break? Well, first of all, that was a much more complicated graph than I showed in the first one. Team's growing, now we start getting into coordination overhead. N squared is no longer going to be small. So, you know, we're talking about 25 people or more. That's N squared is like n squared over 2. We're talking about 300 possible relationships. That's a lot of relationships to keep track of. And honestly, at this point, if you're getting towards north of about 25 to 30 people, in my experience, you probably made a bad hire along the way and you've got to start dealing with that. So, you know, as we have weave later in the humans, we have some nice overlap. We probably have some friction because this lovely person in the middle who's kind of working full stack, hopefully OG number one is happy with them working full stack. And what you actually want, though, is you want to start thinking about this as conceptual units. So you have probably two product lines that are probably those original two OG blowups that I had on there. And you probably have a foundational team that wants to keep the stack together and deal with outages and downtime. And so the conical split you do is you do this, you have two product teams, infrastructure team, and you need to start hiring for managers to run those teams and start coordinating. Okay, so you probably have a tech leader too that you're thinking about turning into a manager. And those people probably don't actually have much of the management experience. If you're lucky, they do and they can take on that role. This is also the point at which when I interview later managers, they usually say like this is the point when they first had their management experience was taking on a team at a small startup. You know, usually one of the founders also takes on the role of being architect lead and generally buries themselves in one of these orgs as kind of the primary tech lead. And so, you know, we're thinking about at this point, two, three founders, company of 20 to 30. That means you need to hire a couple managers with span of control of about 10. That ultimately means that early on you're already having managers who are going to have to be fairly well loaded up to deal with interpersonal work. There's not much in the way of structure for the company, but there will be structure for the coordination and for the buildup. And I'll say, just as a point of order, 10 is a. For those of you who've done it, 10 is a fairly substantial load for management. Usually I think my record is about 14. And that was definitely the breaking point. So things to keep in mind. And the other part of this is, if you look at the pattern here, what we've just done is we've taken a high level of coordination overhead with N squared people and we've turned it into a low level of coordination overhead with N squared teams, where N is now three. So now we're at the stage where we just have interop between management teams and teams that connect somewhat autonomously internally. And so we've kind of divided the problem up now. And as you're kind of loading up, hopefully people you're adding in won't necessarily mess up the feng shui too quickly. You'll have a chance to do some adjustments, some reworks later on. This is also the point where you need some process because now you have multiple teams that are working somewhat autonomously. You actually have to figure out about, well, how do you make sure there's some contracts between teams? This is when you start getting lovely things like actually having to do proper agile or at least ticketing with some sort of consistent profile. You have to start doing planning in some sort of organized fashion. Managers need to talk about people, they have to start talking about career progressions. And I have to talk about what happens if a person wants to go from product team A to product team B. How are you going to move them over? What are they good at? And this is also a point when you need to start thinking about career ladders. So, you know, you start off, you get this lovely lump of a company, you get out the sorting hat and of course you find out sometimes that there is this person, we've all worked with this person, this person sometimes is useful. They're oftentimes fantastic. They don't quite fit into a bucket. And so that gives you a question of are they their own product line or has there been a problem? So this might also be the case where you made a bad hire. You generally discovered that now, and that's where HR comes into play. I found from experience, it's right around the 50 point where you start having HR people. HR is often treated as a dirty word in startups. It's not. It's a good way to start to think about structure of people and process. I mean the word, the acronym means human resources. And this is where you start getting people into actual time to think about things like onboarding and off boarding, as well as adding in the notion of career ladders. And so the theme that I'm trying to convey here is about this is the stage in the company where you start needing to develop some consistency. The org starts off being very lumpy and you need to start making it into more of a structured, organized beast with managers to take on the role of adding that structure. Managers need tools to do their job well and that's where the career ladders and the consistency and process comes into play. It's also about time when you hire it. I think a lot of us technologists tend to kind of handle it ourselves, at least from experience. Up to about 100 people, it's like, oh yeah, we could just like as part of the onboarding process, buy laptops and kind of set them up. That's probably not the best use of your time. It's probably fun. I for one, find it fun. I also generally enjoy running cable around offices. It's very cathartic and lovely. But I would say that getting specialists, this is a good time to do it. So a couple of things to avoid at this stage. I was going to call some problems, but that assumes I don't have an opinion. So first off, should you make levels public? Yes, just make levels public. People are going to talk, they're going to talk about their levels, they're going to compare notes. Should you make comp public? I think that's a harder question. But I would say that structure your comp such that if somebody accidentally printed it all out, you wouldn't have a bunch of egg on your face. And when you say which checking system would we use? They're all terrible. Just pick one. It's not even worth discussing. Just find one that works and move on. Oh, and really important thing, don't make everybody a principal engineer. You're going to want to. You're a startup, people are new, they've been around for a while. They're going to want a fancy title because they've been here forever. You're just taking out management debt or you're going to have to have like a career ladder that Stretches up to uber distinguished principal later and you have to clean it up. So some reasonableness of leveling up front will save you a lot of management burden later. Okay, I'm going to probably skip ahead. Just want to make sure that I'm keeping good track of the time. I'd say that the other thing to mention is that not every tech league wants to be a manager. Just remember that just because people are good at technology does not mean they're good at humans. Humans are terrible turing machines. They require a lot more handling and they require a different sort of coordination. So I'll just leave it at that. And the last thing I'll say about that at this point in the I'll be laboring a lot of points in here, but this is the point where the culture of the company starts to take root. Really. There's enough people, you're starting to get some momentum in certain directions and certain styles of working and be attentive because those tend to have long term implications for the overall culture of the company. Okay, so I already said some of this, but just to give you a sense, a lot of the cultural touchstones that are at Reddit really did take root at this point. Some of our longest term values that have been from about the size of about 100 evolve was one of the ones we've had ever since then. We originally had a value of listen debate ship, which is a good way to emphasize that the discussion is not the deliverable. The actual the shipping is deliverable and everyone does the dishes was a way to say that you have to clean up your own messes and you can't rely on them to clean it up, whoever them happens to be. All right, next phase directors. So good news, you've survived. The team is growing. Now we're back to coordination effort again. And script for teams is no longer small. And actually worse than that, even the execs start to lose track of the threads of work that are going on. Try to keep track of everything. Statistics work against you and you start having dysfunctional teams. And so we're kind of talking about the same problem, different scale. All right. This is also the point in your life when you probably need to house recruiters. Hiring has become a major thing. Definitely hire one and a good one up front because you're going to spend a lot of time with them continuing to grow the team. So we were here with a single pair of product teams. We're going to of course move to this lovely clean situation where we have six now independent units through Our running product. And of course it never works out like that. It actually looks like this. So what you have is this lovely mess of product teams that probably are working different parts of the stack. We have team two that's saying that they're short client specialist. We have a gap in the product that there are dragons there, probably somebody abandoned it, there's probably code still running, there are users still there. And of course you have team five that says things like that's infrastructure's job when they have to deal with backend work. So the role of at this point is to decide what do you want the shape of the team to be. This is when discussions of horizontal versus vertical become important. Horizontal means cross cutting across all initiatives. Vertical means individual products. The preferred model I like to use is usually called the layer cake with candles. As you can tell from some of my pictures, that usually looks like this first picture. So in that case you have a bunch of product teams that all are kind of embedded in this cake, that is infrastructure and plat. Other options are to have fully integrated autonomous product teams. You can also have fully integrated horizontal teams. The problems with those other two is that for the product teams you have to coordinate about architecture and do that in a more ad hoc fashion. For product teams you have to coordinate. Sorry, for horizontal teams you need to coordinate about product. So product doesn't actually have a coordination point in that situation. So you have to actually be deliberate about how you do that. Directors come in directors. I want to be very important distinction here is that directors are not just managers of managers. Directors also have to be able to ensure consistency of strategy which comes along with the job. This means also having a clarity of vision, a clarity of mission. They also need to be able to define an org structure and stick with it. And of course, critically, they have to mint, hire and grow managers, which is another level. This is, you know, it is in order to really know something, you have to be able to teach it. In order to be able to really, really know something, you have to be able to grow it. And that's where I think directors come into play here on the management side. So again, emphasis, directors are not just managers of managers. The exact view of how this kind of works out is you probably have a couple product orgs. You'll see this is a certain similarity with what I was talking about before. This time we have two product orgs with infrastructure team at the bottom. Inside of these little product orgs you have a lovely layer cake as well. You probably have a backend team. You probably have some QA and data science that are going to be horizontal. You have a little bitty layer cake inside the layer cake. Isn't that lovely? And then infrastructure, you probably have some more complicated structure in there as well. There's going to be internal technology, products, a lot to come into play. Ok, this is when reorgs happen. Now, I would like to say that reorgs are usually a dirty word because most people associate reorgs with layoffs. I'm not talking about layoffs. I'm talking about aligning teams with goals, which is going to be a lot of the work you're going to have going forward at this stage in the company. Products will shift, there'll be successes, there'll be failures, you have to kill things. Teams will only exist to fix things for certain durations of time. There will be attraction to want to shift between horizontal and vertical. This is okay. This is natural and healthy. Sometimes it's the right move. It's very hard to tell when. And also, honestly, not all directives will work out. That's kind of part of the game. People will move on, people will develop in their careers. You have to be resilient as a company for all of those things. So let me get out my coffee mug, which I actually forgot to put on my desk as a prop. This is when you have to run a reorg. Reorgs are tricky. I would say the way to think about a reorg is that you really want to get as much buy in as possible and move rather rapidly. Once you've actually decided the reorg has to happen, you want to start at the top. You want to decide on what the rough plan is and then immediately pull in the next layer of management. The idea is that at every step along the way, when you pull in the next layer, you're getting them to be accountable to the result and you're also buying them in on a plan. So you're not just continuously repping your own plan, they're repping our plan as part of this process. So by the time you get to the ICs, it's a formality. Everything's already been decided. You have all of your questions answered because you've already dealt with everything. The important part along the way, too is to iterate as you're going through this whole process and be flexible, because the direction you took up front might feel right. And when it actually hits implementation, you might discover it wasn't quite right. The flip side of that is stick to your guns. You're doing A rig for a reason. It's because something isn't working. Make it work. All right, last up. Kind of give you a taste of where we are right now. This is the place where I have the least answers because I'm actually in the thick of it, the age of VPs and GMs. So at this size, again, I'm at, you know, we're at the stage where good news, it's no longer about breaking. Well, hopefully it's about scaling. You're layering in people, you're hiring people faster. Like your weekly hiring rate is larger than your original startup, which is a terrifying thing to think about, but it's happening right now. You've entered the phase where you can plausibly say things like, I think we need a VP for that, which means somebody who's actually seen it and been around the block before. And it's very difficult to apply my 2D view, but it's also very hard to do with this in 3D. So I'm going to try this. Okay, so my current org, just to give everyone a sense, is I've got four teams. I've got core engineering, which is where our VP of Eng lives, and this is basically infrastructure product. All the primary engineering teams for Reddit live in Core Engineering. We have a team called Data irl, which is a combination of data science, analytics, as well as search, relevance and ML Engineering. This is a blended team that covers all the smarts in the company. To make a little joke about it, I actually should have shifted into my original Boston accent for that one. It would come up more funny. Next up, we've got Trust. This is a very somewhat Reddit specific group. This covers user safety, operations as well as engineering for those teams. Security, privacy, grc. This is the most complicated team in my org. And last but not least, Economy. Economy covers DTC monetization. So let me just give you a real quick run through of what this looks like when you lay it out on the plan that I've suggested. We start off with we have a layer kicking candles inside of Core Engineering. This is our product teams sitting on top of a client platform that runs things like QA and an infrastructure team. And then next to it we've kind of built out an economy team which is under a separate VP overlapping infrastructure, but somewhat autonomous as we moved out to services. So you can do some scale up client platform covers everything because unfortunately everything has to go through the same app, which is a delightful experience to deal with on a two week or one week release cycle. Depending on cadence and then you start layering things like the data org. So data Org has a couple of its own verticals. Those are things like search and relevance and ML and feeds. And in our case we also have a bunch of backend integrations. That is our ML platform which has to integrate with our entire infrastructure stack and we also have up on the top we have our data science analytics stripe which has to integrate with all the product teams. We do that via a federated embedded model. And the last set of boxes before I just abandon ship safety. So trust and safety covers not only user protection up at the top, it also covers security. I probably should draw a box around entire thing to say there's perimeter defense that has to go into play here and they also have their own separate verticals and not shown on this I should mention is the monetization engineering and ads actually runs in a separate org according to our COO. So I think you can understand why VPs and GMs come into play here. So yeah, it's complicated but the lessons actually still apply. What's changed here is that horizontal and vertical split doesn't quite fit anymore. Teams are both. Each org has a business capability that has to be run as a business function which is where VPs come into play and GM start to show up for kind of broad cross cutting initiatives that need cohesive vision. And with that I will consider you having been DDoS boxes and to say thanks for joining and let's switch this over to being a QA session.

Thanks Chris. So first question and thanks everyone for asking the questions. Please do vote up the questions you'd like to see. He asked most and add others is from Stephen Lagoulet. For how long did you experience having the hero culture in the early days and what was the best way? What do you think is the best.

Way to change it? Is it possible to change I guess the question so we saw a transition from hero culture to a certain amount of more like suffering and silence as a side effect of one thing I didn't include in this is the like the OG early startup tension with the subsequent phases of the company because if you look at the early chart you end up having the hero culture is in there because literally everybody has to be kind of a hero to cover things. And it's not that you're necessarily stealing that work away from them as you scale, but they have less control over the ultimate direction of the company. And so there does tend to be a split of usually first 50 employees have a slightly different culture than the next thousand. And so what we found is that the results there are, some people do grow and they turn into architects and they turn into people who are going to scale with the company. Other people will end up finding that this isn't a great fit for them and they want to go off and find another startup. And that's actually okay. It sucks when it happens in the same way that losing an important person you've had a relationship with for a long time will suck. But that's kind of nature of the beast. And then the third group is the one to kind of watch, which is they haven't quite figured out where they're going to end up and they'll stick around and they'll enjoy themselves, but you have to make sure that you're balancing business needs with needs of those OGs. I think that, you know, actually I'll back up though, and say I do actually like having a little bit of hero culture sticking around. The one caveat and the one thing I'll add to it is I had my VP of Engine at one point tell me this. So, you know, having been the original, one of the early, early employees, I tend to be the rent into the burning building type myself. And at one point my VP of Bench turned to me and said, chris, I love the fact that you're the first to run into the burning building. What I need you to do is say, hey, everybody, come run into the burning building with me. And that actually helps to, you know, start encouraging this to be kind of being everyone's problem.

Like it. Oh, and I didn't want to fail to mention that Chris and I are also going to be available in the lounge after this session at Talk Live. So next question. Why don't we take one from Steven? So Stefan.

So for.

No, no, sorry.

Nope.

That's when we just asked. So from Sarah. What are the most tempting systems that people consider during the dawn of managers phase, but which ought to be avoided like the plague?

You know, it's funny, I think that that's the phase where you have to go in. Okay, so if I'm going to be a little bit opinionated, you have to go into something agile. I think that my opinion at that phase is you have to be flexible with your agile approach. I've joked for a very long time that I consider myself to be an agile unitarian, which just means I believe in the deity, but I don't believe in any particular face of the deity. And so if you kind of go in too hard with any one process at that stage it's probably going to end up breaking. So I think I would rather say, rather than talk smack about specific things I've seen go wrong, I'd actually say keep it simple and flexible. I would say that one thing to avoid the plague early on for processes is things like. And one thing we actually fell victim to is don't make your career ladder too granular. It's actually okay to have the first version of career ladder, have kind of broad buckets and just kind of assume that you're not going to get it exactly right. We went a little bit too into having almost like micro leveling and we're too rigid with the way we. And it worked actually fairly well for a while. The side effect is it costs us flexibility. So I think it's okay to have a little bit of wiggle room on that. And so I think the main thing I want to convey here is like don't overgeneralize up front. Kind of do the process that you need to kind of get to the next level. And the process is probably things like basic ticketing, basic agile approach, basic career ladder, and then let HR take on some of the rest of it. Great.

So next question from Kimishan. Is it wise to split products at product team one and product team two at different stages? Could it be useful to do some domain driven design and event storming at the stage to be more deliberate about boundaries?

Yeah, and that kind of goes into the horizontal versus vertical question. So I think what's going to happen with. If not breaking into product teams early is perfectly reasonable. And in fact the way that we originally broke up was not by product surface but by client type. And so oftentimes you see that the original canonical split, back in my day it originally was like the web team and the back end team. As you kind of layered in more clients usually ends up being web team, mobile team or mobile teams, with each one covering a particular client surface and then breaking up by product comes a lot later, like at around the kind of 100ish range in this case. I was thinking of more product as being conceptual unit of the product. That makes sense that that size because honestly there's going to be still a lot of cross coordination at that scale. Where things get dicey is having to move into. We moved into the mode where we actually did the federated mobile engineering approach at about, let's say about 150 people, which was probably a little bit late, but it was about the right time for us, if that helps.

What's your favorite Reddit community?

My favorite Reddit community. Oh man, I've got a couple of them. So I've always been a fan of change. My view just because it's amazing to see that people can have a reasonable rational debate on the Internet even today. I didn't know it seems impossible to talk about or like you can have long form comments that aren't vitriol. So I think that's pretty high in the list. I think I so being a parent, I definitely also am a big fan of things involving kids doing stupid things. I think usually things like hold my juice box is high on the list of just ridiculous situations kids get themselves in. And then the other side of the spectrum I'm kind of I enjoy the makerspace a lot and 3D printing has been my hobby for a while though I'd say actually at this point in my life having two kids in sports. I'm also almost a full time chauffeur, which I don't have a subreddit for right now.

Well, thanks Chris. Well, everyone will be in the lounge if you have any other questions for Chris or anything you'd like to mention to me. Really appreciate it Chris.

Thank you. Thank you.