The Key Advantages of Production Engineering vs Traditional Ops Teams with Facebook

Hi folks, I'm Nandini Javagal. I'm a Production Engineering Manager at Facebook. I've been there for the last one year. Before that I was a Director of Infrastructure Engineering at proofpoint is a security as a service. And at proofpoint I've gone through many roles including Director of Quality and Director of Infrastructure in my prior life. Before that I used to be a software engineer. You can say that I've been around the block a few times. As I start, I want to make sure that I understand the audience. Is there a way, Alex, that we could get the audience to answer whether they are in traditional ops, the SRE or DevOps role, or production engineering? All right, it looks like we have mostly a combination of traditional OPS or SRE or DevOps. We'll go with that. And I hope at the end of this session you would understand what is production engineering and how would you make that? What are the key advantages of production engineering? And if you would like to move to that model, how would you try and move to that model? And in the slides that we will post later, I will probably put some additional reading since we have mostly traditional ops and SREs here. As I see the evolution of operations, we have had a large number of companies do have traditional ops, where you have software engineers developing software and suddenly you get a package, I call it the throw it over the wall model. But you get a package to be deployed to production and after that somehow the operations handles everything. They handle the deployment, they handle the on call, they handle the monitoring of it, and if anything were to happen, they would get called first. So you may also have a knock or a something like that picking up a lot of the calls. Google and a few other companies introduced the concept of sre where they used to hire developers into the SRE role that people understand how to develop and they used to make sure that they have. Actually the Google SRE manifesto has about principles. So the 5% of the ops works kind of goes to the development team. The SRE load can only be 50%. But the most important parts and the parts that really should be carried over is the blameless post mortems, the keeping of the slo, sla, SLI and the error budgets, and also the practice of reliability engineering. So suddenly now we have gone from throw it over the wall to a little bit more skin in the game for the engineering organization, for the software engineering organization to participate in the operational aspects of the service. Some of the teams may also be called DevOps. By the way, DevOps is actually a way of doing things rather than an organizational structure. So I would Veer away from DevOps titles. So this is what Google came up with and it was an amazing evolution of operations. It definitely made a lot more sense now for engineers to participate in the error budgets and understanding what is happening. And that is the genesis of production engineering because a lot of production engineers, the leaders of production engineers actually came from Google and so they said they understood what was happening at Facebook where operation engineers were monitoring the site and things were always crashing and higher, more constantly. And this was in the late 2010 and early 2010, 11 and 12. Pedro Kanohati and Santosh Janardhan along with Jay Parikh, they started this organization called Production Engineering and they slowly took the SRE model but then improved it and made sure that production engineers are more embedded into the Facebook engineering structure. And that is what I will go over now. And as you can see on one end of the spectrum you have throw it over the wall and and on the other end you have really embedded production engineers who are taking part in the whole software development process alongside or in partnership with the software engineering team and the other cross functional teams. It is not just software engineering that we partner with. All right. Production engineering is a hybrid of systems and software engineering. I am trying to keep it simple instead of going through the entire production engineering manifesto. The reason why we have to be a hybrid of systems and software is we have to be able to go up and down the stack and we will come to that in a few minutes. And the reason why we have to be software engineers is because we have to be able to understand and contribute to the software throughout the software development life cycle. And I'll come to that in a second. Let's take the first part. Embedded model. So we don't sit near the software engineers, we don't sit next to the software engineers, we sit with the software engineers. So it is a totally embedded model. We are in all of their meetings and so I don't want to use there because there is no them and us. So we. So we are in all the meetings that are happening in the team. We understand the backlog, we understand the roadmap, we understand all of the diffs that are happening. Sometimes we are reviewing the devs and we are also in the on call. And we are not the only people in on call. Actually the software engineers and production engineers are actually all added into the on call as equals. So if there are 10 people in the team and eight of them are software engineers, two of them are production engineering, then we go on call once through that cycle as well. So that is one of the items that you want to think about when you are in an embedded model. And the role in software development life cycle is very critical if production engineers, as production engineers, our focus is on reliability, scalability, efficiency and dev efficiency. If we focus only on those aspects, then we miss a lot of the business problems that we are trying to solve that will not provide us good longer term vision. The way I look at PE's role in software development life cycle is that you're involved right from the requirements. You have to understand the business problem. Maybe not to an extreme detail as the person who would write the feature itself, but you need to understand enough because you need to be able to look at the architecture and say, is this going to be a good candidate to go into production? What are the right metrics to measure? What are the reliability metrics to measure? Will this scale if you are using 10 resources for 10. Transactions, if you go to a million transactions, will you have a million resources? Is this scaling well? So these are things that production engineer can contribute to right at the architecture phase instead of solving it in a reactive manner. So think about it as proactive during development and testing. We talked about it, you're reviewing diffs along with the engineers. And you production engineers are also contributing to the reliability aspect. So you are coding alongside for that particular project as well. And then when it comes to production, you do want to make sure that all the error budgets, the dashboards, everything, everything is ready and on call is not just production engineers responsibility. The entire softw, the entire team, including software engineers and production engineers are on call. I talk about productivity and tooling because a lot of times we see that. So sure, now you're in production, but you may be having a set of issues that any person on call is spending 50% of their time. Now imagine in a 10 person team, if you're spending 50% of one person time on call, that is a lot of productivity that has gone waste. So these are things that you want to understand. Look at the on call summaries and understand how we can eliminate the on call workload or reduce backlog that you can contribute to on an ongoing fashion. Same with tooling. Facebook has an amazing amount of tooling to be able to understand how we can do some of the things very easily. Whether it is dashboarding, whether it is looking at the trends, whether it is looking at the errors and double clicking into it. So these are things that several teams in the Infrastructure organization have developed for the productivity aspects. I talked about production engineering focus. We have reliability, keeping the service up and running and not only just keeping the site up and running, but also to make sure that the answers given if you are an API service are correct and on time. So think about all of the metrics there that come into picture. Scalability. I talked to you about it. If you have 10 resources for 10 transactions, will it really scale as you go to millions of transaction similar things? Performance. What are the CPU and memory usage that you're using and all of your reliable all of the layers underneath, Are they performant enough or are you going to end up in a waiting manner where you're dependent on a service which is synchronous let's say and then you are kind of getting stuck in those layers. Dev efficiency we talked about it. How can you make all the developers more efficient by developing tool sets, by utilizing the tools and by leveraging some of the tools. At Facebook we do have a lot of these things. But if you are not at Facebook you could consider how are some of the other open source software or other vendors that could be used to improve your developer efficiency. And this includes not just software engineers, it also includes ML engineers as well. Here's a pyramid I will spend a few minutes on but the slides are available for you to refer to later. And this is one of the important parts of the production engineering at Facebook. So at the bottom you have server hardware and provisioning and server monitoring and life cycle. A lot of OPs and SREs tend to focus here and the reason why it is not bad is somebody has to do this and we have to get it done. But after this, as the engagement matures, you really want to move up and up and you want to understand how are you monitoring the service, how are you deploying it? Is the deployment going easy or you know you need manual hand holding to be to ensure that a service is deploying to production and it is not deploying without any issues. So you may want to have gatekeepers so you can roll out for the traffic for 5%, 10% before you get to 100% scaling and disaster recovery. Are you. How are you doing if one whole data center goes off or other issues. So this is where we at some of these best practices and implement that. We talked about performance tuning to see what resources are being used and how we can make sure and then the one person stuff, the team that I personally am supporting, we are in this top area because we have worked. We use the Facebook's infrastructure. So we're not involved in the server hardware provisioning, but we have definitely matured our our engagement with our suite partners to be able to do scaling, performance tuning and definitely contributing to the reliability of our services. You really want to see where you are and how you can get to the next stage of maturity. How would one get started on pe? So like I talked about, it is hard, it is not easy, which is why you need not just yourself, it cannot be a grassroots kind of evolution. You do need some amount of management buy in as well. So you plus the management will need to set a vision of how this would work and how you would make it work and how this would be good for the overall company. Because you are involved in the entire sdlc. So you want to get the alignment between the current organization and the software engineering organization and anybody else, your program management and others as well as you bring in the pe. You have to change your own culture, thinking and skill set as well. So this is something that is very important. Without the cultural aspects of blameless postmortems and things like that, it becomes very hard for you to get involved in an easier manner. And finally there are some pathways I mentioned earlier that you have to understand how to get into the mindset of the developers by being a peer. This is not a subservient relationship. You are a peer because you are an engineer as well, who can code well, who has analytical skills, who has business thinking skills. Because you really have to do this in partnership with your software engineering organization. All right, I want to quickly go over this one so that we can get to the discussion part of it. The top takeaways are to choose your operation model because it drives the culture, the culture of the company, the culture of the software development, it drives the velocity because as I mentioned, if you have throw it over the wall model then you really are developing very fast only to get stuck in the operational area. Or if you're using the SRE model, then only 5 or 50% of this thing is coming into the SRE part of it. You really want it to be smooth throughout the development life cycle. It is probably slower initially, but then you go slow initially to be able to go F. This will enable growth in your entire organization because you have a much smoother pipeline of features from requirements all the way to production and successful production. So that's what I would like to have you all take away and I hope that you can look at how to incorporate production engineering in your own company. Happy to help and and lead you as needed. I think my contact is in the initial slides. Thank you all. Appreciate you joining us. It.