In this talk, Nandini Javagal, a Production Engineering manager at Facebook, explains how production engineering (PE) differs from traditional operations and Site Reliability Engineering (SRE) models and why it offers strategic advantages. She traces the evolution from the “throw‑it‑over‑the‑wall” ops approach, through Google’s SRE principles (blameless post‑mortems, SLO/SLA/SLI, error budgets), to Facebook’s embedded PE model that integrates engineers directly into software teams. Javagal describes the PE role across the entire software development lifecycle—participating in requirements, architecture reviews, code diffs, testing, deployment, on‑call, and tooling—focusing on reliability, scalability, performance, and developer efficiency. She presents a maturity pyramid that moves teams from low‑level server provisioning up to high‑level reliability, scaling, disaster recovery, and performance tuning, illustrating how Facebook’s PE teams operate at the top of this pyramid. The talk also outlines practical steps to adopt PE: securing management buy‑in, aligning culture (blameless post‑mortems), developing the necessary skill set, and treating PE engineers as peers rather than sub‑servient support. Finally, she emphasizes that choosing an operational model shapes organizational culture and velocity: while a throw‑over‑the‑wall model may enable fast feature development, it creates bottlenecks later; an embedded PE model may start slower but ultimately delivers faster, more reliable delivery of features at scale. The session concludes with actionable takeaways for attendees to consider implementing production engineering in their own organizations.
Chapters
Hi folks, I'm Nandini Javagal. I'm a production engineering manager at Facebook. I've been there for the last 1 year. Before that, I was a director of infrastructure engineering at Proofpoint, which is a security as a service. And at Proofpoint, I've gone through many roles, including director of quality and director of infrastructure. In my prior life, before that, I used to be a software engineer. You can say that I've been around the block a few times. As I start, I want to make sure that I understand the audience. Is there a way, Alex, that we could get the audience to answer whether they are in traditional ops, the SRE or DevOps role, or production engineering? All right. It looks like we have mostly a combination of traditional ops or SRE or DevOps. So we'll go with that. I hope at the end of this session, you would understand what is production engineering and how would you make that, what are the key advantages of production engineering. If you would like to move to that model, how would you try and move to that model? In the slides that we will post later, I will probably put some additional reading since we have mostly traditional ops and SREs here. As I see the evolution of operations, we have had— and a large number of companies do have traditional ops where you have software engineers developing software and suddenly you get a package, I call it the throw it over the wall model, but you get a package to be deployed to production, and after that, somehow the operations handles everything. They handle the deployment, they handle the on-call, they handle the monitoring of it, and if anything were to happen, they would get called first. So you may also have a knock or a, something like that, picking up a lot of the calls. Google and a few other companies introduced the concept of SRE, where they used to hire developers into the SRE role, so that people understand how to develop, and they used to make sure that they have, actually, the Google SRE manifesto has about principles. So the 5% of the ops works kind of goes to the development team. The SRE load can only be 50%, but the most important parts and the parts that really should be carried over is the blameless postmortems, the keeping of the SLO, SLA, SLI, and the error budgets, and also the practice of reliability engineering. So suddenly now we have gone from throw it over the wall to a little bit more skin in the game for the engineering organization, for the software engineering organization to participate in the operational aspects of the service. Some of the teams may also be called DevOps. By the way, DevOps is actually a way of doing things rather than an organizational structure, so I would veer away from DevOps titles. So this is what Google came up with, and it was an amazing evolution of operations. It definitely made a lot more sense now for engineers to participate in the error budgets and understanding what is happening. And that is the genesis of production engineering, because a lot of production engineers, the leaders of production engineers actually came from Google. And so they said, understood what was happening at Facebook, where operation engineers were monitoring the site and things were always crashing and it would fire more constantly. And this was in the late 2010 and early 2011 and '12. Pedro Cano-Hatti and Santosh Janardhan, along with Jay Parekh, they started this organization called Production Engineering, and they slowly took the SRE model but then improved it and made sure that we are— that production engineers are more embedded into the Facebook engineering structure. And that is what I will go over now. And as you can see, on one end of the spectrum, you have throw it over the wall, On the other end, you have really embedded production engineers who are taking part in the whole software development process alongside or in partnership with the software engineering team and the other cross-functional teams. It is not just software engineering that we partner with. All right, production engineering is a hybrid of systems and software engineering. I am trying to keep it simple. Simple instead of going through the entire Production Engineering Manifesto. The reason why we have to be a hybrid of systems and software is we have to be able to go up and down the stack, and we will come, come to that in a few minutes. And the reason why we have to be software engineers is because we have to be able to understand and contribute to the software throughout the software development lifecycle, and I'll come to that in a second. Let's take the first part, embedded model. So we don't sit near the software engineers, we don't sit next to the software engineers, we sit with the software engineers. So it is a totally embedded model. We are in all of their meetings. And so I don't want to use their because there is no them and us. So we are us. So we are in all the meetings that are happening in the team. We understand the backlog, we understand the roadmap, we understand all of the diffs that are happening. Sometimes we are reviewing the diffs and we are also in the on-call and we are not the only people in on-call. Actually, the software engineers and production engineers are actually all added into the on-call as equals. So if there are 10 people in the team and 8 of them are software engineers, 2 of them are production engineering, then we go on call once through that cycle as well. So that is one of the items that you want to think about when you are in an embedded model. And the role in software development lifecycle is very critical. If production engineers, as production engineers, our focus is on reliability, scalability, efficiency, and dev efficiency. If we focus only on those aspects, then we miss a lot of the business problems that we are trying to solve. And that will not provide us a good longer-term vision. So the way I look at PE's role in software development lifecycle is that you're involved right from the requirements. You have to understand the business problem, maybe not to an extreme detail as, as the person who would write the feature itself, but you need to understand enough because you need to be able to look at the architecture and say, is this going to be a good candidate to go into production? What are the right metrics to measure? What are the reliability metrics to measure? Will this scale? If you are using 10 resources for 10 transactions, if you go to a million transactions, will you have a million resources? Is this scaling well? These are things that a production engineer can contribute to right at the architecture phase instead of solving it in a reactive manner. Think about it as proactive. During development and testing, we talked about it, you're reviewing diffs along with the engineers, and you production engineers are also contributing to the reliability aspects. So you are coding alongside for that particular project as well. And then when it comes to production, you do want to make sure that all the error budgets, the dashboards, everything is ready. And on-call is not just a production engineer's responsibility, the entire software the entire team, including software engineers and production engineers are on call. I talk about productivity and tooling because a lot of times we see that— so sure, now you're in production, but you may be having a set of issues that any person on call is spending 50% of their time. Now, imagine in a 10-person team, if you're spending 50% of one person's time, time on call, that is a lot of productivity that has gone waste. So these are things that you want to understand. Look at the on-call summaries and understand how we can eliminate the on-call workload or reduce backlog that you can contribute to on an ongoing fashion. Same with tooling. Facebook has an amazing amount of tooling to be able to understand how we can do some of the things very easily, whether it is dashboarding, whether it is whether it is looking at the trends, whether it is looking at the errors and double-clicking into it. So these are things that several teams in the infrastructure organization have developed for the productivity aspects. I talked about production engineering focus. We have reliability, keeping the service up and running, not only just keeping the site up and running, but also to make sure that the answers given, if you are an API service, are correct and on time. So think about all of the metrics there that come into picture. Scalability, I talked to you about it. If you have 10 resources for 10 transactions, will it really scale as you go to millions of transactions? Similar things, performance, what are the CPU and memory usage that you're using? And all of you your reliable, all of the layers underneath, are they performant enough or are you going to end up in a waiting manner where you are dependent on a service which is synchronous, let's say, and then you are kind of getting stuck in those layers. Dev efficiency, we talked about it. How can you make all the developers more efficient by, by developing tool sets, by utilizing the tools, and by leveraging some of the tools. At Facebook, we do have a lot of these things, but if you are not at Facebook, you could consider how are some of the other open source software or other vendors that could be used to improve your developer efficiency. And this includes not just software engineers, it also includes ML engineers as well. Here's a pyramid I will spend a few minutes on, but the slides are available for you to refer to later. And this is one of the important parts of the production engineering at Facebook. So at the bottom, you have server hardware and provisioning and server monitoring and lifecycle. A lot of ops and SREs tend to focus here, and the reason why It is not bad, somebody has to do this and we have to get it done. But after this, as the engagement matures, you really want to move up and up, and you want to understand how are you monitoring the service, how are you deploying it, is the deployment going easy, or you need manual handholding to ensure that a service is deploying to production and it is not deploying without any issues. You may want to have gatekeepers so you can roll out the traffic for 5%, 10% before you get to 100%. Scaling and disaster recovery, how are you doing if one whole data center goes off or other issues? This is where we look at some of these best practices and implement that. We talked about performance tuning to see what resources are being used and how we can make sure. And then the one-person stuff, the team that I personally am supporting, we are in this top area because we have worked, we use Facebook's infrastructure, so we're not involved in the server hardware provisioning, but we have definitely matured our our engagement with our SWE partners to be able to do scaling, performance tuning, and definitely contributing to the reliability of our services. So you really want to see where you are and how you can get to the next stage of maturity. How would one get started on PE? So like I talked about, it is hard, it is not easy, which is why you need not just yourself. It cannot be a grassroots kind of evolution. You do need some amount of management buy-in as well. So you, the, you plus the management will need to set a vision of how this would work and how you would make it work and how this would be good for the overall company because you are involved in the entire SDLC. So you want to get the alignment between the current organization and the software engineering organization and anybody else, your program management and others as well. As you bring in the PE, you have to change your own culture, thinking, and skill set as well. So this is something that is very important. Without, without the cultural aspects of blameless postmortems and things like that, it becomes very hard for you to get involved in a, in an easier manner. And finally, there are some pathways I mentioned earlier that you have to understand how to get into the mindset of the developers by being a peer. You're not— this is not a subservient relationship. You are a peer because you are an engineer as well who can code well, who has analytical skills, who has business thinking skills, because you really have to do this in partnership with your software engineering organization. All right, I want to quickly go over this one so that we can get to the discussion part of it. The top takeaways are to choose your operation model because it drives the culture, the culture of the company, the culture of the software development. It drives the velocity because As I mentioned, if you have a throw it over the wall model, then you really are developing very fast only to get stuck in the operational area. Or if you're using the SRE model, then only 5 or 50% of this thing is coming into the SRE part of it. You really want it to be smooth throughout the development lifecycle. It is probably slower initially, but then you go slow initially to be able to go fast. This will enable growth in your entire organization because you have a much smoother pipeline of features from requirements all the way to production and successful production. So that's what I would like to have you all take away. And I hope that you can look at how to incorporate production engineering in your own company. Happy to help. And lead you as needed. I think my contact is in the initial slides. Thank you all, appreciate you joining us.
