Site Reliability Engineering the Big Picture with Elton Stoneman

Welcome to the Test Guild Performance and Site Reliability Podcast where we all get together to learn more about performance testing with your host, Joe Calantonio.

Hey, it's Joe. And welcome to another episode of the Test Guild Performance Podcast. Today we'll be talking with Elton Stoneman all about his pluralsight course on site Reliability Engineering the Big Picture Elton is a 10 time Microsoft MVP, a docker captain, author, trainer and speaker. He spent most of his career as a consultant working in Microsoft technologies, architecting and delivering complex solutions for industry leaders. He's developed APIs and Azure, serving millions of clients daily, big data solutions processing billions of events weekly, and cutting edge solutions powered by containers. Elton's experience with. NET goes from. NET 1.0 running on Windows Server right up to. NET Core running on Linux when Docker emerges. The best way to run application. Elton made containers his focus so he travels around the world helping people understand containers, Kubernetes and DevOps and what DevOps could do for them in their software delivery. You don't want to miss this episode. Check it out.

Let's face it, performance testing is tough. Traditional load testing tools create scripts that are bogged down in data that's hard to read and require a lot of work, even for simple playback. Load Ninja cuts out the dirty work by using AI to inspect and debug your code almost instantly. No coding necessary. No changing of hands or relays between teams either. Give it a shot. It's free and easy to try. Head on over to testguild.com smartbear and click on the link to learn more.

Hey Elton, welcome to the Guild.

Hey Joe, thanks very much for having me. Great to be on the show.

Awesome. Awesome. Is there anything I missed? An intro that you'd like to share with us?

Well, the only thing was that I worked with Docker for three years when I was consulting and I got really heavily into containers. I worked on a pure DevOps project back in 2014 when it was pretty new and exciting. Got really interested in containers and then I went to join Docker and I worked with them for three years and now I'm freelancing. So now I do like consulting and training. So yeah, I think that was about all you missed.

Oh yeah. So another thing that I said in your bio was that you're a Docker Captain. What is a Docker Captain?

It's kind of Docker's equivalent of Microsoft's MVP program. So it's a recognition of kind of expertise, but also about sharing stuff in the community. So the original captains were the People who were out there blogging and talking about Docker in the early days. Yeah, and it's a kind of recognition. We get some of the same benefits that you get as an mvp. You know, we get to talk to the product teams, we get to raise any issues that our clients are facing, and we can kind of feedback into the product cycle. We get early releases of stuff that's coming out. So, yeah, it's a kind of, you know, it's a kind of reward stroke. You know, we're kind of. It's a mutual benefit thing because we help spread the word for Docker, too.

So you did cover some of this within your course. I really enjoyed your course on cyber liability engineering. I think it's a really great primer for someone that is clueless about cyber liability engineering. And I'm a newbie myself, so it really was helpful, so I really appreciate it.

Yeah, that's exactly what it's pitched at. I mean, lots of people have a kind of intuitive understanding of DevOps. Whether they've done it or not, they kind of get what it's meant to look like. But SRE is this kind of separate thing that people haven't really touched or felt or know too much about other than it's, you know, it's the Google thing. And I think that maybe puts people off, you know, because almost every organization isn't Google and doesn't have the problems that Google has. But actually, when you dig into site reliability engineering, it's a really attractive alternative to DevOps for all sorts of sizes of organization.

Absolutely. This course is broken out into four main sections. It's Intro to sre, Automation, Service Level, and Incident Management. So I thought we'd go a little bit into each one, but the first one you mentioned is DevOps. How is SRE different from DevOps and can they coexist in the same organization?

Good question. So I don't think they would necessarily coexist unless you had an organization that was adopting DevOps and was finding it a struggle. I mean, that's the big thing for me. I mean, SRE is different in many ways. They try and solve some of the same problems, like it's about getting better software out more frequently. Fundamentally, that's what it's about. Giving the business what they actually want as frequently as possible without things going wrong. That's what they're both kind of gearing towards. Removing all the friction between functions of the IT department and between the IT department and other functions of the organization. So, you know, they're all about that. But DevOps is quite heavy approach because it requires you to restructure your organization. Whereas SRE has, has these really clear practices that Google prescribe. You know, it tells you, do this, do this and do this, and these are the results you should expect and these are the benefits you're going to get, which is a lot easier for a lot of companies to kind of pick up with. Whereas DevOps is, is a, a kind of root and branch thing. I think SRE is something you can implement like one facet at a time. So things like post mortems, so that's right at the end, so we'll probably come to that later on in the conversation. But as a simple example, so SRE says if you have an incident and when you fixed it, you should write a postmortem. And that describes what happened, what the root cause was, how you fixed it. And you publish that throughout the company so that people can see what you did, what the problem was. They can reflect on anything that might impact them. Little things like that. So sre, you can take the pieces that you want and gradually evolve it. Whereas DevOps is a more like you have to do the whole thing because it starts with remodeling your team. Really.

Do you see a big push now for companies that went towards DevOps to go now towards SRE? Because it is a term I heard more and more of the past few years.

I think it's getting more and more popular. I mean, certainly some of the companies that I've worked with that were adopting DevOps were finding it really hard. There's typically kind of two approaches. There's either we're going all in on DevOps and like, we've got, we've got all the training, we've got the consultants in to tell us how to do this stuff. They, they realize there's a, there's a cost involved. Because typically, you know, the companies I typically work with are big enterprise organizations. They historically have a lot of older applications that don't have things like automation or monitoring or anything like that. And to move to DevOps is about, well, we need all that stuff kind of before we get anywhere. We need to reorganize our teams so that we have one team that owns the product, which is a combination of dev and ops. And dev and ops traditionally never even spoke to each other, let alone worked on a project together. So there's a fairly substantial change. And what I've seen happen in the past is one of two things. Either it kind of goes south because it just takes too long. So it gets stale. People lose the interest or they were never engaged in the first place. If it's come down as a decision from the top that we're doing this without getting the people who are actually changing involved in the process, or the other thing that happens is it works really well in a small project. So you often see this like this team did it, this is what they've done, they did it really well. And we're going to duplicate that across the whole company. And it just kind of doesn't happen because not every team's the same, not every team's got the enthusiasm or the expertise or whatever and then it kind of stalls. And so I think companies find DevOps hard because, because it's such a big change. I think that's why people are looking to SRE more now. I should probably give a kind of rough idea of what SRE looks like.

Yeah. If you give an example, what does that mean? What does an SRE workflow look like in a day to day kind of operation?

I guess, yeah, great, great point. So sre, it originated at Google. It's how Google run their production systems. The idea is that you have a product dev team who build the application, they build the features, they deliver what the business require from them, and then you have the SRE team who look after the application in production. So ideally the SRE team and the product dev team will work really closely together, but there is a clear delineation which is similar to the dev team and the ops team that you get in a traditional IT organization. That's one reason why it's easier to make that transition, because the teams are effectively changing the way they behave and they're changing some of their approaches to. But they can stay the same teams. You don't have to completely mix them together. The SRE team. The big difference with kind of traditional operations is that automation is at the heart of what SRE does. So part of their goal is that the SRE team spend no more than 50% of their time doing kind of mundane, repetitive work. They'll have 50% of their time guaranteed to do project work, which is about automating away the kind of tedious work. It's about high level beneficial stuff like improving performance, improving monitoring, improving the release process, improving rollbacks, all that sort of stuff. So it's engineering as opposed to just administering the system. So that's the kind of big difference.

My only experience with DevOps is from a huge company that perverted everything. So I don't know if it was really truly DevOps, but it sounds what you're explaining like though, they said we're just in charge of the environment, scaling them up, the process, cicd. But it's up to the team to do all the other things, the monitoring, the automation, all that stuff. So it sounds like maybe this is wrong. SRE sounds almost more agile, where you could do sprint by sprint, and it grows over time based on feedback or what happened during the lifecycle.

Yeah, I think that can happen. So, I mean, the idea is what you described there about DevOps perversion is, I think, very common. Back in the early days of Agile, again, I was very fortunate. I worked on a project, there was a Scrum project back in the day when Scrum was brand new and we did it literally by the book. Everyone and everyone on the team got a copy of the Scrum book and we just did exactly what they said and it worked really well. And then, you know, people see, see Scrum performing and they try to adopt it themselves. So they bend it and they twist it and then it kind of isn't really the same thing. I think that is a danger with DevOps. But SRE, like I say, it's quite prescriptive. You know, if you read the textbooks that Google have produced, there are two books, Site Reliability Engineering and the SRE Workbook, and they kind of say that you should do this. They're very prescriptive in what they tell you to do, and a lot of companies, I think, find that much more attractive. Just tell me what I need to do, tell me, you know, it works and tell me what it should look like and then I can go and do it. And maybe they'll twist little pieces, but ultimately SRE in one place should be very similar to SRE in another place. Whereas DevOps is going to change from organization to organization, I think.

Yeah, once again, like I said, we had a DevOps group like, oh, you need to automate this. And we're like, how do we do that? And so SREs, then, are they embedded in the dev team or they're a completely separate team but they somehow work together?

Yeah, they're a separate team, but they're a lot closer than you would get in a traditional kind of dev and ops organization. So as an example there, we want you to automate this because part of your handover to us, we expect to have an automated release process. If the dev team say, well, we've never done that, then the ops team would just say, well, tough. Spend some time learning. Whereas SRE might seed someone from the SRE team to work with that product dev team to get the automation up to scratch. Or they might flip it the other way around and get someone from the dev team to sit with SRE for a while, see how they do things on other projects, and then feed that back in. It's a much closer relationship. It's not just about automation, it's also about even as big as architecture. If a product dev team do a small kind of prototype project and it works really well and they want to put it live, maybe an SRE team will look at that and say, look, this architecture isn't going to scale. We're expecting this thing to go through the roof, so we're going to spend some time working with you to re architect it and together we'll get this thing ready for production.

Awesome. And you had a really cool slide that showed overlaps between dev teams and the SRE teams. And it looks like there's a lot of communication because one of the bullet points was consistent tools. So who drives that? Is it both teams talking to one another, deciding on what tools? Is it the SRE team saying, this is the tools we'll use and developers, you need to consume these tools?

I think it's partly a conversation that's coming from SRE because we own this stuff in production, we are the team who are. Our responsibility is to keep this stuff running. These are the tools that we know work and if we're all aligned, then the handover is much easier. We're working on the same things together. We can hop from one team to the other and we're familiar with what we're doing. But also, I think it evolves a little bit from whatever's going on in the project. If it's a modern cloud native project, then the product dev team will be using Docker and Kubernetes, and maybe they're already doing stuff with monitoring, like Prometheus, all the stuff that you see in all the modern presentations. And then the SRE team will say, yeah, great, we're aware of all that stuff. Like this stuff. We know this better than you do, but that's fine because if you've got the basics of it, then we can start sharing our tools. It just becomes so much easier than in the old days. The dev team had no idea how the OPS team actually released stuff. The OPS team, I've worked on projects where we built. In my background as a Net guy, we built these MSIs, we handed them to the OPS team. We thought, great, we've got all this configuration in the MSI and the OPS team just threw them away. They took them apart, they took the binaries out that they cared about and they repackaged them. We were putting all this effort into making it easy for them to deploy and. And they literally chucked it all away. When we found that out, then we started working together and it was a lot smoother. But now if you think the components of how you deploy your application, if it is a modern platform, then you've got Docker files that package up your app, you've got your kubernetes, manifest that, deploy it. All that stuff lives in source control. It can be jointly owned, it's jointly worked on. It just makes that we've got one set of tools we use for everything much, much easier.

So I've spoken with a few people so far. This is a newer podcast. I have another podcast I've been doing for six years, but this performance set reliability, I'm only in episode 25. So the few people I've talked to though, they've never talked about this term that you bring up in your course and that's toil. So what is toil?

Toil is the term that Google give. They also call it grunge work. They're kind of mundane. I come to, I'm an OPS person. I come to work in the morning, I log in, I get a ticket that says this machine is run out of disk space and I go and log on and I delete a bunch of old log files and I'm done. And then next week the same thing happens. It's mundane, repetitive, low value work. So some work isn't necessarily worth automating. So you can't automate everything because some stuff is just not worth the value. If I spend five minutes a day doing some boring task and the fix to automate it is going to take 50 hours of effort, it's kind of hard to justify that. But what they talk about in SRE is that most of TOIL is interrupt driven. So it stops you working on the higher value project work, which is improving the product, improving the workflow and improving the runtime in production. It stops you doing that. You have to stop what you're doing, do some sort of little hacky fix and then away you go back to trying to concentrate on what you were doing before, but you've had a context switch so you lose your focus. All that sort of stuff. A big part of SRE is trying to isolate those toil driven interrupt driven operations and have those done by the people who are on call. If you're on call, maybe there are two people on call, a primary and a secondary. They deal with all that stuff and then the rest of the team can work on the project stuff, which could very well be someone saying, look, we're getting this ticket every day about this server running out of disk. So we're going to do a very small project which will automate fixing that problem. And maybe the first release of that fix will just be a script that runs every day and deletes the old logs. Maybe the ultimate fix is going back to the product dev team and saying, look, your logs, we don't use them anyway. They're no value to us. Can we have at very least some configuration settings so we can tune that stuff down so we don't just keep deleting them? This idea of TOIL is this low value work that takes humans to do it when really a computer could be doing it. And then what SRE does is it gives you the space to have these projects that address that toil, to get rid of it and to do the high value stuff.

So maybe that's another difference from my experience with DevOps is the team would just have a huge backlog. I'd say we need to get this automated be on a huge backlog. So sre, it's almost. Is it baked in then that TOIL is going to be part of. Let's stop and let's have someone actually focus on this. Or is it put it on the backlog and then we choose and it goes through this whole huge cycle?

Well, there's two things. There's the acceptance that TOIL just happens because you can't automate everything. But then there's this also there's this guarantee that the team won't spend more than a given amount on toil. So in Google they use 50% and that's known throughout the company. It's part of the job spec. It's part of the job advertisement. If you've got a new starter, you won't spend more than 50% of your time doing these mundane, repetitive things. The other 50%, you'll be engaged on longer term projects to improve the product all around. And it doesn't have to be 50%, but having that figure as part of your organization is really good because that stuff's kind of boring anyway. No one wants to do that all day long. And it's good to know that you're making a more useful contribution by doing these Kind of higher value engineering things. So there will be toil, but there will also be this. Whether it's however you capture it, there'll be a backlog of projects that say, you know, we want to reduce the toil. We've done some analysis and we know that X amount of time is spent doing these top three jobs and actually if we can invest a bit of time then we can address those, automate them and then, you know, free up a whole bunch more time. Because not only do we, do we save the time from doing the toil, but we've got more experience in automation, we've got more stuff going on behind the scenes. So we freed up some of our budget so we do even less toil. So it's a kind of healthy cycle that keeps reducing the amount of non high value work.

I agree with you. Now everything can be automated, but I love automation. So what tools do you use or to monitor or to create this automation?

So that isn't prescriptive because of course that's going to be different for every project. Yeah, absolutely. Personally, with my background I have everything running in containers and I dockerize everything as far as possible. But it's going to be, I think it's going to depend a lot on your, obviously your tooling that you've got. So an older project that I was working on, you're not necessarily going to be able to run all that stuff in containers and kubernetes. You need to work with what you've got. But I think the principles of how stuff runs now in a more modern platform, the idea that your monitoring is surfaced in a dashboard which shows you meaningful statistics, statistics and those stats are used to drive alerts that tell you when the system's about to go down rather than, you know, that it did go down and it's been offline for an hour. Those, those principles that say this is what we're trying to get to. And again, SRE is quite prescriptive about, about what the, what alerting looks like. They have this idea of service level objectives which is, which is a goal of, of the availability of a system which allows for the fact that there will be some downtime because downtime tends to happen when you do a release and if you want no downtime then you don't do any releases and people aren't happy with that either. So this idea that we know what we're targeting for different aspects of the system, whether that's the latency for a response, the number of responses we can handle within A given period, whatever makes sense for your service, and then constantly monitoring with service level indicators that tell you whether you're meeting your objectives or you're about to lose your objective, or there's an alert because something's gone badly wrong. So I think that the goal of understanding where you need to be is what's prescriptive there. How you actually get to it is kind of up to you. So if you had a free choice of stack, then tools like Prometheus and Grafana do this stuff really nicely because a lot of the practices emerge at the same time that these open source tools were getting popular. So a lot of the way people use those tools is based on these sorts of ideas.

You didn't mention SLOs and I've heard of SLOs and SLIs, but curious. I didn't know what's the difference between an SLA? I know what SLAs are, but this seems more concrete.

Yeah, yeah. So I mean, to me an SLA is just this weird, vague single number that says, you know, our SLA is 95%. And you're like, well what?

And how are we measuring that? And what happens if it isn't 95%? You know what happens next? SLOs are much more broken down. So you wouldn't just have one big number that says, here's our system availability. It's broken down into those things that make sense. Kind of those examples that I gave you, like, how many requests per second can we serve? What's our latency? And it's not a fixed number. It's like the 95th percentile of requests will get a response within 500 milliseconds. It's a sensible ceiling that says we're not trying to aim for 100% availability, we're not trying to aim for 100% responses within a second because there are always outliers. And it's incredibly difficult to test that and be certain that you're matching those levels unless you put insane amounts of engineering into it. So unless you want to dual run in multiple clouds to get a fifth nine that you don't really need, then it's a fair assumption to say that there's going to be some downtime. If you're running in the cloud, then the level of availability you get for your hosting platform is probably only three nines. So if you're trying to go up to four or five nines, you're adding an awful lot of engineering on top of that. So the service level objectives are these kind of broken down there May be half a dozen of them for a product or there may be 10 or 12. But they're meant to be user focused. So it's not about how much CPU are we using and how much memory are we using, it's about what's the user experience here. Is the user really going to notice if our Homepage responds in 500 milliseconds instead of 700 milliseconds? If it costs us an extra 10% of our cloud spend to shave off those 200 milliseconds, is that worth having? It's things like think about the user experience. If there's no response to a request that comes in, the chances are that the user is just going to hit F5 and refresh it. Do we really need to worry about making sure that 100% of requests get responded to because we only own a certain amount of that chain anyway? When the request hits our data center, then from the load balancer onwards we own that and we can monitor that stuff. But beyond that, we've got the whole wide Internet and we've got the user's cable coming into their house and their modem and their wi fi and all those things are going to be way flakier than the stuff we own. So by the time it's come to us, it's already gone through what's all the things that might happen to it into the point where we can control it. So let's have these objectives that make sense from a user perspective that we can monitor and that's the important thing. So we need to be able to track the fact if we've said we're going for 95% of requests will get responded within 700 milliseconds, we need tracking. So we can definitely say whether we're doing it.

Interesting about the user experience, part of it. In performance testing, we call this the four horsemen when we monitor cpu, memory, network and disk. But do you have something to call the four golden signals? Are the four golden signals more UX or user experience type monitoring? No.

So I mean the golden signals are kind of the basics of your monitoring, if you haven't got anything finer grain. So you know, it is about latency and saturation, things like that. So the stuff you can fairly easily monitor with any system, you know, you get this idea of those examples you've given are great ones memory and cpu. Okay, so if I do a new release and memory consumption has gone up by 10%, like do we need to back that out? Maybe if everything slowed down, but actually if what we've done is we've reduced response time by 50% by doing more processing. Then the fact that the CPU has gone up by 10% is a good thing because it proves that we're actually driving down the thing we care about, which is response time. On their own. Those are kind of infrastructure level metrics that tell you how hard the machines are working. Doesn't give you the full picture. The SLOs are really about drilling into how you want your product to perform, how you want your users to perceive that performance. But before you, that might be a chunk of work to get to that point where you've got those half a dozen meaningful metrics for your product. And the golden signals, the latency and the saturation, they're a good starting point. If you can't, before you get there, at least start tracking these things, start looking at these things so you know how many errors there are. You can start seeing if things spike. You get a rough idea of how well things are performing as you're building out your more detailed metrics.

So I'll be honest, I thought SRE sounded like a really cool role, but I keep hearing about this part of it on call and just brings back memories of having to carry a pager back in the day. So what is on call? What does that mean as an sre?

Yeah, I mean, it's still that. So, you know, this is one area which is slightly less prescriptive because obviously this depends on the size of your organization and the level of availability in your SLOs that you're trying to meet. But it's effectively, yeah, there's a person on call and anything significant comes into them. So that person has got the pager. That person needs to be responsible, to be out of hours, to not be at a theater with a glass of wine in their hand. They need to be near a laptop, near connectivity to the Internet all the time, that sort of stuff. It's a big deal because it's that situation where if we suddenly lose the system, I'm the person who needs to get it up and running and I'm the person who's going to be getting paged by the CEO saying, where's our revenue generating system gone? But what is different about SRES for a startup? Because that's kind of classified as part of the. There's this whole structure of how you deal with on call because it's recognized as being a stressful, a stressful role. It's shared among the team. Ideally you'd have more than one person, so you'd have a primary and a secondary. And then there's this idea of, you know, we need to understand the limits of what we're supporting here. So we're not going to have a team of people who are getting paged at two o' clock in the morning every day and having to work, you know, 15 hour shifts to get through critical problems. Because if we're in that state, the. The system is probably not fit for production. And in the worst case, the SRE team say, we're not looking after this, and they hand it back to the product dev team.

It's a nuclear option. But the SRE team have that power to say, we're not supporting this because it's doing us too much damage. But you saw this notion of On Call, but you have all this structure behind you. So SRE takes this incident control situation, and they say, if there's a major production incident, this is how we deal with it. There's an incident commander who's going to look after it. There is an operations lead who is looking into the investigation, and there's a communications lead who is dealing with keeping the stakeholders informed. Because one of the worst things about On Call, of course, is you get this issue, you're trying to debug it, you're trying to trace it, and people are firing questions to you all the time saying what's going on? And then you've got other people on the dev team who are firing up suggestions that may or may not be helpful. And the fact that no one's managing it makes it even worse. What SRE does is it puts this structure around there. It says there's a single point of contact, which is the incident commander they own getting the incident resolved. The ops lead owns the investigation. They can bring more people in if they want to, but they report to the ops lead, who reports to the incident commander. And again, the comms lead is in charge of distributing information, whether that's public or to internal stakeholders. They can bring more people in, but there's this chain of command, which means there's a single point that if the CEO wants to find out what's going on, there's a living document that describes what's happening, listing the personnel who are in charge at the moment, and they can kind of reach out and deal with that. So there's still this kind of scary thing about being on call that is potentially a highly stressful few hours of your life, five days, a month or however, however it pans out. But you've got all this support behind you, which is there to. To make sure that you know you're following guidelines, that everyone agrees and everyone knows what they need to do if things go wrong.

So there's so much more we can cover. And you do cover it in your pluralsight site, Reliability, engineering, the big picture course. But I thought maybe we could wrap it up with postmortem. It seems like an important piece, and you brought it up earlier, that makes it really differentiate itself from DevOps.

Yeah, I think DevOps teams do this, they do this root cause analysis, but.

It'S part of the culture and they will have their own ideas about how they do it and how much weight they give to these things. But postmortems are kind of a core part of sre. You're expected to do a postmortem if you. Not every on call. Excuse me, let's try that again. Not every on call issue will become a fully fledged incident with a commander and Ops lead and that sort of stuff. But everything that does become a recognized incident should have a post mortem afterwards to describe exactly what happened. And again, that's a very structured document. So it says there's an executive summary that describes in a few sentences what happened, what the impact was, how it was resolved. There's a problem summary that goes into a bit more detail about the technical level, what was the underlying root cause. Then the more useful part is actually there's a bunch of action items that said we need to do this, this and this to stop this happening again. There's a section for lessons learned that say to do with either how the problem started or issues with the investigation that meant it took longer than we thought, or whatever things that we can take away from this part of that mantra about Agile from a decade ago, inspect and adapt, look back at what you did and try and improve it for next time. And I think those are really useful things. There's a full timeline in your postmortem to say what happened, what steps we took, what worked and what didn't. But I think the really important stuff is that these are the action items to make sure we don't get something of this severity. Again, here are the lessons we've learned and then it all gets published. So ideally it's available to anyone in the company to look at that stuff. And at Google they try and encourage this culture of saying the post mortem is a really important thing.

It'S the closure of your incident and it feeds back into the project and into other projects to say, this is what we did either here's some really good ideas that you might want to bring into the next incident that happens with you, or here are some things that went badly wrong that you need to look at to see if they might impact you as well. So Google do this kind of like post mortem of the month thing where everyone shares it and you read them, you try. And when it's your turn to write your postmortem, you put the detail in it to make it a meaningful document. And that often doesn't happen, not just with DevOps, but with traditional ops. There's a problem, everyone jumps on it, you put the fire out as quickly as possible, and then everyone walks away and goes back to their day job and you've solved the problem, but you haven't done the next thing, which is learning the lessons to take them forward. So, yeah, that's why it's such an important part of incident resolution.

Okay, Elton, before we go, is there one piece of actual advice you can give someone to help them with their SRE efforts? And what's the best way to find or contact you?

Oh, okay, cool. So, I mean, the best thing I think, which is something we haven't really talked about, is this notion of an error budget. So if you're, if you're looking at one thing to say, what can we get out of sre, which is going to give us something quantifiably different from what we're doing now. It's this notion of an error budget which you can, you know, you can read about it, it's very well detailed. The idea is, it's that explicit acceptance of the fact that products just don't work 100% perfectly all the time. And it's that balance between saying, if you want more frequent releases, releases are when things go wrong. So if you want more frequent releases, you have to understand we're going to get more, more regular failures. And it's this, this error budget is your contract between the SRE team who manage the product, the dev team who build the product, and the business people who own it to say, if you want lots of releases, you need to accept that we're not going to be as reliable as if we only have one release a year. It's this concept of, it's what gives you the ability to say, no, we're not doing any more releases this month because we've used our error budget. Fixing the issues with the last release, that's a really powerful thing. If you want to find out more stuff that I'm doing and the stuff I've talked about today is the stuff that interests me, like SRE and DevOps and Docker and Kubernetes, all that stuff. Twitter is where I usually publish my stuff. So Elton Stoneman at Twitter I've got my blog, which is the horribly difficult to Remember address of blog.sixyed.com, which is s I x e y e d dot com. So I blog about this stuff. I've got well over a dozen courses on pluralsight. You can find all that stuff there. And yeah, Twitter blog pluralsight, you'll find something there, I hope.

Thank you Elton for your SRE awesomeness. For links to everything of value we covered in this episode, head on over to testguild.com P30. And while you're there, make sure to click on the Try Today link under the Exclusive Sponsor section to learn all about SmartBear and their awesome performance testing tool Load Ninja. And if you haven't heard already, Registration for the 3rd annual PERF Guild is now open. Make sure to head on over to perfguild.com and get your Early Bird tickets. They're the lowest price I've ever made these tickets available, so you want to get a hold of them before the price goes up. So that's it for this episode of the Test Guild Performance and Site Reliability Podcast. I'm Joe and my mission to help you succeed with creating end to end full stack testing awesomeness. As always, test everything and keep the good. Cheers.

Thanks for listening to the Test Guild Performance and Site Reliability Podcast. Head on over to testguild. Com for full show notes, amazing blog articles and online testing conferences. Don't forget to subscribe to the Guild to continue your testing journey.