SRE 2021 Report with Leo Vasiliou

Welcome to the Test Guild Performance and Site Reliability Podcast where we all get together to learn more about performance testing with your host, Joe Calantonio.

Hey, it's Joe. And welcome to another episode of the Test Guild Performance and Site Reliability Podcast. Today we'll be talking with Leo all about a new report I found on Site reliability engineering for 2021 that he recently published. Leo is a former longtime IT Ops infrastructure practitioner. He still carries battle scars today, which is part of the reason he is now part or works in product marketing and research to help drive relevant SRE DevOps business type decisions. He was the primary analyst and author on the fourth Annual SRE Report that we're talking about all about in this episode. Site Reliability Engineering has been a big topic in the past few years and I think this report is really full of really interesting insights. So you don't want to miss this episode. Check it out. This episode is brought to you by SmartBear. Listen, load testing is tough. Investing in the right tools to automate tests, identify bottlenecks and resolve issues quickly could save your organization time and money. SmartBear offers a suite of performance tools like Load Ninja, which is a SAS UI load testing tool, and Load UI Pro, an API load testing tool to help teams get full visibility into UI and API performance so you can release and recover data faster than ever. Give it a shot. It's free and easy to try. Head on over to smartbear.com solutions forward/performance testing to learn more. Hey Leo. Welcome to the Guild.

Joe. Thank you very much for having me. It's a pleasure to be here today.

Awesome to have you. I think I really botched your bio. So before we get into it, say that I missed in your bio that you want the Guild to know more about.

No. That was a wonderful introduction and I love the comment about the battle scars. It really rings true actually.

Awesome. Yeah. So I'm really excited about this. I thought we'd just start off by diving in really high level on the fourth annual SRE Report you just released on sre. So I guess you know it is the fourth annual one. What is the SRE Report? What's the main objective you have by releasing it every year?

Well, when we talk about the objective of writing a piece such as this, the goal for us is for this report to be the most data backed report of its kind to help drive decision making. That's really the long and short of it, Joe.

Nice. So is there anything? I guess we'll we'll find out. But is anything that really stood out between this report compared to the other three or any, any areas that you really want to focus in more on this that may have changed over the past few years.

Well, last year's report was a two for one because we did the initial survey at the beginning of the year which was a pre Covid and then everyone started working from home. So we actually did a supplemental survey after everyone was a few months into Covid. And so this year what we wanted to do in this, the 2021 report was we want make sure that we established a good set of baseline data for comparison. So if you think about the core tenets of SRE like the dev versus ops split, the amount of time on call and the level of self reported levels of toil, then the other thing we wanted to do this year was look at one of the trends and try to dissect it. And this year's trend we looked at AIOps specifically and then we tried to close with another general theme of the report, which is the it to business relationship. And we took that piece a step further this year by offering a reproducible path to value because we felt that was something we had never done before was in addition to just talking about our findings and say you need to show value to the business. We actually wanted to offer a way for people to repeatedly do that.

Very cool. And I guess I do find that as an issue that a lot of people don't know how to actually demonstrate value or explain the value to their organizations.

You know, it's an interesting question and when we kind of dig into that a little bit, I'll answer in the form of the answers we receive to particular question types. So for example, when we very simply ask what drives successful SRE implementations, answers like reducing incident times and reducing time between failures dwarf other answers like expanding the business into new markets. So it's within that context that we make statements about the gap that exists between SRE teams and the business, especially when they're having business conversations. So absolutely interesting.

Yeah. So I thought maybe we'd dive into the key findings and then see where it takes us. I guess the first one was about toil lower around the world, which seemed kind of unexpected. So a little bit about key finding 1. Why do you think TOIL was rated lower this time than before?

Arguably one of the most fascinating findings. And if you don't mind, we'll just kind of give the quick definition that Google uses. Right. The type of work tied to running a production service that's manual, repetitive and devoid of Any long term enduring value. And what's fascinating about this data point was not only was the average value lower by 1, 5, 15%, but it was lower across the entire distribution of the data set, across all of the percentiles, across all of the geographies. Right. North America, EU and Asia. And so I don't know about what everyone's feelings or sentiments were, but I know when I had the opportunity to work on the report this year, having been stuck in my home for the past year and a half, there was just something that felt, felt different about working on this report. Like it just felt special. A way to connect externally, maybe as a way to replace all those lost social interactions. Which makes us kind of wonder, are operational elements like toil related more to social components than we initially thought? Joe, the most fascinating conspiracy theory here is when we run this survey again next year, if people are starting to return to work or some type of hybrid work environment, will those reports ordered levels of toil rise again? That's going to be the interesting question to try to answer in next year's report.

Absolutely. And I know, well, I won't say this, I was to say Google is making a push to have people come on site now rather than work remotely. So it would be interesting to see that for sure. Other thing I thought we'd talk about is besides toil is you had some things underneath toil was a lot of times when people think of sre, they think of automation automating all the things. But I think you mentioned earlier that it's not necessarily tied to business or why, why is it important to automate all the things? And why do you think it help helps people actually focus on other things that maybe could be more beneficial to the business?

Well, I would start the answer to that question by first talking about the concept of establishing some type of baseline so that you know whether some of these conditions, if we can use that word to summarize what we're talking about here, whether conditions have objectively changed, that has gotten better or gotten worse. So the idea of automating all the things is really about freeing up time to work on more value based activities. And we like to say. Well, we don't like to say, but we would generally describe when we think about the SRE Persona, they are double struck with challenges because they are essentially doing two jobs in one. There are operational activities and so they need to increase the efficiency of those operational activities. Then there's development activities and other transformational activities like migrating to the cloud. And so they need to mitigate the risk of those transformational type activities. So the idea of automating all of those things is so they can work on more valuable activities and mitigate levels of risk. Those are the two key drivers for why we want to automate everything.

So I guess the breakdown I saw you have a nice SRE activity breakdown chart in the report itself. Curious to know, was there anything that stood out to you as what people are spending time on as opposed to what maybe they could be better served by focusing on other things.

You know Joe, one of the reasons we asked the activity breakdown question year over year is because we want to ensure quality answers for the efficacy of the report. So we make sure to include operational activities like responding to incidents, but also development activities like developing capabilities. So we were happy to see that those types of activities did range in the top half of the breakdown. But a couple of surprises that stood out to me was I thought the activities of being involved in release roadmaps would have been a little higher, especially if we're talking about product releases and then this idea of chaos and load testing. I was a little surprised to see those toward the bottom of the list. I will just make one special call out to the training activity which is providing training on third party platform capabilities. Because we actually made the decision to explore the use of multi platform implementations this year. You know, given the prolific rise of the use of third party platforms, we wanted to explore like our people taking the capabilities of those platforms, developing them and offering them as internal capabilities for other teams to reuse as a way of achieving level of scale. So that one I think is also where of its own call out now.

Does that have to do with the multi provider strategy that you mentioned later on in the report how a lot of teams don't have a multi provider strategy in place to handle that situation.

So if you consider these two stats 90 90% of SRE organizations use at least one third party provider, one third party platform like DNS CDN and then 64% of those organizations have a multi provider same service implementation. So multi DNS or multi cdn. And we feel that this has to do with the org structure question where SREs are generally per the Google definition, aligned with business products or services. So we definitely think that has something to do with it, an underlying theme because when you talk about different teams using different platforms with their own set of APIs and capabilities, that's going to prevent them from achieving levels of scale.

Nice. So interesting about that as well is I think around the Same question was how teams are organized centralized versus is decentralized. And I think you had another option. Is there a typical best model for sres that you've seen or is it just it all depends and based on what the organization is doing?

Well, the shortest possible answer is there is no one best model. The bit longer answer is we wanted to do some correlations to things like number of SREs and number of employees. So the choices for that question were are you centralized, are you decentralized, or do you have some type of hybrid? Hybrid mix. And if you look at the answer the data set in aggregate, it might give the optical illusion that centralized edged out decentralized by a small margin. What was fascinating though was when we broke down and looked at the structure based on the number of SREs and number of employees, the organization structure shifted from being centralized to being decentralized. So while that was not necessarily surprising, it was rather a way to back up the theory that larger and more complex orgs got the more decentralized they become. So just to bring it back to the original answer to your question is there is no one size fits all model, especially for companies that try to compare to Google's definition. Which brings us to the overall theme of the report of Free to be an sre.

Right. And interesting, you also have that as a question or you have that as a spotlight on does Google's definition of SRE matter? So why was that there? Google's definition? Have you found a lot of companies because all their needs are different, necessary, so they don't line up what Google defined as an sre.

You know, it's an interesting question, Joe. And what we decided to do was this is the fourth annual SRE report. In the previous three reports, we tried to have a disposition of matching Google's SRE definition. But this year we said, you know what, maybe we need to take a different approach. And instead of worrying about how Google defines sre, that it's okay if you don't perfectly match their definition. Which is why we wanted to also give the baseline so people could see where they fell on the overall curve. But that was kind of the flip flop we did this year versus the previous three reports.

Cool. And another thing you called out in this key insight was SRE meets platform ops. So why is platform ops important?

Platform ops is a way for orgs to break through what we talked about as this looming scalability ceiling. As a former IT and ops practitioner before this prolific rise in use of cloud and third party platforms, I remember when I Had to fight for every square inch of power, cooling and rack space. Getting a server in the server room, then going through the daunting procurement process, waiting weeks if not months for it to come in and then when it finally did come in, arguing with the network team over getting ports open in the firewall. Great. The shift to cloud removed a lot of those operational constraints that I just described. So the next kind of phase of this is because orgs were structured by business, product or service. Those different teams are using different platforms underneath and those different platforms have different interfaces and instruments for developing capabilities. So the idea here is because of those stats that we talked about earlier, right? 90% at least one third party provider, 64% using a multi same service provider implementation, that there's a higher level of scale that can be achieved if you dedicate engineering and software development resources where platforms are the primary dimension, to bring them front and center develop capabilities even though the underlying instruments and interfaces might be different and offer those capabilities as part of an internal marketplace that can then that can then be reused by many different teams. That's the core concept of this platform operations idea that where is being talked about here as a way to achieve higher levels of scale. So that is an absolute critical piece to this year's report.

So has that been a roadblock then? Because they're not reusing certain services that are available across the organization, they're reinventing the wheel or is that what the holdup is or what's stopping this from scaling?

Not so much a roadblock, but this has to get back to what we were talking about just a moment ago where everybody was baselining against Google's definition of sre. So if you think about that, the SRE teams, right? If we just talk about the shift to DevOps where the old waterfall model went bye bye and teams were then assigned horizont across the entire development life cycle. They were assigned horizontally by business, product or service. So all we're saying here is to take a tactful look at instead of having everything be assigned where product and service are the primary dimension, to look and see where it makes sense to assign platform as the primary dimension. So said another way Joe, if you've got 10 different teams using 10 different cloud providers and 10 different SREs developing 10 different sets of capabilities across 10 different sets of APIs, you can centralize that. Maybe that's what I should have said.

Yeah, it's just like software development. It's the same thing. When you work for large enterprises, a lot of people aren't Getting the reuse that they could just working across their organization for sure. So I guess another piece that out of the report that was kind of stood out to me and based on the pre show it sounds like it stood out to a lot of other people was the shift towards AIOps is slow. Now you said it's slow. It's not. You didn't say it's not happening, you just said it's a slow process. So I was curious to know why do you think it's slow? Come from an automation testing background and AI is like in every piece of literature for anything. They just throw it in there almost. So it's almost like a marketing term. So I don't know, is that the same thing with sre? Is AI overhyped? Is it a real thing? Big question, but just curious to know why what your thoughts were on that finding that AIOps is slow or the shift towards AIOps is slow?

Joe, you hit the nail right on the head when you said you wonder if it's a marketing term. You know several hundred SREs who took this year's survey might agree with you. We looked at at the number of monitoring data sources as well as a precursor for this section, we very simply asked what monitoring tools frameworks do you use? And AIOps was the fifth lowest on the list with a whopping 39% saying they never used AIOps. Then the next question we asked them was what is the received value from AIOps implementations? Gave them a 1-291-9-9 value scale and the sentiment nearly equally spanned the scale from 1 through 9. So now to get back to your question about why that is. Well we think it's because AIOps is so big and bloated and it's like what the heck does that actually mean? So our thoughts and our recommendation there was very simply to break down AIOps into smaller ability sets and be leery of some of them. So if a vendor's coming in talking about auto remediation or self healing, probably want to place a little more scrutiny on those sub capability sets and maybe look at things like event correlation again from all those different monitoring data sources or root cause analysis. So I absolutely do think there might be a little bit of a situation where the market hype around what AIOps promises is still a little too leading versus what SRE and other IT teams are able to implement with what they know today.

Absolutely. And it's a shame because with automation it's a little a little tougher I think for AI machine learning. But with this I think because you have all these logs in the two areas you just called out event correlation and root cause is perfect, I would think, for machine learning. So are there actually solutions out in the field that could, could help sres? And the other kind of whiz bang type of proclamations of how it can help is kind of overshadowing the real thing that could help them. Probably where they spend the most time in is root cause analysis and event correlation.

Yeah, I would agree with that. I would agree with that. I mean, I would probably still just go with the concept of breaking down into some smaller subset abilities there. And maybe how I would think about that was one, if you break down ops into some of those subsets like we were talking about, I would actually maybe go back to the SRE activity breakdown and then see what type of activities those broken down components could then be applied to. So for example, you know, you talk about incident management as one of the activities. Well think receiving pages and alerts where maybe the shift there is instead of a human having to take action, some type of AI, no pun intended, could actually take an action for you and then maybe automatically invoke some type of playbook. So maybe that's how I would think about it Is break down AIOps into its components and then see what activity SRE activities could then be applied to.

So the fourth key finding was observability must include digital experience metrics and business KPIs. Is that what we started off the call with, where there seems to be some sort of disconnect between the SRE activities and maybe what the other value they could be adding to the business?

Joe, the reason we wrote this section was we really wanted to kind of put the word of caution out there that and this applies to anybody, not just SREs. People that are working who fail to deliver customer value run the risk of being stuck in an operational toil rut. And conversely, businesses failing to recognize the importance of SRE activities and the work that they're doing run the risk of losing talented employees and competitive edge. And so we have constantly the ad nauseam talked about this it to business conversational gap. And we wanted to use this as a perfect opportunity to help bridge that gap. Because SREs are not one to be bound by tradition definitions of things. And so this concept of observability, which is probably more well known in the application, you know, in code level space, is a prime opportunity to take the key concepts of observability, right, look at outputs of a system and infer the state of its inputs and expand the boundary to include business and experience metrics as part of that conversation. That is the key way to think about this section here as a prime opportunity to bridge that it to business gap.

In this section once again you have really good chart on this is called monitor data usage drivers and 66%, a whopping 66% I guess spent most of their activities augmenting, troubleshooting and root cause analysis. And as you go down it's more comes more about the customer experience almost. Was this surprising?

Unfortunately no, it's not surprising. It's just something that we wish would change at some point. Right. It's not surprising. We wish that the idea to, you know, provide analytics to business teams or do a better job of supporting collaboration efforts. If you think about SRE is an implementation of larger DevOps philosophies and principles, the latter being all about communications and collaborations. At some point we would like to see this chart inverted where those are at the top and things like augmenting, troubleshooting are a little lower down.

Absolutely. And I don't know if this is from the reporters what you recommend. You have a successful SRE implementation drivers and it's broken down into percentages. So like how quickly we resolve incidents, the amount of time between features failures. Just talk a little bit about that chart.

Successful SRE implementation drivers was the question of why the heck do we do what we do and resolving incidents, reducing time between failures. Not surprisingly at the top of the list, expanding our business into new markets or understanding cause of social media sentiment. Examples of drivers at the bottom of the list. When we think about a way produce customer value, unfortunately the items at the bottom of the list are more important than the items at the top of the list and we just want to see those flip flopped. And so we said to ourselves, we said how do we get that flip flop? Which is why we came up with the idea of offering that value chain path of activities for SREs and IT teams in general to follow to consistently reproduce value for the customer. And that was really the fundamental IDE between that section, that piece in the report.

So those were the four key kind of areas you focused in on. Like I said, you go into much more detail than we have time to cover. Okay, Leo, before we go, is there one piece of actual advice you can give to someone to help them with their SRE efforts? What's the best way to find contact you or have folks get their hands on this report on the 2021 SRE report?

Well, Joe, thank you for asking. We write this report year over year with the purpose and goal of being the most data backed report of its kind. If people want to explore and read for themselves any of the findings that we talked about, then please visit catchpoint.com or simply do a search on Google for SRE report.

Thanks again for your Performance Testing Awesomeness. Links to everything of value we covered in this episode. Head on over to testguild.com P272 and while you're there, make sure to click on the Try them both Today link under the Exclusive Sponsor section to learn all about SmartBear's two awesome performance test tool solutions, Load Ninja and Load UI Pro. And if the show has helped you in any way, why not rate and review it in itunes? Reviews really do matter in the rankings of the show and I read each and every one of them. So that's it for this episode of the Test Guild Performance and Site Reliability Podcast. I'm Joe. My mission is to help you succeed with creating end to end full stack performance Testing Awesomeness. As always, test everything and keep the good. Cheers.

Thanks for listening to the Test Guild Performance and Site Reliability Podcast. Head on over to testguild.com for full show notes, amazing blog articles and online testing conferences. Don't forget to subscribe to the Guild to continue your testing journey.