Establishing SRE Foundations with Vlad Ukis

Get ready to discover some of the most actionable DevOps techniques and tooling, including performance and reliability with some of the world's smartest engineers. Hey, I'm Joe Colantonio, host of the DevOps Toolchain podcast and my goal is to help you create DevOps toolchain awesomeness. Hey, it's Joe and welcome to another episode of the Test Guild DevOps podcast. Today we'll be talking with Vlad all about establishing SRE foundations. If you don't know, Vlad is the head of R and D for Siemens healthineers. I think it's Health Imagineers, but it's healthineers team which does a lot with digital health platforms and reliability. He's lead for all of the Siemens Digital Health products. Previously as a software development lead, he drove the continuous delivery, SRE and DevREL transformations, helping their large distributed development organizations evolve architecture, deployment, testing, operations. He's done it all and also culture, which is the most important thing I've seen. Implementing these new processes at scale, which I think is really critical actually taking these processes and scaling them up. You're going to learn all about it. He is also the author of the book Establishing sre, a Step by Step Guide to Introducing Site Reliability Engineering in Software Delivery Organizations and we're going to have a link for that in the show notes. So you definitely want to check out this book after the interview as well. Really excited to have Vlad on the show. You don't want to miss it, check it out. This episode is brought to you by SmartBear. As businesses demand more and more from software, the jobs of development teams gets harder and harder. They're expected to deliver fast and flawlessly, but too often they're missing the vital context to connect each step of the process. That's how SmartBear helps. Wherever you are in the testing lifecycle, they have a tool to give you a new level of visibility and automation so you can unleash your development teams full awesomeness. They offer free trials for all the tools, no credit cards are required and they even back it up with a responsive, award winning support team show on your path to great software. Learn more@smartbear.com today. Hey Vlad, welcome to the Guild.

Hey Joe, thanks for having me. So really looking forward to the conversation with you.

Really excited to have especially authors on the show and SRE as well. I haven't spoken to someone in the SRE space in a while, so I'm curious to know how it's been evolving. But before we dive into it, I always ask authors this because it takes a long time, I think, to write a book. So why write a book? And why write a book on establishing SRE Foundation?

Right. So let me tell you a story of the book. So when we started with implementing the TeamPlay digital health platform, which is the platform I work for at Siemens healthineers, we were all coming from the on prem development world. So we were totally new to software as a service. And once we started doing this and after a couple of years of implementing the platform, we actually understood that we are responsible for operating the platform as well. And that was a totally new world to us because all the developers who are just coming from the world where the developers just develop and then they hand over the deliverables to the customer services and then they do the rollout of the software into the hospitals, because we are in the health domain. And once the realization came in that actually we are now responsible for operating the service all the way for the end customers, then we started experimenting with ways of doing this. And then at some point I learned about SRE at the Yukon conference in London and then brought it into the organization. And then we started adopting it. And then after a couple of years, we started publishing articles about how the adoption went. What did we do, what did we implement, and so on. And then I was approached by a publisher to actually just expand the articles that we're publishing online into a book. And then my first reaction was, well, actually, you know, I'm not a writer and I've got job to do, so I cannot do this. But then Corona came and with that the home office came and so that allowed me to actually spend time on writing. And that's how it came to be nice.

There are a few things I want to dive in. The first one is, how do you define sre? Is this a role or is it a technique? You know, a lot of times with the Sprints you have developers and they have to do testing, they have to do performance. Is SRE just another checkbox? Is it actually a role that needs to be there in your organization?

Right. So I think if we are to define sre, then we need to go to the origins of the discipline. And therefore I'd say SRE is a discipline. It's a discipline within the realm of computer science, like many other disciplines, and it allows you to run digital services reliably at scale. This is the definition that was given by Google when they originally wrote up different practices that existed at Google to operate Google into a book and called it Site Reliability Engineering or Abbreviated sre. So overall you will have your organization practicing SRE by having different roles, doing different jobs. And it's likely that you might have people whose role will be called sre, but that's not the most important thing, so that's not a necessity. So SRE is a discipline that you establish in the organization in order to run services reliably at scale and that then will entail certain tasks for different roles and might also invoke new role like SRE to be created in the organization.

So what I also thought was interesting is you work for a healthcare organization. I used to work for GE Healthcare and it made it a lot more difficult, I thought to implement like agile because of all the other extra things you need to take into place. So did you find that with SRE scaling it at scale at a healthcare, there were like maybe nuances or things that came up that other people may not think about when they're in different type of organization perhaps.

So I was actually positively surprised that we were able to introduce SRE in a healthcare domain within a medical device vendor actually relatively smoothly from the quality regulation, from the medical quality regulation point of view, actually it might be because we were the first one want to introduce software as a service into the organization and therefore there were no pre existing methodologies and processes to prescribe what to do. So we, what we created for us was then considered good enough for the wider organization. But actually I didn't encounter nearly any pushback or so when we were doing the introduction I think it was actually even rather the other way around. So the ability to be able to quantify reliability was considered very positively by quality regulation folks. So typically they haven't encountered something like this where you can actually to the point in real time tell what your availability is or tell what your latency is and so on. So typically because they also didn't deal with software as a service before because the company didn't have software as a service products, they didn't see something like this before because with on premise products that's usually a different world.

Nice, right? That's good to hear. So if anyone's listening and they work for a large enterprise or healthcare, I mean it sounds like SRE there shouldn't be a roadblock for you to implementing it because I think you go over the book's actually broken up into three phases. I think you have foundations running transformations and measure and sustaining the transformation. And so I think we're in the foundation type questions now and I thought like one of the chapters is on Challenges. And I was thinking maybe implementing in healthcare would be a challenge. But were there any challenges that did come up that you thought, oh, I better cover this in the book, because I know it's something common that a lot of people will face when they try to implement SRE within their organizations.

Yeah, definitely. So one of the most frequently asked questions before writing the book, during that writing the book and after writing the book was actually, how do you orchestrate organizational buying or sre? So there you go, you go into a conference, you learn about sre, you come back and you might be full of energy to try this, but then you understand in order to actually try this, you need to bring along all those different bodies because that's the methodology that actually unites them in service of reliability of the system. So you cannot just try it kind of in the very small. Of course you could do this on kind of some minute scale, sort of on your laptop, but you know that's not the point. Right. So if you really want to run an experiment with sre, then it already involves quite a few forms. And if you really want to kind of set on the road of rolling it out in the organization, although it might take a long time, but already first steps will require quite an alignment that's necessary. You need to bring on board product development and product operations and product management. So all those three pillars, they need to be in agreement that they want to try this. And you need to approach each party in a unique way in order to explain to them the potential benefits it will have if the methodology is introduced successfully. And of course, as with everything socio technical, as with any socio technical change, you cannot predict whether the change will blossom or not. Therefore, it must be run as a set of experiments where you progress based on evaluating the feedback from the previous steps. So you also cannot guarantee the results up front. Right. So you need to have a certain way of engaging the organization and therefore you also need to know the people well, who is having which view on things related to operations in order to understand where people are coming from and what they want in order to orchestrate the alignment.

Absolutely. And I believe you covered that in accessing the status quo, getting the whole team aboard, finding out what could be potential roadblocks. I assume. So do you almost treat it like a proof of concept or you just, you buy into SRE and you just move forward, or is it just all little experiments and you just build on top of each experiment as you go along, starting small, then increasing scope perhaps?

I think it can only work as an experiment, because the change is too profound. So the changes across the board in development because the developers now start spending some time for running the services, then the change is also in operations because now instead of operating the services yourself, you actually need to build up frameworks that would enable developers to do the operations, which is a total mindset change. Because now you suddenly become a framework provider, which you might have no experience with in the operations organization and in product management, you now start becoming part of the reliability conversation, which you actually might not have done before because you were focused on features and functions and customers. So the change is across the board and therefore it requires quite a bit of effort up front, which is totally non technical, which might be counterintuitive, that must be put in place before you can start running the experiments. And with each experiment you can get a bit more and more and more confident. And with some experiments you might get less confidence and you just went back a bit and then move forward two steps and so on. But I think something like this only works as a set of experiments where all the bodies involved learn over time about what we are doing and about how that positively or negatively impacts what we want to achieve.

Nice. So when you stop running that first experiment, you stop running the transformation. I guess that's the second part of the book. You have a section on coaching. How important is coaching and training when you're going through this process? Do you just make assumptions like, okay, I talk to everyone so I don't have to do anything more than that, or is it really hands on hand holding throughout the first experiments that you start running?

In our experience, coaching is absolutely, absolutely and absolutely essential for adopting something like SRE in an organization that's new to operations. Because in an organization that's new to operations, people come to the topic with the ideas from on prem world where the product management defines the requirements for the product. The product development then takes those requirements and implements them and then hands over the product to product operations who then do the rollout to the customer sites and take care of customer issues. So now once you start doing software as a service, you then change all of this. And with development you then start operating the services. With the operations you start enabling development, operate the services, and with product management you start from the beginning, define what the reliability objectives are and thereby you also then agree to the development, spending time on keeping the services, running the services to the service levels that were defined before. And you then get a live feedback loop from production where you see whether the defined service levels actually fulfilled or not. And that now starts influencing the backup prioritization and all that is.

Absolutely. So how do you come up with those SL, those service levels? I know you cover SLIs, SLOs, SLAs like, is there any.

Key takeaways? What does it all mean? What do you need to get in place to make sure you're not just coming up with some ridiculous type of measurement?

Right, yeah. So one of the core concepts here is the so called service level objective SLO. It's actually written down in Google's SRE principles that you need to manage by SLOs. Manage by service level objectives. So in order to be able to manage my service level objectives, you need to define some first, right? And you need to have some fair idea that what you define actually makes sense. So in order to come up with good SLOs, you need to bring together again product operations, product management and product development for a particular suit. So that means you bring in the product owner of the service, then some developers of the service and somebody who ran the service in production. You bring them together and then you define the first of all the most important user journeys, which again might be counterintuitive because that's not technical. Right? So you're a product owner on board and that's a good person to have in the conversation when you talk about user journeys. So then once the most important user journeys are clear, you then say, okay, so this user journeys, they typically generate these call cadences between the involved services. So that service calls this and that service then puts a message there and then some other service takes that message out of this queue and then something happens and then you go to the data layer and so on. So once you understand this at this level, then the next step is to say, okay, so typically the most important foundational thing in terms of reliability is availability. Because if something is not available, then it's just down and you cannot do anything. So therefore usually availability is a good place to start as the so called service level indicator or sli. So typically it's kind of very foundational. So then you start looking at your cold chains between the services that you've identified before and say, okay, so this thing really must be reliable because if this thing doesn't work or is down, then nothing works. Right? So therefore let's assign some higher availability to this. And at that point, if you're doing this really for the first time, then you just might have on your mind all these 99.9999 and so on. So you might be inclined to to set reliability for this in a very very. In such a way that the availability is very, very high. So many nice. And then once you've done that process, then you'll have the definition of availability for different parts of the system. So then you will hand over these definitions to your newly created SRE infrastructure that can track the fulfillment of those SLOs. In that case, then availability SLOs. And then you will start getting feedback from the infrastructure giving you a hint when you don't fulfill the SLOs. And then typically what happens to a team that's new to the party is that they see that they've actually defined the SLOs in such a tight manner that the system cannot fulfill it. And this is kind of the first feedback loop and this is something that you need to get into the habit of reacting to. So you put something into the SRE infrastructure that you think your system will fulfill. Then the SRE infrastructure starts tracking this and telling you that you probably don't do this. And then you go back and think, okay, well, do I actually really need to fulfill this or can I back out a bit and lower my reliability threshold without impacting the user experience of my users? From then onwards, this is a constant feedback loop. So, right, so that feedback loop goes from production again into the trio of product operations, product management and product development again, feedback loop, and again that goes into that trio and they decide what to do and so on. And this is from now onwards a never ending story. At the beginning you need to iterate faster because you are new to this, so there will be lots of feedback to react to later on, things will stabilize and you don't need to. You react as frequently to this. But this is kind of the fundamental way of running services under the sre.

Nice. So once you have this all in place, you're starting to get those feedback loops. The third part of your book goes over measuring and sustaining the transformation. So I'm focusing on sustaining the transformation. I don't know, like, if you ever done anything with test automation, a lot of times it starts off really well. You get the first few tests automated, looks like they're running, and then all of a sudden they start failing. People start not paying attention to them and then the automation effort just dies. I don't know if the same thing with SRE happens. You get some feedback loops going and do people, is there anything that stops it from sustaining it? Do people, if they start ignoring it, like they ignore like a flaky build or something, that isn't really a bad build it's just something wrong with the code, with the process, not the code. Do you find those type of things as well with the SRE type process?

So could happen. But what I found is that because from the beginning you involve those three product management, product development and product operations. So it's kind of not. If you started this way, then it's not a thing of a single party, right? So it becomes a thing that was born by the trio, so to speak. And therefore it's got less of a chance to just die down like a red pill that only the developers need it. So on the whole, though, of course you need to put in place mechanisms in order to keep this alive, because in a large organization you've got many initiatives going on and they all compete for the attention of the same engineers. So you need to have some means in order to tell the organization what's important, what's still important, and so on. So to that end, a couple of things can be done. So you mentioned at the beginning a role sre, depending on the setup, that might make sense to be introduced because now you put really people who are operating the services the SRE way, so it might make sense to actually call them SREs. That becomes something more official now because we've got even a role and some people can be either part time or full time SREs, depending on the setup and so on. So that's one. Then another thing that can be done is once you've got several teams running services according to Fuel as a reway, then you actually want to establish some cross pollination between those teams so that they talk about the best practices, so that they help each other and so on. So then you introduce something like community of practice. And if you want to become a bit more formal about this, then you might want to run it as a real community with real community leadership and some charter, what they want to do and why they are coming together and. And that gives them again another reason to talk about this, to jump together and not to drop the ball.

So then you can also, if you've got some kind of continuous improvement framework going on in the organization which already has got some dimensions, like for example, you mentioned testing. So maybe as part of this you look at your say, test failure rates and so on, something like this exists, then inserting another dimension there, like reliability is a very good idea because the organization is already in the habit of continuously improve in certain dimensions. And now a new dimensional liability goes into this. And as part of this you can then again have an ongoing conversation Whenever you talk about continuous improvement, whether we are improving continuously, also on reliability, because now we are having SRE as a methodology underpinning the way we work there and so on. So this are the ways that you can use in order to ensure that SRE becomes something that is self sustaining in the organization because of the things that you put in place.

Nice. So you mentioned SRE marketing funnel. Is this part of the SRE marketing funnel? Is this completely different? What is the SRE marketing funnel then?

That isre marketing funnel that's in the book, that's at the beginning in order to orchestrate the organization organizational buy in for sre.

Okay, gotcha.

Yeah, so that's where at the beginning you see an idea that this might be something for us. And then you take that idea through several kind of steps, which is a funnel where at the end you achieve the alignment.

So you know, I guess in the last process, then sustaining the SRE movement. I think you have something on SRE and CD indicators, right? Do you merge SRE with your CICD efforts? What did you mean by that particular part of the book?

Yeah, so actually when I wrote this and also afterwards when I was working with the continuous delivery indicators, so DORA indicators basically and SRE indicators, which is what are liability over time with several SLIs, like what is my availability over time, what is my latency over time, and so on, whatever applies. So actually what I was. So my conjecture was, my hypothesis was that if your DORA metrics, basically if you continuous delivery indicators, if you are doing well on them, then you will also do well on the SRE indicators. And so far, actually, unfortunately, or fortunately don't know right now I wasn't able to confirm that hypothesis. So I thought that if you are doing well kind of before production with your pipeline. So basically if, if the flow of limits is fast enough, if the failure rates on the pipeline environments are not high and so on, then you would automatically also do well in production. And so far I couldn't make that connection. So I think that's still an area of experimentation for me and for us. So I'd like to dig deeper there in order to understand what can you do before production in order to do better in production and somehow measure this. And I thought to do this by combining the indicators, DORA and SRE indicators, but so far I wasn't able to really combine them. So I think it makes sense to look at both because they measure different things also different things in time, so to speak, but a real combination. So far I wasn't able to achieve.

All right. All right, cool. So speaking about another experiment, you're probably sick of hearing out people are, but AI. I don't know if you mention AI a lot in the book, but now past few year, generative AI has really taken off. Do you see that impacting SRE at all? Can that help sre? Do you see AI? Do you have any thoughts on AI helping or like helping support an SRE type role?

Absolutely. So imagine you are a new engineer on a team, right? And the team already runs the SRE wing. So then you come in and you start familiarizing yourself with the code base, with the test code base. And now you also want to familiarize yourself with how they run the services in production. So you go and want to now look at some logs. For example, you want to start doing some queries in order to understand, to understand how the services are doing in production. So whether there are some breaches and so on. Where are the associated postmortems? Imagine you could do all this by speaking to chatbot. Wouldn't it be cool? I think you would become productive much faster than today without this.

Absolutely. Is that something you're experimenting with or is that something on your.

Absolutely.

Yes.

Awesome. So, Vlad, this book has 15 chapters. We've only covered the surface. Having taken a deep dive, I highly recommend people check it out on Amazon. It's called Establishing SRE Establishing Step by Step Guide to Introducing Site Reliability, Engineering and Software Delivery Organizations. And we will have link for it in the show notes. But Vlad, before we go, is there one piece of actionable advice you can give to someone to help them with their SRE efforts? And what's the best way to find or contact you?

Yeah, so the best way to find me is on LinkedIn. So just go in and you'll find me. And also what I do on LinkedIn, I publish my chapter summaries. So if you don't want to read the entire book, then just go to LinkedIn and you'll find on my profile the chapter summaries that have come out so far and I continue to regularly publish them.

And for links of everything of value we covered in this DevOps toolchain show, head on over to test guild.com forward/p120 and while you're there, make sure to click on the Smart Bear link and learn all about Smart Bears awesome solutions to give you the visibility you need to deliver great software. That's smartbear.com so that's it for this episode of the DevOps Toolchain. Show. I'm Joe. My mission is to help you succeed creating end to end full stack DevOps toolchain awesomeness. As always, test everything and keep the good. Cheers. Hey, thanks again for listening. If you're not already part of our awesome community of 27,000 of the smartest testers, DevOps and automation professionals in the world, we'd love to have you join the fam@testguild.com and if you're in the DevOps automation software testing space, or you're a test tool provider and want to offer real world value that can improve the skills or solve a problem for the guild community, I'd love to hear from you. Head ON over to testguild.info and let's make it happen.