Building an SRE Process with Evan Niedojadlo

Welcome to the Test Guild Performance and Site Reliability Podcast where we all get together to learn more about performance testing with your host, Joe Calantonio.

Hey, it's Joe. And welcome to another episode of the Test Guild Performance and Site Reliability Podcast. And today you're in for a special treat. We have Evan joining us back again to talk all about building an SRE process on the shoulders of technical debt. If you don't know, Evan is an engineer that currently focuses on infrastructure, distributed systems and process with some security sprinkled in. And he runs a YouTube channel and Meetup that focuses on helping others learn from the ups and downs of his experience. While I'll definitely have links to both his YouTube channel and the meetup in the show notes, because I think if you have anything to do with, say, reliability engineering or automation in general, you definitely want to follow him on all the socials. So you don't want to miss this episode. Check it out. This episode is brought to you by SmartBear. Listen, load testing is tough. Investing in the right tools to automate tests, identify bottlenecks and and resolve issues quickly could save your organization time and money. SmartBear offers a suite of performance tools like Load Ninja, which is a SaaS UI load testing tool, and Loaduipro, an API load testing tool to help teams get full visibility into UI and API performance so you can release and recover faster than ever. Give it a shot. It's free and easy to try. Head on over to smartbear.com solutions forward/performancetesting to learn more.

Hey Evan, welcome back to the guild. Hey Joe.

It's great to be back.

Awesome. Awesome. So it's been like a year. I think when we first talked you were making a transition into an SRE role. So before we get into it anything, I'm missing you a bio though that you want the Guild to know more about?

No, I think that covers it. Yeah, it should be good.

Cool. Cool. So as I mentioned in the beginning, like about a year ago, you started off in sre and I think you've been involved now in starting initiatives at your company to get a process in place. Why don't you walk people through like where you started, how that transition happened and what you walked into as like the starting point.

Sure. So just a while back, it's kind of put into the quality process and at the time we didn't really have a process. So what I ended up doing was just getting buy in from, you know, and of course at the time was a much, much smaller company. So, you know, Basically what I ended up doing was, you know, to get that buy in for quality and to kind of scale that team, I guess. Strange thing to say these days, but to sell automation, what it ended up doing was talking to just kind of the executive level people and, you know, essentially asked them, you know, do you want to release more frequently? And as a result, you know, that was, that was it. It was just able to kind of translate, you know, the technical, fun, fun stuff that I love working on into more of like, that business impact, you know, what does it ultimately mean for the business. So for sre, I was pulled into this role at the time for a larger project and I was really doing, you know, more DevOps work than anything, you know, so that was. It was kind of a thing there. So I was doing primarily, you know, infrastructure focused work and building it out. So, you know, that project ultimately failed for various reasons. I was able to deliver on the infrastructure side, but it was, it was quite ambitious and there was far too much reliance on the open source side, which I can speak to, and I should probably do a YouTube video on that at some point just so people can hopefully learn from the mistakes there. But, you know, fast forward a couple of months and I was put into more of the SRE role itself because some things I started to notice at the organization was, you know, the lack of. The lack of alerting on specific items, the lack of instrumentation as far as, like, what we were trying to monitor and all these kinds of things. And really that's where what ends up happening is you have all these existing processes, but you start to notice just kind of the lapse of, I guess, the lapse of what we need and what we needed as a business. So I kind of took that to note and came up with a bunch of ideas and kind of learned more about SRE as a whole just by watching probably an unhealthy amount of YouTube videos and reading books and all these kinds of things, and came up with more of a process and kind of presented that for buy in. Eventually, after some documentation, after some conversations, we were able to pull some more people into sre. And this is. Yeah, so it's an ongoing process, but just based on the last couple of months and what we've done is been. It's been nice, you know, it's just been something the business, in my opinion, desperately needed.

So first off, was there actually a role called sre or did you say, hey, look, there's a need for sre, therefore I should have this title?

So it's funny because I see this at a lot of organizations as well. And I've kind of. Some of the things that I've noticed, you know, even, even with like, you know, just with the SDAT role it was kind of like, you know, with the SD just it's like, you know, kind of, you see organizations kind of like wave the magical wand and then, you know, testers become anesthet, you know, that kind of thing. And I saw that with SRE as well. Like at a lot of organizations where like, you know, you, you might have somebody with a sysadmin background and then, you know, they're magically, you know, an sre. So in my case, I guess it absolutely was that circumstance where, you know, the magical wand was waived and I was suddenly it became this sre, but I wasn't, I wasn't doing SRE work for at least, you know, for, for some of that time. I would argue that it was more ops focused, you know. And of course the DevOps title is thrown of argued about if it even should be a title and all these kinds of things. So for us, and what I was doing particularly was more infrastructure based work. So I was building out the AWS infrastructure for us and I guess even further back at the time I was building our Google cloud infrastructure and then I was kind of pulled back in AWS land. But yeah, I would say I was more given the title. And then you like, because of that, you see other people in the industry kind of talking, you see, you know, you see the thought leaders talking and what it is to them and all these kind of definitions and you read the books and you kind of learn more about it. And as a result I kind of brought that and made light of that and as a result kind of got pushed into more of the actual role and the day to day that, you know, we should be kind of focused on given the context of our organization.

Nice. So do you think most people assume that because you're in DevOps that SRE is a part of DevOps so therefore they just assume you're doing type of SRE activities? Or do you think that they just don't even know there's a division of labor almost between the two?

I think so. I mean, I think, I think there's really like this idea and this understanding of like there's no difference between the two. And I see, I mean in our organization particularly, like, honestly, like one of the first things that I had to do was get first, you know, the executive leadership buy in, of course but then the engineering team buy in because you know, if you're doing things like service level directives and all that you need to communications key, you, you know, you can't just spearhead SRE alone. You have to involve the entire engineering organization. But one of the first things that I did was actually diagram using like lucidcharts, using like lucid charts and diagrammed out each role because we had ops. But what, you know, what did that mean? So it was kind of making sense to leadership. And I set up a meeting in the morning and got everyone involved and we, you know, there's like 30 or so people on the call and we kind of went over each of the roles and this included it actually because we have an internal, more of a traditional IT team as well. And there's like a lot of overlap. But I think overall in the industry there's a lot of overlap between OPS and sre. But as far as like the day to day, you know, what I wanted to focus on for us was like, you know, more of a production focus. Our overall job is to essentially keep the lights on. That's what I wanted to kind of focus on for sre. Given the context of our organization, it could be different other organizations, there could be different titles for this kind of thing. Sre, I think Facebook calls it like infrastructure or sorry, production engineering and Amazon has a different term for it. Apple actually uses SRE term or the title. But yeah, just, just so that's, that's one of the first things that I did because I wanted to kind of, you know, this showcase, like this is what we do and I, I hope there's not going to be, you know, kind of like too much of an overlap where like you know, we're working on something and it's happened where we're working on something and then potentially another team might actually be working on the same thing. So like that communication is, is so key there. But anyways, that's this long version.

Nice. So you, you actually sold it to management and after you sold it, how did you put it into place? Jeff Steps. Did you start small? Is it hands on? Did you just say thou now shall have s here, here's what you're going to do. Like how did you start rolling it out to your organization if it was kind of new to them?

Yeah, so, so outside of, you know, just the initial things that I did there just with like that, that executive leadership buy in and all that, then kind of the next step thing was getting more people involved because during the same Period, when it was essentially, you know, me just kind of working on this thing. It did a lot of documentation, so I got the approval for, like, a confluence space. So I was just kind of able to document and learn, spend a lot of the time learning through the community, you know, outside of work. It was just a lot of, you know, learning from the experts, you know, watching YouTube and following some people on Twitter and all that. Cool. And then, you know, documenting everything. But the next steps to actually get things moving and get things rolling. I ended up doing a roadmap. So I roadmapped out basically, like, everything that I saw that we needed, you know, from a business perspective, like, what do we need to do now? And, like, some of it was just shockingly obvious. Like, it was just. It's, you know, if other people are listening to this, they'll laugh. And, you know, like, incident management was a big thing. Like, we didn't have a real process around that. You know, there was essentially one person getting a call for an outage, which is, you know, if you read about, I mean, any of the kind of the incident management approaches, you know, you know, that's a no, no. And as a result, like, so, yeah, I road mapped out, like, all of these items, and then I got a meeting together with one of our VPs and some other folks, and I basically, from that roadmap, what I ended up doing was from, you know, I kind of pulled a couple items that I thought would make the most impact just from. From the business standpoint and, you know, an overall engineering standpoint. And then I got feedback from the VP and basically was able to pull a few items. So our roadmap was built out, and then we got quarter goals. So there's. So some of those items were pulled out of the quarter go. You know, the quarter goals essentially turned into sprint items. So we had to kind of create a process. And we already kind of had that process, but we kind of had to tune the process. And that's where, like, that technical. The shoulders of technical debt thing comes from. So we had to tune that process to. So for sre, here's our quarter goals. Let's turn those into stories. And then, you know, we had some stories to work on for that quarter, which we're finishing up now. And we've, shockingly, we've done fairly well. So I'm kind of rather excited about that. But so that's kind of where it's at. The roadmap is kind of the biggest piece there. In addition to the documentation alongside the Communication because if you're not communicating this across teams and you're not showing, you know, we had different syncs throughout the Sprints and quarter where we met with all of their team members and kind of showed them like, hey, we're going to work on this, this and this. So you know, you have your kind of stereotypical Sprint meetings and all that. But yeah, so we use the roadmap as kind of a backlog but that's kind of the initial beginning steps. You identify at your organization what could be improved upon, what's going to make the most impact now. And for us the main things initially like were incident management related items and then the next quarter we're looking at service level objectives. So.

So the roadmap almost sounds. Sets the expectations for the whole. All the teams look at the roadmap say, okay, here are the expectations for this Sprint or this release.

Oh, for sure. Yeah, that was, that was a big thing. Like I said, that communication piece, it was a matter of posting it to Slack and just getting that, really getting that buy in, in in the communication across teams and those Sprint meetings helped. And then eventually like when we kind of got a swing of things in progress, we also, we got our own Slack channel. And that's, that's kind of what I knew like things were getting serious. I was like, oh, here we go. You know, SRE has their own Slack channel now. So it was a big deal for us, kind of jokingly, but y. And then, then we had a sync put in place as well. So that sync which was, you know, we meet once a month, it was a cool, it was a pretty cool time because it was kind of like, you know, we worked on this, this and this here. You know, here's some issues we might have here and here and there. We need some help with ops here, that kind of thing. So yeah, it was just kind of this progression into more of these additional processes in place. But yeah, nice.

So you did mention something about some headaches and mistakes you made. One of them was, I guess you went in with the DevOps mindset and you mentioned something about open source and open. Doing that created some sort why everyone thinks open source is the solution to everything. So where do you differ on that with SRE and how's it different from your past experience then?

Oh, absolutely. So yeah, just this initial product I was, I was a part of and this was, you know, kind of the initial reasoning as to why I was pulled in SRE from, you know, more of that SDT role that I was Doing initially because before that I was, you know, I was working on our Selenium test suite which was a lot of fun and Pytest and you know, all that fun stuff and I loved it. And it's actually still running in production, which is kind of cool. But so why I was initially pulled in SRE was because of this larger initiative to kind of rebuild everything we were doing from an engineering perspective. So it was this ultra ambitious, far, far too ambitious. So I guess there's like that cultural piece to it where you know, I guess really you can kind of say like ego kind of got in the way. And then how I was pulled into it, which once again I just like, I think from the SRE perspective when you're starting out with something like that and it's this new greenfield project that you're working on. Ooh, get more people involved, you know, because when I was the only one building out our Google cloud infrastructure GKE and using Terraform to do all this myself without knowing kubernetes and learning it on the way it was was very much an unhealthy amount of time put into that, you know, and it was this kind of like this driven approach to things. But enough about that piece, you know how, you know, kind of like the multiple pieces that. So that piece was done, like we had the infrastructure and everything else was kind of running and it was all good. But there's this focus, this really, really, really big focus from an architectural standpoint on this particular open source project which I'm not going to speak to, I'm not going to speak to it. But you know, I think really what ended up happening was we just didn't have enough knowledge at a high level, like I guess at a high level to kind of paint the picture co generation, you know, this co generation project which will essentially build, like allow business owners to build out particular services without involving development too much. So that's kind of like at the high level. And then from, you know, kind of low level stuff, we focused all that project was almost entirely focused on this open source project which you know, kind of like in hindsight you look at the involvement of that project when you're kind of like working with a newer open source project, you have to put the effort in as a team to also contribute to it as well. And that was kind of like one of the biggest takeaways from that. When we relied heavily on a team team to kind of build it up for us, you know, problems occurred and we were kind of waiting on things and realistically, in hindsight, like, we should have kind of taken that under our wing and really kind of did half and half because, you know, it's kind of open source, you, you take some and you give some back. So that's kind of my piece there. But. And then, yeah, I mean, like the cultural thing too, it's like, you know, you only have so few people working on it and you're really relying so much on automation to kind of solve your problems. You know, it's timely. When the world shut down, it didn't help there either. So it was, it was kind of a whole slew of events that turned that more into success in some ways and a failure in others. You know, at the end of the day, you learn from that and you, you kind of do these podcasts and you speak to it and hopefully people can learn from your mistakes and do good things.

Nice. So when you said you had a heavy reliance on an open source solutions, it doesn't mean you went to a vendor solution, but just like you should have carved out more time in order to contribute to the open source solution you were using.

Yeah, yeah, absolutely. Because we were involved in the sense of just communication, so we were, we were in their Slack channel and everything else and then we didn't have a backup, so there was really no backup. When, when that project, when there was like, you know, lack of people contributing to it and all these kind things, at the end of the day, we should have went all in, you know, I feel as though we should have went all into that particular project. I haven't looked at that project, particular project since, but I don't believe in a few things to just to look out for. Like I would say, number one, you know, contribute, you know, contribute to the project. Try despite how small your team is and all that, try to contribute to that project as well. The other piece there is, you know, just look at the state of the project. Look at, you know, how many, how many stars there are. What are the conversations, you know, are there a lot of open issues on it as well? You know, are there a really large amount of open issues on that particular project? Which is something we didn't look at, but yeah, great.

And another thing you mentioned was technical debt. You had a lot of years of technical debt. You said you had a roadmap. So is the roadmap the main driver you use to help break up that technical debt or do you have tips that people can implement to help look at that technical debt and say, okay, let's do this and this and wait for this. Or how do you handle technical debt? I guess is the question.

Yes. With the technical debt piece, I think, you know, just this particular organization that I'm at right now, it, it was more so and I, I've been here for some time now and you know, for me it was always interesting just because it's like this, this exciting thing where I've been there where you know, they' a month left of less than a month left of Runway and all these kinds of things. So for us it was more so of survival. So this technical debt thing as opposed like the sprints that we've had were to get items out, it was almost a matter of rushing to some sense. So as far as technical debt is concerned, there is that piece. So if you're at a startup like I was, you know, you just try to. And I've almost always entirely been at startups my entire career I've been at like 2 prior to this, to this organization. So it's always been like this wild ride. I. I don't know, like I would say just kind of. It's difficult to say because you know, from a business standpoint it was like survival. So try to like I've always been building processes around this release pipeline. That's been my career. That's almost, almost been my entire career. I was doing more programming prior to this, but it's almost been. Yeah, just if you're moving fast in your startup and you know, in the inner trying to reduce the technical depth, I, I just build processes around it. And that's kind of what I've done here with the test things that I've done with the SRE thing that I'm doing, build this process around it. And if you get to the point where shockingly find yourself at this company for a couple of years and you know, it' to mature and everything else, then I would suggest what I did was just kind of build a roadmap around the very obvious items you can find that will reduce the issues coming from that technical debt. Because initially what we tried to do was build a greenfield project to kind of almost entirely replace everything that was going on over there, which didn't end up happening. So now I kind of had to reverse my mindset. This Google cloud project we were working on that I put so much time into, wave it goodbye terraform, destroy all this beautiful infrastructure that was built and then get pulled back into AWS land. So it's a mind shock for sure. But then look at that and look at what you have, look at what you have already and then build that roadmap around improving that as much as possible. And there you will find, I guess they call it low hanging fruit or something like that, but you will find those items, I promise you, as we have. And then of course to get that buy in from though, just kind of highlight those, showcase all that, those items.

But yeah, so I guess I'm the opposite. I've always worked at large enterprises so they're used to processes. So it's always almost in this case you have a startup, how do you put a process in place then when you're not hindering the company from moving fast and pivoting so it doesn't become like a, like a weight on, on everyone in general?

Oh, it's tough.

It's so difficult.

It's, it's probably one like, I mean there's all the technology stuff and you know, there's all the fun tools and all that kind of stuff. But it's been, it's been one of the most difficult things of my career. You know, just working in this kind of space. Like I was kind of joking saying like there's you know, these people reaching out and whatnot. Like saying if I'm interested in some things, like I just, I don't know if my next role, I don't know if I'm interested in a start. But it's just such a, such a weird space where you know that there is that balance of you don't want to be the brakes, you don't want to be those brakes when there might be some seasonality issue with organization and you know, you either have a certain amount of Runway that you have financially so it becomes problematic there. I guess one of the best books that I've read was like a book called Explore it by Elizabeth Henderson. And like for me I just remember that analogy of the Net. So like, you know, in my friends of mine, and I'm from the east coast, so friends of mine, you know, they're either fishing professionally for a living now or they used to. And I just remember like the Net analogy that I saw in that book kind of spoke to me because you know, you have, you're going full steam. You know, there's this analogy, you know, you can picture a train going full speed and there's that, that meme where there's like a wood in front of the train. They're kind of picking up the wood as the train's going forward and that. So it's kind of the Same sense where you have to get that executive leadership buy in, as I've done over the years, but you just have to make sense of, like, how you're doing it. You know, do you like, in selling it in the sense of, like, okay, do you want to release more frequently? So use that analogy of the Net and keep adding to it and feel as though, like, you know, with the test stuff that I was doing, you add this layer and then further down in that. Within that layer. So, like, you have all these automated tests and now you have like this exploratory testing approach, which is cool, but then we were lacking, you know, I guess a lot. A lot of people call observability now. So you have the observability piece where you have all these pillars of observability, and you're kind of like, you keep adding to that Net. So I don't know, I've just always, like, thought back to that photo, to that picture in that book. And I was just excited about my bookshelf right now. But it's just helped. It's, you know, it's helped with this kind of environment you're trying to lay. Even if things are moving forward each sprint, you're trying to lay that foundation or that net rather, to catch those issues. There's another good book which I've skimmed through. It's like test. I think it's testing DevOps where it kind of has a similar photo, but except they use bugs. And the kind of. The bugs, you know, evolve, I guess. Well, not evolve. I'm thinking of Pokemon. But like, they grow into. They grow into a thing like. Like, you know, an insect to a few punches to some kind of butterfly or something like that. So in production, it's like the butterfly, you know, floating around, and it's a real bad bug.

But yeah, awesome. So you mentioned observability. I've spoken to a few people that said you can't really succeed with SRE unless you have observability and telemetry in place. So where do you start then? Do you start with looking at your. Like this seems like so many steps to get an SRE process in place. Do you go all in on one thing and then move to the next? So you do multiple things at one time. How does that work?

So I've. I've just been. I mean, this is from personal experience. I've just been kind of spreading out the work. So. So we've had. I've worked with Neuralink for years, and this is kind of why I went down that route. You know, a lot of people have different, you know, everyone has opinions on everything, right? So a lot of people have some strong opinions about, you know, which tooling you're using, understandably. And we chose New Relic a while ago and this was just to give us an understanding of application performance monitoring. This was just give us an understanding of, you know, how like to some level and mind you, you know that that's monitoring, right? So like the full observability was missing there. So like, that's also kind of speaking to like, you know, the shoulders of technical debt. So we've had a process and now there is a matter of just improving that overall process. And I guess thankfully, because I didn't, I didn't want to go down this route of like, you know, moving to another stack or something like that using, you know, and I do, there's some amazing companies out there like Honeycomb and Datadog and all these companies. They're even just rolling, you know, Prometheus and Grafana on your own. But it made a little things a little bit easier. So I did spread out the work and some of the work that I focused on so far it's been more so focused on really that observability piece. And you know, New Relic, I seemingly, I guess they kind of rebranded a lot of, funny enough, a lot of organizations seem to have rebranded and to utilize this term now. But what they ended up doing was, you know, during the shutdowns and everything else, we had to kind of cut costs. And I was kind of put into this position to, to start reducing costs. So one of the items I looked at was New Relic. And this was some time ago, like a year, you know, some time ago. Right. And I reduced our cost in New Relic. Fast forward. We're lacking observability. We're, we're truly lacking it. You know, we're not getting good, good distributor tracing. We're not seeing what our users are seeing from a latency perspective and all these kinds of things. So New Rel ended up rebranding and now as opposed to kind of selling each individual product and offering they have internally now they've kind of unlocked it for you. While costs go up from a user perspective, they've reduced costs on data because they kind of like want all of your data to go in to their platform. So, you know, the fact that they've kind of released all their, all their products, you know, it was kind of a selling point for us, which has helped us. So up until this point it's helped us, which is nice. There's still work to be done there, of course, but that's this kind of like, you know, to get to get up to speed with observability. It was more of a longer process for, at least for us. And this was. Given our team size, it could be totally different other organizations and like how long other people have or how large the team is and all that kind of thing. I think for us it was just a matter of like we've already had something, that something evolved to provide more offerings for us and as a result we're implementing those offerings and so far we're happy with what we've been seeing and it's helped us a lot in a lot of different ways. And just initially, like, you know, what we ended up doing was identify what would have the most impact from the business perspective once again. And one of the items was like APM was a big one. We had to do more alerting, so we had to reduce our alerts. There's so much alert fatigue going on, you know, that people were kind of ignoring the alerts. So it was like, oh my, that's kind of problematic. So we had to kind of work on that piece, reducing the alerts. What alerts make sense to us given our team size and all that kind of thing. We'd create an on call rotation process and we decrete like, I guess some people have a different name for this, but we, we call it internally postmortem. So like when there's an outage, how do we do that? What do we use? We ended up using opsgenie, so we have the Atlassian suite we're kind of working with. So there's like these little pieces down there observability route that we were taking, which led to more little pieces that we had to do, which eventually led into like a final edge project like Cowork Mine just finished up one of the pieces identified which was like third party monitoring. Because our organization and the context of organization were so heavily dependent on some third party pieces, we didn't have really a true monitoring sense on that. So we had to, you know, do that. And internally we use, we did scripting for something called neural, like synthetics to kind of check for those up times and all that. But yeah, so that's the, the long version, but yeah, great.

So how did you measure success? Do you have key APIs, you think everyone, KPIs you think people should look at when they're starting to say, okay, here's Our starting point and here's our end goal and here's how we can meas success does go to management, say, hey, we did this, we said we do this and here's the result.

Yeah, absolutely. And I'm actually bringing up my. The sync that we have for this right now.

Oh, cool.

I think so. Like some of it, so just, just right off the bat, some of it for management to see and to kind of, you know, showcase, like what we're doing really is sprint items, you know, I. And it's a weird thing for SRE teams because some of the work should be done, you know, and this is something. I've kind of acted as a bard internally and kind of sang the song of making sure we. Making sure we do half the time for fixing issues and then kind of like half the time for project work and we do measure that. And so some of the success there is like what we're releasing right now. So what are we releasing and what does that impact have? So the quarter goals kind of go, you know, kind of go hand in hand with that. So that, that's one thing there for monthly syncs, you know, what we're doing right now. And I, I do want to say, like, we kind of did this a little differently. Maybe almost kind of like an older approach to this. Because now a lot of organizations that I see, I mean, a shout out to Noble 9 and other companies that kind of focus on this is like that server service level objective piece. Because right now, I mean, some companies can quite. If you're getting started with SRA, you can qu. Literally drop an slo theoretically in like a GitHub repo for a particular service and say like, hey, we need to try and abide by this particular SLO per service team, depending if once again, if you're doing service teams and all that kind of thing, right? So some teams can, I mean, if you were starting out and if I would start out again, I would try to push the SLO piece. But it was a little more difficult because, you know, internally I'm not, I'm not. It just depends on, you know, who you're working with in the culture and all that kind of thing. Right. But for us, internally, monthly KPIs, what we're looking at right now is once again heavily focused on incident management because we were kind of totally lacking that. And we're looking at incidents resolved, which, you know, it's one of those kind of shoulder shrug metrics.

Right?

Meantime recovery last month, meantime to acknowledgement last month, all of these are kind of being pulled right now from the analytics suite within opsini. So I guess it's an. It's a good thing for us to look at just because right now, given the state of like, you know, the organization and all that, it's something we want to look at right now. Eventually I want to look at SLOs and kind of focus entirely on that. But things we're looking at is service performance. So we. I noticed we didn't really have a whole lot going on with performance. This was something I did a while back when I was kind of more focused on the test side of things. And we just kind of identify and highlight, hey, if this service was, you know, there's kind of an anomaly here, like, you know, and it would kind of work with that team. So that's something we kind of bring up as well. Another thing we look at is Velocity, and this was relating to Sprint work as well. So we're looking at kind of the basic stuff there. And then otherwise, I mean, those are kind of like the hard KPIs we're looking at right now. And then we have team updates, post warning, review, and then just kind of secondary topics which. Which talk about, like some things I'm working on right now is KS engineering. So inner Ql we're kind of working on right now. More alert, channel consolidation, service level IND disaster recovery, and then the readiness metrics. So SRE readiness, like, where are we as a team? Who knows what? Who might have more experience with go, who might have more experience with Python or Bash or whatever that might be. And then observability readiness showcases, like, here's the stack and what we're looking at from like an observability standpoint. And what are we. What are we lacking? So we kind of quantify that and turn it into a metric that we can kind of look at each month and where we're lacking on instrumentation.

So, yeah, you start off in an SDA, SREs become big. Is there any overlap where. If an S stat's listening or an automation engineer, where they can do something, something within the scripts to help sre, are they completely, like, unrelated? There's nothing they can do to help one another?

Oh, for sure. Yeah. So I work with the test team, so I bring the test team in. So, for example, I mean, there's just there, in my opinion, I mean, there's. I feel as though there's a ton of overlap. Like, I think with sre, you just have that mindset. You know, you have that testing this Is testing mindset of just like you're thinking of what can go wrong, right. A lot of the time, at least that's what. That's what I was doing. I was thinking of, like, what could potentially go wrong here from like, you know, from a service perspective or something like that. So, like, at an SRE level, you have the big picture, you know, you're looking at like, the platform, the infrastructure, the, you know, all of those things now, the services. So I think there's just from. From like an SRE standpoint, even test testing to SRE standpoint, right. There's just this. A lot of overlap. And I think it's helped me. I think it's helped me as my career, you know, especially being like an individual contributor because you're just able to see so much just for coming from that testing perspective. I really do think it's amazing. And that's why I've. I've helped pull in the R test team with the chaos engineering efforts that I'm doing right now, because there's just so much, so much input, so much amazing input from them in multiple team do. I work with the OPS team and the testing team and pull them in for things called game days. And with the game days, it's this fun event where, you know, you come up with an experiment and a hypothesis. And right now we're focused on staging. We're not too mature in that process yet, but our focus is on staging, to break things and identify, you know, what could go wrong with that. And, like, from that regression testing mindset, you know, if you have an outage like we just had recently, you know, what can you learn from that outage, but also from the regression piece, like, can you write, you know, a KS engineering suite or put into your pipeline, your release pipeline, where you can ensure that that particular out could not happen again. So I think there's a lot. So we have this regression test suite, and the same thing could be done for SRE and KS engineering and getting that input from OPS folks in. In the test team as well.

Awesome. Okay, Evan, before we go, is there one piece of actual advice you can give to someone to help them implement the SRE efforts and what's the best way to find or contact you?

Absolutely. Yeah. So I would say, I mean, this is, like, it's a difficult thing to say, but, like, find the confidence in yourself to push these improvements forward because it's so difficult when you're working on so many different things. But it's really just a matter of finding that confidence and just highlighting it. And it's one of those things where you just kind of make the effort to do it, show what is lacking in the organization. And I think it's potentially an easier thing to do at startups, just smaller to medium sized organizations. But it's helped a lot. It's helped me particularly build out like a test process, an SRE process so far and really kind of you can, you can utilize that, that confidence and just to really build processes itself and if, if you can speak the business language, maybe you can take it further. And that's, that's kind of my advice there and from there as well. Just based on what I've learned. Like learn to love yourself, do that because you know when you're putting in a lot of time like I like I was doing on the Google project.

Wow.

Yeah. Find the time for yourself. You have to. Cuz you know, you just can't do this stuff all the time. And then as far as finding me, find me on YouTube all around the Internet, of course, Twitter, YouTube, it'll be in the show notes LinkedIn. I kind of just used to kind of post the YouTube stuff and all that, but I do the YouTube stuff and all that kind of thing just to just so people can learn, not the, you know, to avoid the mistakes that I've made. So yeah.

Thanks again for your Performance Testing Awesomeness. Links to everything of value we covered in this episode. Head on over to test guild.com P7.8 and while you're there, make sure to click on the Try them both today link under the exclusive sponsor section to learn all about SmartBear's two awesome performance test tool solutions, Load Ninja and Load UI Pro. And if the show has helped you in any way, why not rate and review it in itunes? Reviews really do matter in the rankings of the show and I read each and every one of them. So that's it for this episode of the Test Guild Performance and Site Reliability Podcast. I'm Joe. My mission is to help you succeed with creating end to end full stack performance testing awesomeness. As always, test everything and keep the good. Cheers.

Thanks for listening to the Test Guild Performance and Site Reliability Podcast. Head on over to testguild.com for full show notes, amazing blog articles and online testing conferences. Don't forget to subscribe to the Guild to continue your testing journey.