Making the Move from SDET to SRE with Evan Niedojadlo

Welcome to the Test Guild Performance and Site Reliability Podcast where we all get together to learn more about performance testing with your host, Joe Calantonio.

Hey, it's Joe. And welcome to another episode of the Test Guild Performance and Site Reliability Podcast. Today we'll be talking with Evan all about making the move from SDAT to sre. So Ian will probably also touch on other topics like working from home. Evan is an engineer with a wide variety and range of test automation development, security, software quality, DevOps, site reliability experience. So he's done a lot of things. He also organizes the Austin Automation Pros meetup for everyone who's interested in learning. Really great meetup. I had them on test talks. I'll have a link to that in the show notes. They've done a lot of great things there. So if you're in that area, want to. Once everything calms down with the coronavirus, you definitely want to go and check it out, but you don't want to miss this episode. Check it out.

Let's face it, performance testing is tough. Traditional load testing tools create scripts that are bogged down in data that's hard to read and require a lot of work, even for simple playback. Load Ninja cuts out the dirty work by using AI to inspect and debug your code almost instantly. No coding necessary. No changing of hands or relays between teams either. Give it a shot. It's free and easy to try. Head on over to testguild.com smartbear and click on the link to learn more.

Hey Evan, welcome back to the show.

Hey Joe. Yeah, it's great to talk to you again.

Very cool. So it seems like there's a lot that's been going on with you since last time we spoke in the Test Talks podcast. So before we get into it, can you tell us anything that I may have missed in your intro?

Sure, yeah. I mean, basically just ended up switching over from, you know, mostly focused on test automation and quality in general and then just more into the SRE space. But yeah, not. Not a whole lot. Yeah, it's pretty much covers it.

So I actually think that's a pretty big switch going from SDAT to sre. Maybe I'm wrong. That's what this show is all about. But why did you make the switch?

I made the switch. I was just interested in learning more about operations. I know this is something that's been around for quite some time, since the beginning essentially. But I just, I heard, I think it was. It might have been a meetup, but it was basically just like ideas of testing infrastructure as code. And that was just something that kind of sparked an interest in learning more about the space. So, yeah, that's kind of how I got into it.

Nice. Now for folks, I don't know, I guess we should start with this. What is a site reliability engineer anyway?

Sure. So site reliability engineering is pretty much at a high level. It's just a mix of responsibilities at this point. Operations being solved as code, just reducing toil, just adding automation to the mix, identifying reliability in systems and ensuring it.

I think that covers it at a super, super high level. And then of course, as I mentioned in the article, you want to check out Liz Fung Jones and Seth Bargo's discussion on this role on YouTube because it goes a lot more in depth and differentiates what it actually is versus something like DevOps.

Cool. So we used to have my other company, a DevOps team, but it sounds like this is. So this team would be separate from your DevOps team.

It would. And in my particular organization right now, it's just kind of a mix of roles, particularly because we're just a smaller team. But in larger organizations, you might see SREs.

Just something as simple as an infrastructure Engineer.

Or site Reliability Engineer. There's other titles for it as well, but ultimately they're going to be hugely responsible for just the uptime, the reliability of these applications, of these services.

Nice. So when you made the move, I'm just curious to know skills you needed that are the same and skills that are different. I think that'll help people know how big of a transition it was. So moving from an SDAT to an sre, what skills were the same that you used as an SDAT that would be the same that you would use in an SRE type role?

Yeah, absolutely. I mean, really what it comes down to is, you know, the main takeaway here is definitely understanding code at this point, because a lot of what I'm doing is just driven by that. I mean, at a high level, just a curious individual, of course, you know, just the ability to question everything. So, you know, perhaps you're breaking things on purpose is one of the things that I'd mentioned. And, you know, taking on the roles of users. So potentially, you know, thinking about security, thinking about, you know, as you're setting up infrastructure, you're setting up a firewall, whatever you might be doing, and just thinking about how you can potentially, you know, break that or how you can get in there with infrastructure. It's interesting because, you know, you can set up an infrastructure, you can set up. You know, in my, in my personal experience with Kubernetes engine, I'm using gke, so that's Google Kubernetes engine and we're setting up services. So we're setting up a bunch of microservices right now. And all of these are pods and they're running fine at the moment. But realistically you want to get in there and you want to basically just remove one. For example, what breaks, which microservices were reliant on each other. So it relates to fault injection as well, it seems. But that's some of the high level stuff. Context, of course, understanding some deeper purposes of some of the tooling that you're using. Because in the OPS and SRA space there's just so many, I mean in the test world as well, there's just like this overwhelming amount of tools. Like whether it was open source or whether it was like vendor specific. There's just a lot to keep up with.

How do you keep up with it then?

Lately, a couple of different ways. So a podcast is definitely like one way that I keep up with it. I mean, I'm in a different position where I do have a commute, not incredibly long, but to the point where like I can enjoy some podcasts, I can enjoy some audiobooks. There's quite a few podcasts out there, like the Kubernetes podcast from Google. Because that's, that's mostly what I've been working on. And mind you, Kubernetes isn't something that's specific to sre, but it's just something that I'm specifically working on. I mean you could have like more of a traditional infrastructure, you know that and still be an sre. But just with the introduction of this, all this new tooling, you know, you can pretty much expect to be working on microservices Kubernetes platform. But yeah, just getting back to it, just the podcasts. Podlets is another good podcast. Obviously Task Guild, whether it's performance security on call Nightmares is pretty interesting because it talks more about the space of essentially being woken up in the middle of the night. If there's an issue with the system. The whole aspect of having a pager, the pager duty.

Test and code. I'm kind of a big fan of olliecast, so that's for observability. That's a really interesting one. I've listened to a lot of great ones from there. And that's over from Honeycomb I.O. and Hashacast, of course. And then those are some of the podcasts, there's obviously books, but it really depends. Sometimes I'm digging through books because over the past two years or so I've just been, I've always been into books. So for me that just works. For me, something can work entirely different for someone else. But I mean, just throughout the day and find myself digging through documentation, but I mean that's just staying on top of things. Twitter as well. I get so much information from Twitter and it's definitely overwhelming. So just target who you follow, whether it's sre, ops, whatever, whatever they're tweeting about, whether it's a new language or new tool set, whatever it might be, you know, it helps out a lot. And because, you know, we're multi cloud at the moment, so with AWS and Google Cloud, I found myself just reading their, reading what they're releasing because oftentimes like what they're releasing, you know, relates to something that I might be working on. So that, I mean really that helps me stay on top of things as well. But at the same time, like you just don't want to overwhelm yourself because it's super easy to get overwhelmed in the space. Yeah.

So I'm just curious, you know, if someone is trying to think, oh, I'm an S dent now maybe this sounds interesting. I like to be a little more involved with cyber liability engineering. What skills do you think they need to hone in on that they may not have currently to help make that transition over?

Yeah, absolutely, I would say, I mean, because my traditional background was just in purely exploratory testing, I was signed on just to click around and just to identify issues with the site. And then over time.

I just ended up picking up automation and with automation.

You know, focus on languages. So I mean really like a big step in the move, in my opinion would be just to lock down a language. I know when I was working on, you know, more test automation problems, checking whatever you want to call it.

I really enjoyed Dave Hefner's book. And that was this selenium testing book. And basically like, you know, what I really enjoyed about the book is you had several different options for several different languages, language bindings. And I had chosen originally Ruby because I was building out a proof of concept for us and I just got really, really involved with it and I actually ended up going through a code bootcamp. So I thought that was interesting and that helped out quite a bit. But once again, like I said, you can do a lot of self learning. There's a lot of Free resources out there like pre code camp. But that book made a big impact in terms of learning a language and then just kind of applying that. Eventually went into Python and then Python kind of, you know, works hand in hand with a lot of things but you know, in a lot of ways with infrastructure as well, like Boto 3 with AWS. So I would, I would definitely say like hard skills like you know, pick up a language, get, get comfortable with, you know, potentially Python, potentially working with Bash, something like Bash, definitely the command line I like as just in like traditional testing and traditional exploratory testing. I wasn't working a big deal with command line tools and stuff like that prior to actually getting into more of a test automation role. So that's just something to keep in mind as well. If you're coming from purely just a testing background, definitely take some time each day and just focus on a core language because you're going to be working with a lot of different things, you're going to be working with a lot of different languages and that's something to keep in mind as well. And then definitely familiarize yourself with infrastructure because that was a concept I wasn't familiar with either. I knew to go to a specific endpoint and then just begin running through the site and testing the site, whether it was from exploratory testing and time boxing that or whether it was through a list of items to actually test and then eventually to automate. So that's something to keep in mind as well. Just like how infrastructure works, particularly public clouds. And of course it depends on what organization you're in right now or you want to go to. But I would say the majority of what I've seen from small to medium sized businesses, they're definitely dealing with public clouds. I mean the current organization I'm at, our exception is we're a multi cloud to some degree, so we're using AWS and Google Cloud.

So that's definitely something to keep in mind there as well. Just understand infrastructure, understand a language and get really comfortable with that language. Try to get really comfortable with that language and look at, maybe go to linuxacademy.com and then say for example if you chose Python and then go through their Automating AWS with Python 3 that helps out a lot. Even if you go through that course you, you know it's going to take you step by step through understanding the language itself, but also like using it with.

Like a library like Boto3 and then you can understand each piece and mind you like another Takeaway is just like, you don't have to understand, like, all of the infrastructure, but just like some specific, like, maybe even like an administration perspective of like, you know, setting up IM roles and maybe deploying like, you know, a couple EC2 instances or whatever it might be. But, yeah, that's what I would look into. And even to push that further, maybe even looking at kubernetes. But once again, it just depends on the size of the organization you're in and then what tools you're working with.

A question I would have is, how big of a role do you think this is? Meaning how popular do you think this role has become? Is it something that's just common to maybe Boston and Austin and San Francisco? Or do you see this as a role that is growing in popularity and that people would benefit from learning more about it?

Yeah, totally. I think, based on what I've seen, definitely major cities. And I'm not like, I'm originally from the east coast, so, I mean, me moving to Austin, Texas is where I'm at right now. I was just kind of pulled into a world where, like, yeah, there's a lot of new technologies happening, a lot of new technologies happening fast. And not only that, but like, a lot of new titles that you just see. And like, sometimes, like, it's like recruiter driven, right? So maybe like, you know, you're doing like, basic, you know, traditional ops or something like that, and then all of a sudden you just start seeing, like, these roles pop up for sre.

I think there's definitely a massive demand for it. I'm not going to deny that. I feel as though, you know, just looking at it from a numbers perspective like OPS and SREs, it's just there for now. Right. And that goes with everything. These things change. You know, I've been in the industry for, you know, five plus years now, but I've just seen a lot of change. But I feel like right now it's definitely in demand. But at the same time, I feel like there's just definitely like, a disconnect between some organizations understanding what, like, truly, like, a DevOps role is. Like, you're working on CICD and like, you know, you're working on a lot of implementation as opposed to something like SRE with like, you know, more of that reliability aspect of, like, you're maintaining systems and you're like, you're. You're pretty much ready to be woken up. You're kind of signing onto that. You're battling fires in the middle of the night. If you get woken up. So get those runbooks ready. But the takeaway here is yeah, I definitely see it as a larger role, as a required role. But yeah, we'll see where that goes.

So, I mean people still trying to get their head around this. Can you walk us through a day in the life of what you do an sre so maybe they get a flavor of, you know, what, what this really is all about.

Yeah, absolutely. So, so from my day today up until now, you know, I mind you, this is, and I want to provide the context here as well. Just from like the day in life perspective. The context right now is I'm working on a greenfield project and we're working on a new, you know, just a new approach to how we're doing things. Like we have, we have a site that's running right now. It's, you know, it's making the company money. Great, that's awesome. But at the same time we got some buy in because we had just this enormous technical debt on our existing system. So we got the go ahead to develop this new platform essentially and we were choosing all the technologies.

So whether that was aws in our case, Google Cloud. Right. So from the infrastructure perspective, from the SRE perspective, I chose Google Cloud because they were going to go with microservices and they are going with microservices and I just kind of like googled around and kind of understood and dove in the community and understood like, yeah, okay, Kubernetes is the way to go. All of our applications are being containerized. I need to go with K8s and then one of the options there for us was GKE and for what I wanted, it just solved everything. So that's kind of the context. Right. So because we're on a new project, actually I would have to say very much like doing a lot of ops work right now in the sense that today I woke up.

Went to the office because I'm like I said, and we're dealing with, you know, absolute world crisis with, with the coronavirus. And I went to the office today to pick up a few books, headphones, et cetera because we're all working from home at the moment, very, you know, lucky to be able to do that and then review my emails, hopped into Slack, you know, kind of looked at what everyone else was working on today, kind of posted what I was up to because that's kind of what we're doing remote wise and then dove into some documentation. So yesterday I was, I was working with something called ZB and ZB is like this workflow modeler and I'm installing it on our cluster. So we have a running cluster in Google Kubernetes engine. And yeah, I was using HELM for that. So Helm is just like, I think it was like a package manager just for Kubernetes. So you have all these applications set up in charts. And then I just did an installation of ZB into the clusters to get one of the latest versions on there for everybody. Our infrastructure and our microservices are super reliant on that.

Then from there I went in updated Confluence. So that's some of our documentation. And then I'm going back and doing some research today on our continuous delivery approach. So I'm looking at Keel SH versus gcp, Google Cloud platforms, all these acronyms. They're continuous delivery models. So they have some pipelines that they released. It was like maybe like a week or two ago as an alternative to using.

Something like Spinnaker or something else. So that's what I'm up to today. Just some busy work, some documentation and then just some research and then just I would have to say like over the past couple of months because this is a new project and because you know, I'm like this is a new role for me as well. So it's just been like a lot of research and a lot of like a lot of doing things and just like learning from doing them wrong potentially and then going back and changing it. There's. And I do want to note like I'm, you know, because we're such a small team, we're like a team of essentially a bunch of seniors, you know, except for myself in this particular role in like the test automation space. The testing space. I'm more of a senior in that role. But in this one most certainly not. So yeah. And the interesting thing there is like I'm the only one working on this at the moment. So a lot of what I do is pretty crucial. But that's, you know, that's my day to day on average. I mean people. It just depends entirely in the organization. Some people can be, you know, writing Kubernetes operators and further automating their deployment process and other organizations. The title could be something different. And they could be working on writing some terraform, for example, and then applying that in however they apply it or just looking for things to automate, lessening that toil. Maybe they're working on some error budgets, maybe they're actively just relying or tinkering with their observability models.

So that's my day in the life and then potentially some other Srez in the life. It seems like a post I should write as well. Yeah, it's a really good point because a lot of people hear this title and they see some of the requirements for it, but they don't necessarily know what the day to day is. So that's really interesting.

Yeah, it could be when you see a job posting for sre, like, oh my gosh, this is a unicorn. No one could do all this. So. Right. I mean, to me, when I see an sre, I'm like, that's like every, every. You know, that's. I don't know anyone that could do all that, but it sounds like maybe.

Yeah, I don't know, pick a few that you're really good at.

And I totally agree with you. Yeah, it's like one of those roles where like, yeah, a lot of people. And that seems to be like a lot of roles nowadays. Like even like, like a full stack engineer. Like, I know back in the day it was like, it was like one of those unicorn finds. Like, oh my goodness, somebody can work on like front and back end like, you know, and actually update the postgres database or something. But yeah, I agree with you. Like, some days, some days I'm working on like, you know, I was doing some postgres work and like, I think like just one of the main things just in engineering in general because like, yeah, ultimately at the end of the day, like, everybody's an engineer, but realistically, like, not everybody can do everything. Not everybody wants to do everything either. So I totally agree with that. But I do find, because I think, Mike, my context, my context is like a bit different than some others because I do find myself doing just a lot of different things. Right. Like I said last week, I was working on some postgres stuff and I think what it comes down to is just really having that knack of just picking something up and learning a little bit about.

Kind of sucks for me sometimes because I'm the type of person who I do want to dive into a book and just really, truly, solidly understand something. But sometimes when you're working in this space and like, maybe, maybe, you know, you have to find yourself like running through the documentation and just getting something done, you know, within a reasonable amount of time. Right? So that's. That's kind of how I see it as well. Like.

It'S interesting too because like, yeah, I totally agree. Like sometimes I see like security requirements as well, but like, not everybody can just have that. Right. So you have to read into it further and just kind of apply what you know at that time. But yeah, it's, it's just like mistake driven.

Mistake driven implementations and then you just kind of like learn from that or just, or just go back or like, you know, maybe you have. I have, I'm fortunate to have like a really good community of. Because I run the meetup. I run Austin Automation Professionals with some other fantastic organizers. And like I've been just fortunate enough to be able to learn a lot from larger organizations and how they apply things. I've used that as a model for myself to, you know, just, just having friends in like all these amazing companies, like, and just trying to learn from them, not, not bothering them. I'm not like, you know, I'm not like, hey, how do you, how do you do this? Like all the time? It's more so just like in conversation. I'm just like, you know, I'm running into this, how do we do this? And like, you know, the beauty about these, these meetups and these social organizations as well is just kind of like understanding how somebody's doing something and that's. And that. And I have a friend of mine and it's interesting because he was also an SDOT.

Test Automation. Like I said, once again, whatever you want to call the title. And he did make that transition over to sre Ops, whatever, once again, whatever you want to call it. And.

I've been fortunate enough to learn a lot from him. So that's a really good thing because my team here is small. His team was small as well. He had two or three people and he kind of took the lead over there for what he was doing at the time. He was able to tell me about Ansible and you know, how they're using it with postgres and was really nice and just sent me articles and stuff and so I thought that was good. So if you do find yourself in a position where like, you know, it's only you and like you're kind of like, you know, going through the day to day and it's on you to like kind of make these decisions and you're kind of thrown into the mix, definitely get involved with the community, whether that's in real life or right now, because we know we're dealing with a crisis. It's on Slack or Twitter. Just find your people, Find your good people to learn with and then give back, please.

And it sounds like a fun position. I can't imagine having another position where all you do is Write scripts that drive a browser. This seems like you get the best of both worlds. You're still able to code, you're still able to automate, But Automate really sounds like interesting things, not just a browser. And so it seems like the best of both worlds to me, for sure. Sure.

Yeah, absolutely. It's been, you know, it's been a lot of fun, you know, and I agree with you. It's like once you, Once you write that, like, shout out to hashbord, just like, just amazing things that they've done with like, Terraform and even Vagrant back in the day. Like, a lot of people are still using Vagrant. Just a lot of cool stuff with Terraform. Just like the moment that you write some of that, some of that code, whatever, you know, and like, you just see your infrastructure spin up like an EC2 instance or something as simple as, like, you know, compute engine instance in gcp. It's like, it's amazing. It's really, really amazing. I just remember, like, the first time I was in Sauce Lab and I just, like, automated. I went through one of the tutorials and like, automated a browser and I just saw that, that browser instance pop up and just like, I was. I was just amazed. That was fantastic.

You mentioned observability a few times, and this is a term I've been hearing more of maybe the past year. I never had anyone tell me, though, what is observability? What is observability? Like I said, you have mentioned it a few times.

Sure, yeah. I mean, it's, it's. It's one of those things, like, I've just yet to find, like, this, this solid, like, pinpointed definition. It's. I mean, Google definitely nailed an sre. I just want to, like, definitely encourage everyone to go over and check out to see what Google's up to with the SRE space and of course, the SRE handbook and everything else. But getting back to observability, I feel as though it's like. I feel like it's just this aggregation of information. Whether that's monitoring, it's apms with application performance monitoring, whether that's ROM real user monitoring. It's just like everything aggregated to the point where, you know, you have a full viewpoint and it's not necessarily like somebody like, staring at a browser all day and staring at graphs all day. It's, you know, you have a lot of data there, you have this high cardinality where you know, you can trust your system to tell you the right information and not even tell you the right information. But I mean really for us, especially in the SRA space or anyone like on call to alert you on the right information and then, you know, from there it goes even further. It's like to not alert you all the time because you're going to, you know, you end up hitting alert fatigue, but just to alert you at the right moments for the right data. And that all relates to know your service level agreements and all of that. But I mean that's my takeaway. I just feel as though it's like an aggregated amount of like monitoring and all of these things just to the point where it's like streamlined where people can actually like ingest it properly and you're not bouncing between a bunch of different, you know, applications that tell you, tell you something. And that includes logging too. I feel as though almost that includes logging. But there's, there's so many people that have like real better definitions than I do, I promise you. Because I mean my contact once again is I'm using, I used to use New Relic and we still use New Relic for our legacy application. But I'm actively looking into like for gk, like it was kind of out of the box. I'm just using stackdriver and I'm trying to set things up so it's, you know, it's alerting us at the right moment. But I would highly advise checking out Honeycomb IO and just everything they have to say about observability.

Because I almost feel as though like it's one of those words that everyone else started consuming. Everybody else started taking the word and using it in all of their marketing material. Whether, you know, I'm not going to name any organizations, but yeah.

So as a civil liability engineer, you didn't. We know we are living through coronavirus as we speak today and a lot of people are now working from home. And so I assume that a lot of companies are experiencing things on their infrastructure that they may not have anticipated. Now that their whole workforce is coming in from the outside. Is that something a site reliability engineer would be involved in or is that a totally different group when we talk.

About infrastructure, I definitely think maintaining the application reliability and all that is a big deal. Right. So.

From a remote perspective, people that are working from home right now, I mean there's traditional infrastructure, there's definitely a lot of organizations, banking and some of the more security space, whether it's Big Gov or whatever it might be. Yeah, there's definitely people on staff, there's network operations people and all that kind of thing. A lot of it. I don't know if it feels like a lot of it can be done from home now, and especially during a crisis. Right. You start to see a lot of positions that you thought were only in office. Now they're suddenly becoming, you know, a work from home role. And it's kind of shocking to a lot of people to see that as an employee, it's like, you know, you're kind of surprised, like, oh, I could have just been doing this. And like, I'm more comfortable, you know, just working from home or whatever it might be. But, yeah, getting to the point, it's like, yeah, from an SRE perspective, yeah, we can definitely do this, a lot of this from home. Especially if you're in a public cloud. I know a lot of SRE roles are remote. You know, there's tons of those roles. You know, so much of this is remote. You know, you just have to make yourself available and communicate. I mean, me personally dealing with this crisis. Slack has been super helpful so with them since working on all sorts of different stuff, but, yeah, I hear you. That's wild.

Okay, Evan, before we go, is there one piece of actual advice you can give someone to help them with their SRE efforts and what's the best way to find or contact you?

Yeah, absolutely. I mean, yeah, it absolutely goes beyond any of the resources that I posted. But a big deal is just don't stop learning. But ultimately don't get stuck in tutorial hell. Don't feel as though you have to go through a million tutorials before you actually hop into something, before you actively get your hands on something. That's just what I've learned the hard way, because that was me for a while. And then the main thing is if you want to change roles, you don't have to leave everything behind. So that's a big deal. There's a lot that applies to everywhere else. And then just in the context of what's happening in the world, definitely the Mr. Rogers quote. Look for the helpers, but also be helping yourself. I feel like a lot of people helping each other out through this. I see people on Twitter posting zoom links to chat rooms and stuff like that. It's super helpful for people who are remote right now, who aren't used to it, who need to socialize. I think it's super cool. So do that help each other out right now because it's a pretty crazy time. You can reach out to me on Twitter. I tweet a little bit. I want to tweet more and definitely my blog. I want to write more as well and just help out as many people as possible. And I want to give a shout out to Lisa Crisman for helping me with this blog post on the Dev Test Ops community. And yeah, that's it.

Thank you Evan for your performance, testing and site reliability awesomeness. For links to everything of value we covered in this episode, head on over to Test Guild Test.

And while you're there, make sure to click on the Try Today link under the Exclusive Sponsor section to learn all about Smart Bear's awesome performance testing tool Load Ninja. So that's it for this episode of the Test Guild Performance and Site Reliability Podcast. I'm Joe and my mission is to help you succeed with creating end to end full stack automation awesomeness. As always, test everything and keep the good. Cheers.

Thanks for listening to the Test Guild Performance and Site Reliability Podcast. Head on over to testguild. Com for full show notes, amazing blog articles and online testing conferences. Don't forget to subscribe to the Guild to continue your testing journey.