Bootstrapping a Security Process for Software Teams with Evan Niedojadlo

Get ready to discover some of the most actionable DevOps techniques and tooling, including performance and reliability with some of the world's smartest engineers. Hey, I'm Joe Colantonio, host of the DevOps Toolchain podcast and my goal is to help you create DevOps toolchain awesomeness. Hey, it's Joe and welcome to another episode of the Tuskill DevOps toolchain. Today we'll be talking with Evan all about bootstrapping, a security process for software teams. If you don't know, Evan is an engineer that's currently focused on all things. I believe right now he's focused on security, but he's also worked on reliability, distributed systems infrastructure, all kinds of things, DevOps software testing. He's done it all. He also is the co organizer of the Austin Automation Professionals and Austin Computer Book Club and he really focuses on providing a welcoming space to folks to learn. I have a link to all these things in the show notes as well. Really excited to have Evan back. He's one of those people I look up to because he's always seems to be learning and implementing and trying new things. So it's always great to have him back on the show. You don't want to miss this episode. Check it out.

Hey, if your app is slow, it could be worse than an error. It could be frustrating and in my experience, frustrated users don't last long. But since slow performance is a sudden, it's hard for standard error monitoring tools to catch. That's why bugsnag, one of the best production visibility solutions in the industry, has a way to automatically watch for these issues. Real user monitoring. It detects and reports real user performance data in real time so you can quickly identify lags, plus get the context of where the lags are and how to fix them. Don't rely on frustrated user feedback. Find out for yourself. Go to Bugsnag.com and click on the free trial button. No credit card required. Support the show and check them out. Hey Evan, welcome back to the Guild.

Hey Joe, it's good to be back.

Awesome. So I think last time we spoke you were doing more like platform engineering, but you submitted this topic about security. So I guess tell me a little bit more about are you now doing security and if so, like how. How did that transition happen?

Yeah, so we're building out pretty much like a formalized security process, right? Like this is something I've always somewhat have done, at least throughout the past couple of years, especially in high school too, which I can talk about, but specifically with work and career. Yeah, I've just got to a point where. Well, the past couple of things that I've done were specific to oasp zap, implementing that, adding that to Selenium. So that was back when I was primarily focused on software quality and specifically like the automation behind it. And then from there it went on to ops. And then, you know, I've kind of identified some issues that were just kind of there. So Secrets and, you know, implementing Hashcard Vault, which eventually switched over to AWS Secrets. So I've done stuff like that. And then more recently, it's been a matter of addressing a problem because an organization, it seems like when the organization becomes more of a brand. One of the things I've noticed is that tax seem to be increasing, you know, like, the more that it's out there, the more advertisements, the more issues. So that's at least what I've noticed. Cool.

So, all right, so you got me in high school, you stumbled upon security. I guess you have to expand a little bit on that one. Yeah.

So basically, I don't know, this is like 2007, 2006. So I was very interested in. Well, I got interested in computers before I did anything else. Right. Like, so I was just interested in security mostly because I got. It was like Kevin Mitnick's book, one of the many. But this one was the Art of Deception. And I remember reading it when I was working at a grocery store and just between my lunch breaks reading it. Eventually the school got a web design class and all these other classes, which is really, really cool. And that introduced me to everything. But friends of mine and I would probably very bad now, but back then we'd create phishing websites. So we would basically develop a site that looked like MySpace and you could essentially send that to somebody through IM or whatever. It wasn't like, incredibly sophisticated, but for the time, it was just something, you know, to do. I mean, it was one of those scenarios where, like, it introduces to security social engineering. And, you know, just looking back at just all of these things that are happening now just at a frequency. So it's kind of funny like that and all sorts of nonsense admin passwords. And there. There's just. I apologize to our system administrator for the high school, but we were just trying to figure things out.

I guess what's scary though, it's with that social engineering, it's still around, and I guess with AI and things, it becomes more and more sophisticated. So I guess we'll touch on that a little bit more but any thoughts on that? Is that something you see as being on the rise now with, or is it always going to be around? Because it's probably seems to be the lowest hanging fruit.

I think social engineering is one of those scenarios where if somebody's doing two fa, if they, you know, if they're utilizing SSO and OKTA and all these, you know, major implementations, you know, if you're not like a security researcher, you know, and if you're just something like hacker, then, you know, I guess one of the scenarios you could do with social engineering, you know, it's. And as far as far as it like being involved with AI, like I've noticed without a doubt like an uptick in automated emails. So automated campaigns, automated like spear phishing campaigns and things like that seem to be more prevalent now, like no doubt about it, because it's just easier to script up some message. Much, much easier to automate it, define targets, let it run loose. I mean even just in general, like even some of the companies I've contracted with and that I work for now is, you know, have been in scenarios where you deploy like a new case cluster and then you see scan, you know, you're going through the logs, the visitation logs and whatnot, and you're seeing these automated scans, you know, and you know, it's automated because they're just going to, you know, it's. They're testing for like PHP vulnerabilities, right? Like in WordPress vulnerabilities, things like that. They're very common things, but like they're targeting the most common things. And then if it finds something, my assumption is that it sends it back to a group of people or an individual to act on that vulnerability.

Nice. So, Evan, you know, a lot of times people call DevOps dev stack ops. I don't know if it's stuck, but because this is the DevOps toolchain. Curious to get your thoughts on does security belong in the DevOps kind of vertical?

Yeah, absolutely. It's one of those things that I think it was like 2018 or so. I saw an issue where we could run this OWASP zap scan manually. But the issue with that is this manual approach we weren't getting. We'd have to do it consistently. It just didn't make sense for what we were attempting to do. And what we were attempting to do was release more frequently. So I think overall it's similar to testing and similar to ops in the sense that if you refer back to the book, the goal and everything else, it's like we're trying to streamline everything and with security it's like interesting because. And with testing, there's just some things that you cannot add automation to. Right. Like in my opinion, it's becoming easier. But as far as it being involved with like a delivery pipeline, yeah, it's absolutely necessary. There are so many aspects of it, you know, that you just, you want that immediate feedback. You know, if you're like running a containerized environment, you want those containerized scans, you want to limit your base image. There's all these scenarios that can be automated with some free tools, but primarily like, you know, you're paying for some of these things. You know, AWS is offering a lot of these now, which you pay for. But I think to answer your question though, it's, it's one of those things that is much needed because it's, you know, we don't want to fall into the scenario where we do a deployment and then you, you know, you just throw it over the wall and then, you know, you have to wait. So like, you know, you're still dealing with competition. It just depends on the industry as well. Like I think it's much easier to do in like the private space. But for some of the, some of the contracts I've done for the DoD through another company is much, much, much more sophisticated. You know, you're very principled approaches to these things. So it just depends on the industry. Same with healthcare as well. Like I don't have experience in healthcare, but it would without a doubt be there. There's more compliance requirements.

Cool. So you know, in the pre show you mentioned how you started to focusing on security or building out security for your organization based on a recent security incident that took place. Are you able to talk about what that security incident was or is that confidential?

Yeah, I mean I can because CISA reported it publicly. Like basically what's happening right now is there's an uptick in, there's an uptick in DDoS attacks and you know, just denial, distributed denial service attacks across businesses and probably critical infrastructure and whatnot. But that's totally different thing. Know that most of that stuff's air gapped anyways. But getting to the point, CSA released something last, it was like last month where you know, they're basically saying like more of these organizations are just getting targeted for these types of attacks. Regardless of, you know, who's doing them is up to question because you know, it's just unbelievably hard to track that kind of thing. There are Ways, but it's just one of those scenarios. But yeah, that's kind of like what led to this initiative. Just to kind of formalize it and kick it off and make it a thing. Right. Like before we were just, you know, the idea was, I mean, same goes for testing too, like in ops. Like the idea initially was like we wanted developers to do everything right. Businesses love that. That's, that's the ideal scenario, but it's just not, it's just not realistic. And over the past couple of years, we've implemented scenarios where we do certain things and like I said, secrets and you know, rotations and all of the things with. Yeah, basically all of that. So that led to us formalizing it and just kind of acting as like a security group. And then within that group we define the standards procedures, some work that we do. So what that looks like specifically, we have some KPIs, we have some metrics, we're formalizing some of the tasks and epics associated with it. I think we're like debating on doing Kanban or just standardizing it. Like the rest of the teams, they're just doing like a scrum approach to it, taking on a couple of tasks, distributing some of the other ones that are more relevant to some of the other teams. So yeah, it's like a whole team approach in some sense. But at the same time there's like some things you just, you can't. It depends on the team. Right. Some things are much, much easier. Like if somebody just has a core background in software engineering and you know, they build on over the years and they're like some super senior or something like that, you know, maybe they have a better understanding of that. But the reality is like, sometimes you need guidelines set up and everything from the software development process or the infrastructure to compliance that we're doing, you know, that's kind of like the idea. So. And that's actively being built out.

Nice. So you're kind of like a superhero of getting things started. Like every time I talk to you, it seems like you're starting on a new, new initiative, getting buy in. So how do you even know where to start? Especially like with InfoSec, I assume it's not an easy thing, especially if you don't already have one in place.

Gotcha. Yeah, I mean, it's one of those scenarios where I had an initial background and interest in it as a kid, you know, basically, which led to a different career path for me. Like I've done so many different things in software, but getting back into security was a matter of. It's been on and off for me over the past couple of years. But if you're just starting out, my advice is to honestly like look into, look into the security plus, like if you're interested, depending on, also depending on the type of work you're trying to do, if you're trying to do like DOD work, like this is one of the baseline requirements, you know, like it's just going to teach you the fundamentals and foundations that you need to succeed in, you know, as a SOC analyst or you know, whatever you want to define it as. So I think there's ways to it. There's of course so many different books that you can get as well. But like I think a good foundation for people who are interested in cybersec and all that whole realm is a security plus. I think it's a pretty decent certificate. And of course certificates are questionable across the industry. People have different opinions on them and whatnot. But I think they're pretty solid for people who are doing infrastructure and security. I think they're pretty good. It's a decent foundation for people to basically showcase, hey, this is what I can work on, this is what I'm tested to know and these kinds of scenarios. I don't think certificates are good for programming languages. I think that's more of an interesting thing for open source. So just open source in my opinion, for that kind of thing. But yeah, that's how I've gotten into it, how I've kind of been in and out of that realm just by helping other organizations and internally within the company work on some of these things and making suggestions and you know, it becomes much easier to push these processes along when an incident happens. You know, that's just I think a lot of a reality of a lot of organizations. Unless you're some company, some brand new company and you're hiring like a ciso, like for maybe for like compliance or legal requirements or something like that. It's usually one of the scenarios where like you can push a lot of compliance stuff off. Like if you're using Stripe, you know, you can reduce that PCI compliance requirements and logging retention and all these kinds of things. But eventually I think it catches up to you. Like the bigger the organization gets, I think it comes back to you.

Absolutely. And it's a good point about certs. I know in testing it's not looked down on, but it's not a lot of value put into it. But with security, every time I talked with security folks, it seems like certifications is not only helpful, but in some times it's almost required to even get a job. You need a basic certs for certain things for sure.

Yeah, absolutely.

So you know, besides that, you know also with testing sometimes it's hard to get buy in from business leaders because they see it almost like a cost center. Do you see security as the same thing or do you when you talk to higher ups like the C suite that they know automatically, hey, security has to be an investment that the team makes. Yeah.

Once again I think it depends on the organization but based on what I've seen, it's always a matter of, it's always a matter of cost. You know, like is it going to cost? You know one the general consensus that I've seen from like business leadership and whatnot is like oh, the developers can do this too, that kind of thing. Right? Like in up to a point. So there's that. So like there's differing opinions of it but I think for the most part it's just a scenario where you know, it becomes a requirement. So that's just kind of my thought process of it. Like it's a necessary thing, it's a necessary evil. Unless like there's multiple ways to address it such as various insurances that you can get and all these kinds of things. So like there's, and that's what I'm doing right now as well is just that cost to risk analysis and that's a huge thing. You know, that's more of the high level stuff where you have to identify like what cost you want. Like do you have to buy all the tools, are you rolling your own, are you developing your own, all those kinds of things. You know there's really, and that's what I've been dealing with lately is this justification of, you know, here's the tool suite that we want in comparison to a bunch of other fairly well known tools that are out there. Do we do this, you know, can we get this kind of funding for it? And I've gotten a couple of items across the finish line for finance, AWS Guardduty being one. I think Security Hub being the next. Like there's all of these out of the box. Like depending on what cloud provider you're using as well, sometimes there's out of the box solutions which makes it much, much easier. Unless you're going to like Lacework or Sysdig or something. And these are Lacework particularly is going to be more expensive. So yeah, it's a real justification of like hey, here's the business impact of what could be lost, especially when it comes to something like compliance or even just reputation nowadays it's one of those scenarios of just like you don't want to be on the front page of TechCrunch or something. It's one of those situations. Yeah, that's what I think. It's a balance and it's a, it's a hard balance and you need the right people making, pulling those levers and making those decisions to, you know, just justify it. Honestly, that's a big one. How do you justify it?

So no, so you know, when we talk about something being testing code, it needs to be built to be testable and I would assume for security code needs to be built to be secure. So as it's being developed it needs to be thinking about security. But it sounds like from you you're saying we've put so much on our developers that it may be too much. How much security should developers be involved in then? Or should they be thinking about it at all when they're creating their code?

Yeah, that's the goal. Honestly. One of the goals is to get, I mean there is a whole team approach to it, but within limitation you don't want the whole team to have access to Okta and secrets management and all these kinds of things. You got to worry about all these different types of rotations and whatnot. But yeah, absolutely. So there's an aspect of infosec where you could bake some of that in and if you're using a framework, if you're using the. NET framework or any other framework really, there's a lot of built in functionality to handle security scenarios and issues. So that's a big piece of it. So there's that, I mean you could understand, I mean depending on what language you're writing, if you're writing in C or something like that, which like I said last time I wrote C was skool. But the idea is that there's ways to exploit it, right? And that's why the Linux, I mean all the packages and everything else were just so exploitable. And to relate to that. Well, one, there's different ways to address it. Like you can do container scans and you could do advanced, like GitHub advanced security which we use. So you can identify specific issues and secrets in the code and there's things like that. And then there's the thing with Kubernetes as well where you know, if you're, you're using Kubernetes secrets, but it's all encoded in base 64. So it's all accessible. Ultimately you can decode it, right, like. And that's why a lot of that has been pushed. You know, I know Google Cloud was pushing a lot of that and Hashicorp was pushing it at zero trust and there's a lot of marketing efforts and whatnot. But there were some, there's some real legitimate scenarios playing out there. So. Yeah, but I think you can bake security into the code up to a point and then you're just going to be dealing with dependencies. So now you're going to be dealing with, you know, if you're running a REACT service like we are, you know, importing all these different dependencies and now these dependencies have, you know, CVE is common vulnerabilities and all that. And now you get to deal with that and you have to, you know, debate on whether or not you justify using an S BOM and, you know, all of these different scenarios play into that as well. But I think there are ways to bake security into the development process as well. Part of the DevStock Ops approach and then more recently, some work I was doing for contract was an organization that I was helping in the acquisition process of migrating items and whatnot. But what they focused on was they focused on Linux packages and how so many systems, you know, across the world really rely on these packages, you know, and there's vulnerabilities in the packages, you know, so you're ingesting all these packages and everything else and they could be exploited. And the idea was to take these packages and scramble them, you know, so you're doing like binary bit shifting and everything else so that if, you know, it just reduces the attack surface. So like, the idea is that you're secure by default, you know, like, that's super interesting to me. It was a really, really fun project, still somewhat ongoing, but yeah, there's lots of companies have thought of these things. I know sboms are a pretty big one, but yeah, that's. I think just overall we can bake some security into applications and I know Rust has been pretty big on that with memory utilization and all the safety concerns, you know, so there's languages that are being worked on, there's frameworks that are being worked on to do it. There's more and more things to make it a reality, which is nice because that's what we need.

Absolutely. So when you're developing software, you know, a developer thinks about, how do I get this to work? A tester usually puts on the hat, how can I make this not work or break it. And I guess as a security analyst you need to think about how do I attack this as an attacker. So with your background in testing, can you talk a little bit more about how would MIA helped you with threat modeling or any other type of concept that makes you a better security attacker or thinking like an attacker?

Yeah, with testing, I guess like the best, best thing about testing was especially like attempting, I mean one attempting to find the issue. You know, it's, it's super similar. You know, like you're in this mindset, unfortunately you get in this mindset of what can I find that's bad? Which isn't the most ideal scenario. So you have to kind of like keep, maintain this positive outlook out of things as well. But at the same time you have to think, you know, you have to think of what to find wrong, you know, and in some sense you have to think like, you know, a criminal in some sense in the, you know, where you have to identify the major issues that could take place. You know, what is interesting about this company or this service that could be beneficial to somebody that's not, that's an, you know, external threat basically, or hopefully whatever you want to call it. Yeah, just in general, I think the idea of testing is, helped me identify the issues and it helped me in that process of understanding the context. So of course like you can do this context driven approach with infosec as well. Understanding the system, doing reconnaissance, all of these kinds of things and then figuring out, I mean you can do it from a black box perspective or you know, it's much easier when you have the code. But you break that from infosec perspective, you break that into, you know, red team and blue teaming and that whole realm of things. So you can do active game days and whatnot, which is super similar to like SRE as well. You know, you have like a game day scenario where you can intentionally break a system so you can do like chaos engineering. But a lot of it's just fluid across software in general. There's so many different things that are similar that each of these organizations might be tasked with and might be more familiar with. For example, like Infosec might, you know, they might know NMAP and they might know they might work with Ghidra depending on what you want to do. So there's a lot of that. So there's overlap. But then I think testing helped out a lot. Like if anyone's a tester out there, it's just one of those scenarios where like, if you break into infosec, it's your mindset is there to find the issues.

And do you see security as being more in demand now than just general software testing? I guess it all depends on your area and where you're located. Maybe.

Yeah, it's an interesting one I think I'm not since I'm so new to that space. I mean I can't speak to it as well but I can tell you without question that there's a demand.

Across.

The government and potentially even state organizations as well. And going back to the certs, one of the requirements for the duty is like security plus. So for those of which were interested in Cyber Command or whatever, whatever, it's something to check out. So yeah, I think there's a major, yeah, it depends on the organizational size. So if it's a large organization, medium sized organization and they have like income and revenue and they have usernames and passwords and emails, like there's an aspect of security that needs to be handled there. But yeah, I think it's in demand. I don't know if it's. The issue that I'm seeing right now across the industry is there's demand and then there's this just emergence of these roles, right? Like and it's unfortunate because like there's these senior people and now there's this expectation of like oh you, you know, you know security, you know, you know how to handle ops, you can work on kubernetes, you can do all these different things. And I will say it's becoming easier because you know, like more recently with like guardduty you run scans on across AWS identifying like malware issues, right? Like there's those scenarios and then if you have a background in development or whatever you can, you know, which I did like you can take Python scripts, Python 3 Lambda and you know, essentially write it so that it automates that process, right. It runs automated the manual scan so you can actually get the issues that you need to. Or going back to the DDoS you can do AWS, WAF and you could write that Python to automate, turning on a specific rule that might affect, negatively affect other users who are legitimately accessing the site. So there's scenarios like that. But yeah, I think that's an issue where organizations demand too much. But we're getting to a point where it's becoming a little bit easier for people who might not be familiar or have a deeper understanding of some of these things to get involved with the process at least, hey, like run a scan or understand this tool to do these kinds of things. It's what we used to do with a lot of people who are more focused on like exploratory testing as well. You'd write iterate, like a test suite to wrap around something that makes it easier and more accessible to everybody else on the team. So you're doing like tool building, which I've done a lot in my career now.

I love tool building, so. Absolutely.

Cool.

So we talked a lot about like baking in creating a system. Once you have kind of a system in place, is there anything people can do to monitor their efforts? Like any KPIs or metrics that are good for if you're trying to really push a security effort?

Yeah, some of the stuff that I've kind of. I mean, there's a whole bunch, right? Like there's an entire confluence doc of KPIs and metrics that I found across the Internet and having talked to friends and just having a background in it. And then I took those, reviewed it internally, identified some specific ones to start with. So I think there's like, you know, you can start with a bunch, but the idea and the goal is to start with just a couple, you know, and then I took those and I did a talk. I think it was like maybe a month ago now, like, we're still so early in these stages. And I, I basically reviewed some of these things that had a high level with like, you know, potentially experts and whatnot, and just got feedback. That's what I'm interested in doing these podcasts and talks as well is getting that feedback of, you know, what I'm missing, what we're missing as an organization. Right. So I think that's pretty beneficial. But as far as like specific KPIs and metrics, some of the ones that I took from like SRE even is like meantime to detection, meantime to recovery. I mean, that is even arguable. Like people have argued about MTTX and saying you should just use SLOs and they're useless and all these kinds of things. But I think for certain applications it's very beneficial. Like, you know, being able to identify like, particularly with our incident, you know, it's just an embarrassment where I was on a call for hour, two hours until we could figure out what actually happened. And you identify like some collection of root causes or in this case a root, actual root cause, which call it cause the problem. So, yeah, pretty interesting. But yeah, I think that there's some pretty decent ones to start with incident response. So, like, how long which, you know, related to MTTD and whatnot, how long you take to recover, how long it takes to detect a particular issue, which might be difficult if malware's been sitting on your servers for years and all that kind of thing. And then I think the other one we started with was automated cases created, right? Like, we have systems internally that find issues or detect issues potentially in like a cloud platform and automatically create those tasks. So that's a metric we're looking at. I didn't want to do a metric of like, something like bugs found or something like that. I think in our case, like, incidents found per month might be interesting. But if it becomes like, that's more of a metric than a KPI. Because I don't want to fall into a scenario where, you know, we have a KPI and it's like, you know, it's required for our organization. And then people like, there's just so many bad situations that can happen out of that, and particularly like bugs found, people can lie. And it's one of those situations that I've seen before as well. So, yeah, those are more so metrics. But, yeah, those are a couple that we've worked on that we're working through. We're still drafting them, we're still building them out. There's a whole bunch out there. Even like, like with compliance, like, you know, depending on what cloud service you're using, you could utilize a metric as you could use. One which we're going to do is like the AWS security benchmarks. You know, something super simple, right? Like you can enable this in Security Hub and it gives you a percentage. And maybe your goal for the quarter is to get like 80% of that. Or like, maybe you're, you know, if you're PCI compliant or whatnot, you can get some percentage to that. So there's different. Those are some other metrics and KPIs you could use as well, but those are a few that we've used that we're kind of drafting and working on and work in progress.

Awesome. Awesome. Okay, Evan, before we go, is there one piece of actual advice you can give to someone to help them with their DevOps security efforts and what's the best way to find or contact you?

Yeah, for everyone out there, I think one of the things is feel free to branch out, you know, if you're bored in your role or, you know, if you, if you want to do something different, branch out, you know, ensure, you know, you make the time for it. You know, like, that's just what I've done over the years. You know, you could identify, you can make the time and put in an hour, a day or whatever it might be to learn something new and it could benefit your your career and completely change your life from, you know, many perspectives. So you can find me all over the Internet, but I'm on YouTube at random thoughts Tech. That's primarily where I'm going to start posting some more items.

So and for links of everything of value we covered in this DevOps toolchain show, head on over to testguild.com P123 Remember, latency is the silent killer of your app. Don't rely on frustrated user feedback. You can know exactly what's happening and how to fix it with Bugsnag from SmartBear. See it for yourself. Go to Bugsnag.com and click the free Trial button. No credit card required. So that's it for this episode of the DevOps Toolchain Show. I'm Joe. My mission is to help you succeed, creating end to end full stack DevOps toolchain awesomeness. As always, test everything and keep the good. Cheers. Hey, thanks again for listening. If you're not already part of our awesome community of 27,000 of the smartest testers, DevOps and automation professionals in the world, we'd love to have you join the fam@testguild.com and if you're in the DevOps automation software, a testing space or you're a test tool provider and want to offer real world value that can improve the skills or solve a problem for the Guild community, I'd love to hear from you. Head on over to TestGuild Info and let's make it happen.