DevSecOps Blind Spots with Wilson Mar

Welcome to the Test Guild Security Podcast where we all get together to learn more about security testing with your host, Joe Calantonio.

Hey, it's Joe. And welcome to another episode of the Test Guild Security Podcast. And today I have a special treat for you because we'll be joined by Wilson Marr and he's going to Talk all about DevSecOps around a bunch of different topics of security activities he's recently been working on. If you don't know, Wilson Marr has been building and bringing enterprise applications to market on major platforms from mobile to server clouds. As an architect, a developer, he's been a performance tester and a manager and now he dipped his toe into the security testing realm. So really curious to learn all about that. And he's one of the first folks I've ever knew about that had a site dedicated to any type of testing. Wilsonmar.com I think he recently ported it over to Wilson Marr and at GitHub IO. So you definitely want to listen to this episode. Has a lot of experience around all types of testing. You don't want to miss it. Check it out.

It's no secret that the number and complexity of applications are growing. But what is alarming is that 90% of security incidents result from exploits against defects in software. The need for better application security has never been higher and Micro Focus Fortify is here to help. Fortify is the recognized market leader in application security and it is the most comprehensive and scalable application security solution that works with your current development tools and processes. With Fortify, you can start security applications in a single day, including custom code, open source or commercial components and scale as your needs grow with an on premise as a service or hybrid implementation. Check them out. Head on over to micro focus.com Apple Security Hey Wilson. Welcome to the Guild.

Thank you. I'm very honored. You have such great hosts, you know, such a great set of guests that you have. And so it's a high bar I have to meet here.

Well, I really appreciate it. Like I said, you're one of the first folks I ever followed. I really was into testing. So your site's been around forever. It's been a great resource for me. So thank you. But Wilson, before we get into it, for the folks that don't know, is there anything I missed in your bio that you want the Guild to know more about?

No, that's about it. I'm kind of like addicted to the website. Do that instead of watching porn or something or having some kind of a sports hobby.

Cool.

Yeah. I enjoyed it because I just basically, you know, took notes for myself. And pretty soon I thought, you know, people found it helpful. So I'm glad for that.

Yeah, I love it. So, Wilson, another thing that I mentioned is you've been doing all types of testing from at least as far as I know, like the 90s. So new topic that maybe you've been involved with before, but it's the first that I knew about, is, you know, you've been involved in functional testing, performance testing. But what got you interested into securities testing?

Yeah, because it is the issue that's really bothering us. Right. I mean, because, you know, it was one thing to be slow, another thing to not quite work is quite another to get ransomware, you know, notices. Right. So I think that that is really the biggest concern I think a lot of companies have. And so I got kind of dragged into it, you know, to say, hey, you know, you've looked at this. What do you think of, you know, these things? And I would just horrified to see, you know, how some organizations leave big holes in their. Their system. Like, I was this one outfit, you know, they didn't even have like, the TLS setup where, you know, they were like, they weren't using a TLS one three, which is the latest now, and they just, you know, they just only had like 11 and 1 2. And. And so that really is big, huge thing if you ever looked at the concerns that people have about being able to get snooped. And also, the more I looked into it, the more I realized that it is, you know, that's where the money is. Right. So that's the reason why I kind of pulled into that kind of stuff. And so over time, you know, I've learned a lot about protocols. And so I took the Security plus exam few months ago and was able to pass it, and. And so that kind of nailed it as far as know what I can do.

Nice. So, yeah, what is the Security plus exam? Do you recommend it for other folks then that want to get into security testing or security in general?

Oh, yeah. It's in fact, I think that there was a project that I did in federal government, and that was one of their prerequisites to get in to do some government projects. And so that's a big thing. So I think. And it's a tricky exam. It's not really that everybody says it's so tough, but I think the issue is just that it's tricky. And so I found some sample exams and just kind of went through it just out of curiosity and you know, as a performance person, we've had to deal with protocols a lot of times and we've always had problems with encryption. Right. In terms of how long they take and so forth. So it was, you know, that wasn't something that was totally foreign to me. So it was, you know, it was good. One thought I had for today, Joe, was like some of the blind spots that people have in DevSecOps specifically like around metrics and what they have to do to get breakthroughs of the in DevSecOps. I don't know if any of you other people have looked into that, but that was what I'm thinking, that we can take a look at that.

Yeah, absolutely. Before we dive into it. Like I said, I think a lot of people listening to this may be beginners to security. I know I am. So let's just set the stage. How do you define what is DevSecOps? Because I think different people have different interpretations of what it is.

Yeah, that's a good question. And really we do need to identify that. And my biggest go to is of course Gene Kim. He's no relation, but he is a just basically through his DevOps handbook Phoenix Project and his work on Accelerate, they also have this, if you go to it revolution.com for forum paper access, they have a group and he lives in I think Portland and he gets them together, a group of people together every year and they write these papers. And I think that is a huge effort that not a lot of people know about. He's been doing it since like 2016. But the biggest thing I think that his contribution has been is not just in popularizing the word. And you know, he speaks a lot of conferences because he's published. But I think the biggest thing that he's done with his team is this DORA report D O R A and it stands for DevOps Research and Assessment. And what they've done is to talk about the correlation between performance in terms of the, you know, financial impact as well as achievement of availability and against these metrics, these four metrics that he's got. And I think that's been very helpful because then people, you know, know that, okay, you know, we've got uptime we have to get to. We have this recovery point objective that people need to define in terms of how much data loss are you willing to give up after recovery. And then this recovery time objective, which is really the same as the MTTR mean time to restore and a few others. And so I think those metrics are really been helpful because now people have a smaller number because usually you know, you've got some management, they come in and they have like 15 different metrics, right that come in. And so he, his group shortened the list to like 4, 5 if you already count availability. So that's to me is this kind of like a summary of the whole situation. The you know, it's time to restore how much lead time that you have from codecommit to running in productive use, not cycle time. By the way, you know the difference between cycle time and lead time it cycle time is an internal calculation how much time it is for an internal process. Lead time is counted in terms of the customer viewpoint. And I think that's really the key to effectiveness for DevOps and one of the bind spots as well because being looking at the customer is not just an agile concept, it's really a good business factor. And so it's easy to forget about that kind of stuff. And also deployment frequency, how often code deploys to production. And the flip side of that, the last one here is change failure rate. Now you would think okay, change failure rate. It's just you want to keep measuring my failures, right? It's not like your mother in law. There is a balance between I think change failure rate and deployment frequency and also lead time. Because if you squeeze people to deploy faster and you want to squeeze people to have more lead time, I mean shorter lead times, shorter time to restore, you need to also accept some difference in change failure rate. So the issue I think for us is how do we achieve better results in all five of these, you know, in order to get good uptime, not have as many change failures, you know, have lowered lead time, higher deployment frequency and shorter time to restore. And I think that's really the game really in terms of figuring out what to do. And I can go into a couple of things that I've seen in terms of what people do to achieve that stability and yet have a faster rate. And that faster rate really is the key for not just DevOps but also for security issues. You know, how I got into this, this whole security looking is that you know, they're literally people staying up late at night, right, Trying to hack in. And you can only really, you know, have a recover from that in a not easy way. Right on in the whole issue with you know, ransomware and so forth. So you really got to keep the doors pretty well locked because all it needs is like one to ruin your reputation. Right. And so I think that is really frankly to me the Biggest driver for a lot of this work. This DevSecOps works because the faster you can prove that you can change things, the faster you can respond to any of these, what they call the CVEs or the vulnerabilities that they found. And so to me, that is really the key, is that you're not doing it just to squeeze people and make them work harder. But this speed is needed in order to respond. As soon as you have a published vulnerability in the product that you're using, you've got to move fast because by the way, if somebody finds an error, it finds a vulnerability. They have like less than 90 days before the government publishes and says, okay, this is a defect. And you have this situation called this, you know, the zero day vulnerability where there are no patches, right? So the time that it takes to create a patch and all that, it's really crucial. And I would think that, you know, and I, that's what I do when I evaluate a product to where we bring in, the first thing I do is look for vulnerabilities that they've had over time. And this national vulnerability database, you know, maintained by Mitre and the US Government would tell you, you know, for every product, you know, what is their history of vulnerabilities that you found. And if you see that, you know, there's a bad trend or, you know, things are not attended to, then that's not a very good vendor, you know, to do. So that really is the, you know, it's the issue, I think today.

Is it the software itself or is it the libraries that they may use from GitHub or things like that you flag as being security issues?

Oh yeah, it's both. You know, we have this, there are these products, they call them the, you know, they would scan through the, it's called the component analysis, the software component analysis. And what that does is basically, you know, when we have, let's say a Docker image, it would look through inside and see, you know, any of the dependencies for it that they would say, okay, this particular library, does it have any vulnerabilities? And then it would look at the libraries it uses and figure out, okay, does that have any? So that it's kind of like a recursive analysis. And that's really important because, you know, what we do is that we run it every day, you know, to see any new ones that creep up. We want to be notified. And that way we can start work on, you know, avoiding it or fixing it, either way.

Great. So I think you Know, as a functional automation and performance testing, as you know, over the years, maybe it didn't get enough attention. So how do people know before we even get to blind spots, how did people even know that like they need security and how do they implement DevSecOps before we actually dive into those metrics a little bit more?

Yeah, well, you know, there's a magic word for testers called dast, Dynamic Application Security Testing. And inside there, at least in some of these products are really a kind of like selenium or like kind of situation or some kind of a functional testing tool or scripting tool that it would use dynamically. It would be used to run the application as if it was a regular user and then during that time then you would identify issues. So I think for a lot of automation people, take a look at the various vendors, I won't plug any specific ones, but there are several of them out there. Is dynamic application security testing would be, I think, a good follow on to automation because that's really needed in terms of automation in order to find out what's going on. And if you haven't heard from the security people, I would suggest you reach out to them and say, hey, you know, I've got a bunch of these automation scripts and maybe it might be helpful and I'm sure they'll welcome you with open arms.

Absolutely. And so Wilson, both those metrics, usually to get a metric you need some way to monitor it or to kind of use a tool to capture it. So for each one, like time to restore, is there a specific tool or technique you would use to capture that metric?

Yeah, that, you know, everybody has a different story. But the issue I have is that this blind spot I'm talking about is to avoid having the person who does the restore be the arbiter of the timing. Right. So for example, you would say somebody who says, okay, I started restore at you know, 1am and I recompleted it in 120. And so the time to restore is 20 minutes. Right. That's not the case. Right. What we really need to do is to look at the time between the error detected in production where a user can't use the system anymore to the time where the production is operational, again, meaning that the users actually have done something. So there again I think the functional testing people, end to end tests would be very helpful because then that way they can work with the monitoring system to get those statistics because again, the only metric that really counts is where the users come in. And so that way your Availability would be reality rather than just what happens internally.

Great. How about lead time? You mentioned that it's important to get it from a customer's viewpoint. How do you do that?

Well, for example, there's another situation. So the lead time team, they said, oh yeah, lead time is one day. And so between the time we commit to the time it goes into production is one day. And then I talk to the users and I just go, well, how's the early time? And they go, it sucks. And the reason was, is because the users look at it in Jira with time requested, the date that it was requested, and it was like three weeks or something. And so to the users it's three weeks. And internally, yeah, you may have good lead time, but you're not really satisfying the user. And so that's really the concern that I think there's a blind spot that a lot of people have reporting internal lead time for changes rather than the external perception of where things are.

Great. And the next one was, I believe you said deployment frequency. How does that impact security or what do people need to be concerned about with deployment frequency with that metric?

Yeah, well, you know, deployment frequency is important for performance because I was at this one place, early days with Java, there's like a full time group of just rotating, bringing servers down, you know, and bring it back up again just because of memory leaks, right. You remember those days. And so in today's world, right, A lot of times I ask them, I just go, okay, well do you recycle? You're like, oh yeah, well we make changes so often that they never, these servers never have a time to create a memory leak because they're up and down all the time for various reasons. So it's a self sustaining situation that they would have. But yeah, deployment frequency I think is important because the more you work on something, the more you kind of notice new stuff, right. And that really is really a key, is that, you know, you'd be able to know that things still are still working because, you know, you know, when you have a software product and it's two years old, you don't have it. There's going to be security patches that haven't been applied and all the dependencies are key towards a specific version. Right. And so if you get too far and not check those versions, it could be tough. And if more frequently you do it, the more up to date that you can stay. And having up to date security patches is the number one way you can avoid problems.

Right. And the fourth one I think was Change for failure rate. I don't know if I had that correct. How is that a blind spot again?

Well, the situation is that you would have different levels of failures. You can have a build failure issue or application test error. But you can also like with Dash, you can have issues with that as well. And I think one of the blind spots that people may have is to have a separate organization for security testing versus a application testing. And so the push right now is to have the these security processes occur along with the test. Like for example, we would have a scanner run to check for passwords, for example and also a check for lint. Like for example, like if you have Python you have like PI flake8 as a 1 or pytest that we could run as part commit process or before the commit process. So that way you can see if there's any issue. You know, every single language has its own linter. Even like Terraform has its own linter from terragrunt so that you can run those even before commit. And it's so much easier to do it before commit because otherwise it's stuck inside that long history chain within GitHub that it's. Then you have to do a, you know, reset or you know, some additional issues that you would have when you try to remove things in history and you have to change the codes. And so it's really kind of a mess for everyone if you commit, you know, something that you've got to take out later.

Absolutely. I think one complaint though from developers is there's a it takes a while for these things to run. So does this add a lot of overhead? And do you see people opting out of running certain critical scans because of that?

Yeah, well, you know, take a break, drink some coffee. No, I'm kidding. No, you're right, it is. And I think if they can have a bigger machine, you know, that sometimes help and also running things in parallel and there's some new technologies out now where they call it interactive testing, interactive security testing where they are analyzing the code and incrementally finding things out. So the timing of these tests are reducing pretty dramatically over the last couple of years. You know, as they find AI that is that going to be employed and that kind of thing. It's improved quite a bit. I mean you don't have to do it. I mean you can have it this in parallel. Right. Is you can they start the process and then cycle through and then you can continue to do something else and something else is some other coding. Right. Because you can then Set it off as a batch process immediately and then you get the results a little bit later. But you can, you know, certainly, you know, do some things in parallel.

Awesome. So I believe we went over four blind spots. You said there was a fifth, but I don't remember what, which one it was.

So I don't. I don't either.

Yeah. So I think the biggest thing is that the idea that you can scan through and find things, I think the biggest issue really is figuring out how do you get the developers to know intimately about what they call CWEs, a common weakness, enumerations potentially in source code. So there's a number of tutorials and lessons that you have and if you run sonarq, for example, they have a complete set of, for example, java, they have 550 different areas where you can know about, but commonly there's only a few really major ones that you need to take a look at. If I were going to interview a Java programmer, or really any kind of programmer, I would think that, you know, asking those questions about, okay, you know, take a look at this code and what's wrong with it and for them to identify security issues would be a key thing. As much as I think a, some kind of a tricky memory allocation thing or something like that. I think that it's more important than them knowing many more of this, you know, tricky coding tricks. Right. Because not knowing these security issues, it's going to kill you. Right. I mean, it's kill the company. Whereas being a little slower, less elegant, it's not going to kill the company. So I think that security questions need to be an integral part of evaluation and training for these developers. And I don't think there ever been a place I've seen where they. I think there's enough of that. So like every, I have like a session every Friday morning to go over, you know, one each, but you know, with that I can only do like 50 in a year. Right. So it's really up to the individual developer to have a professional attitude towards understanding techniques for security.

No, that's a great point. And I was going to say how much do these scanners actually help? I mean, it may give a report, but like how do developers know based on the report how to fix it? And it sounds like it is an education process that if you want to do DevSecOps, you want to get it in, you want to shift it left like anything else. But in order to do that, you need your developers to be on board with the best practices and be aware of what is Secure code and what's not.

Yeah. And I think one part of it too is that you can prioritize somewhat. Just last month there was a report from Snyk S N Y K that I thought was really interesting. And because they do security scanning, they keep track of how many projects got impacted by certain vulnerabilities. And what they found was that there were some that were infected a lot of different projects, which is not a good thing.

Right.

You want to avoid that blast radius. And so what they found was this Object prototypes in JavaScript in Lodash, jQuery, this prototype, what they call the prototype pollution. And that was a huge problem because it infected so many different libraries that you know it's a problem. And you know what. The same kind of architecture is happening in webassembly. I'm very concerned about webassembly because you can inject stuff that you'll never know because webassembly is like, it comes in as a potentially as like a human unreadable code, which is what makes it fast. Right. And so problem is then you can have people injecting stuff potentially and malicious and be able to, you know, run whatever they want on your browser as a result. So I would be very, very careful about using webassembly based upon that kind of analysis. And. And it's not so much a individual CVE or whatever that comes in, it's because it's so many affects so many different libraries that it's an architecture problem that makes it very difficult to detect any problems, you know, that you have. And not just a particular product being written wrong. It just architecturally, there's like an open door in the situation. And so that's a huge blind spot I think a lot of people have with webassembly. So I'd be very careful.

Containers or microservices, newer developer approaches, they open to certain blind spots. Other type of technology and development is not.

Yeah, one thing is that a lot of places they would have like no authorization to get into the Docker Registry, right. So unless you work with like Nexus or Artifactory, they wouldn't have a certification or a way of authorizing people to change. So as a result, what could happen is that somebody can sneak in there and recompile and, you know, dockerize something and put it in there with your name for your particular image and it looks like the same name. Right. But you would never know. So a lot of companies now is, in addition to having authorization for rights of the Docker into the Docker Registry, they would also have this certification, what they call a notary, where every single image that's created would be stamped or it's. We would get a certificate in working with it. So and then they would pass this certificate over to Kubernetes. So and they would look for the image based upon the sha, you know, that they would generate rather than just the name itself. So I think that's the proper way to do this is to have a notary that would maintain these certificates and then you work with certificates. Right. And it's part of this whole overall what they call the Zero Trust framework where each application defends itself rather than expecting the firewall to save everybody.

Absolutely. I need to get you back on the show. I like to keep things about 30 minutes. So is anything you wanted me to definitely ask though before I ask my last question that you think?

No, no, that's fine. I'm just here to, just to say hi.

Awesome.

And yeah, I think one thing, I think I spoke briefly about the RPOs and RTOs and availability and I think that one of the blind spots that developers have in their teams is to not be realistically about identifying what it is. I think is power to identifying that because there's like a pressure to say, oh, you don't have to set, you know what the rto, RPO and availability is. You've got to really negotiate that up front by. Because the architecture choices that you make affects, you know, how quickly you can restore and how much time it takes. Right. And a lot of times you don't get that time to get it ready to go. And so as a result people look askance. Even though they've done well, they still fail because they didn't negotiate properly the availability and the SLOs ahead of time, you know, with their management.

Okay, well before we go, is there one piece of actual advice you can give to someone to help them with the security testing efforts and what's the best way to find or contact you?

Yeah, I think the main thing is that people forget about easily is this user focus and going fast, getting into production as quickly as you can. And that really is really a key. I see a lot of people here still get stuck in this, this non agile or waterfall thinking is. So I would think that when you start an application, get into production along in parallel would be my suggestion. And so that way then you can test the architecture of the processes that keep you from getting things right in production.

Nice. And what's the best way to find or contact you?

Well, luckily they gave me a weird name so you just google me And I'm on LinkedIn and it's not Google. Wilsonmar GithubIO and I'm on Twitter as well so look forward to hearing back from everybody.

Thank you Wilson for your Security Testing awesomeness. Links to everything of value we covered in this episode. Head on over to testguild.com S20E and while you're there, make sure to click on the Try It Today link under the Exclusive Sponsor section to learn all about Micro Focus Fortify, which is the recognized market leader in application security. So that's it for this episode of the Test Guild Security Podcast. I'm Joe and my mission is to help you succeed with creating Full Stack Security Testing Awesomeness. As always, test everything and keep the good. Cheers.

Thanks for listening to the Test Guild Security Podcast. Head on over to testguild.com for full show notes, amazing blog articles and online testing conferences. Don't forget to subscribe to the Guild to continue your testing journey.