Developers are Taking Over Application Security with Jeffrey Martin

Five, four, three, two, one.

Drop that.

Welcome to the Test Guild Security Podcast. We're all getting together to learn more about security testing with your host, Joe Calantonio.

Hey, it's Joe. And welcome to another episode of the Test Guild Security Testing podcast. So, as we enter in the new year 2020, I thought it'd be a great idea to go over the state of application security and how developers are getting more more involved with security efforts as more and more companies shift their security testing efforts left. And I think it's a trend we'll see more and more of in the coming years. So we're going to go over some of the reasons why application security is a top priority today for companies that are delivering software, especially as we head into the new year. And Jeffrey Martin will be joining us from White Source and we're going to actually be going over their report. Developers are taking over application security. You don't want to miss this episode. Check it out.

It's no secret that the number and complexity of applications are growing. But what is alarming is that 90% of security incidents result from exploits against defects in software. The need for better application security has never been higher and Micro Focus Fortify is here to help. Fortify is the recognized market leader in application security and it is the most comprehensive and scalable application security solution that that works with your current development tools and processes. With Fortify, you can start security applications in a single day, including custom code, open source or commercial components and scale as your needs grow with an on premise as a service or hybrid implementation. Check them out. Head on over to micro focus.com appsecurity hey Jeffrey. Welcome to the guild.

Hey Joe. I love that, love the noise. Makes me feel very welcome.

Cool. Awesome. So it's great to have you back. Last time we spoke, I think it was way back in 2015 on my automation podcast. But now it seems like you've pivoted over to security testing. You're now the director of Product at White Source and I thought we'd talk about actually security and specifically about. I think White Source recently came out with a report on developers are taking over AppSec. So I thought we'd dive into that a little bit. Jeffrey, before we get into it, what have you been up to since 2015 since last time we spoke?

Yeah, it's been a while. So I did stick around in automation for a little bit. Jumped over to the other side of that coin. Security and privacy, actually. But security, especially open source security, is what I've been focused on for the.

Last Bit very cool. So I'm assuming this is a brand new show, but I think more beginners, people just getting familiar with security would be listening and I think I would have episodes targeted to towards beginners. So when you made that switch over, was there any similarities you used from automation, security or like what did you do to get yourself up to speed with security testing that was different than maybe you were used to with functional testing?

Sure. So it's actually interesting because I think security is going through the same phases that we saw with automation, functional automation before. It's just coming up a little bit afterwards where it used to be manual testers and they were automating and then integrating into their pipelines. We're just in that phase now still of taking security and treating it as another piece of code, actually automating it, integrating it into the pipelines and really embedding it. Embedding it into the. Your model, essentially your development model. All of that was, I don't want to say it's done with automation, certainly, but it's much more mature as an industry than security testing is. So for me, it was a way of taking a lot of the lessons I learned from helping people with those transitions in automation and now helping people with a very similar transformation in security.

Awesome. Now one of those transformations you might be thinking of, and I thought of right away, is shift left. Seems everything has been shifting left. It's a getting overused term now. But security testing, just like performance testing, I think is lagging behind functional testing. So when you say transitions, is that one of the transitions you see happening recently now in the industry where people moving security more and more into the development lifecycle?

Absolutely. And actually it's kind of funny because oddly, security can actually be shifted even farther left than functional testing because so much of security does revolve around which components you select to use from third parties. So some of our own research shows that around 80% of all code, if you were to look at just a given product's code base, is actually comprised of open source or third party components, only about 20% and in some cases much less is proprietary. So because of that you can actually in a way shift security so far left it's even pre code by seeing if the component has known vulnerabilities before you even download it.

Oh, interesting. So how does someone find out if a component has known vulnerabilities literally before.

It touches your code? So it's quite hard to get more left than that.

How's that done though? Is there an index so there's several.

Ways to do it. Kind of the way that I hope most of the audience would be at least beginning to do this is literally looking, looking at the repo for that particular, let's say open source project, seeing if there are known vulnerabilities and that you're using the most recent version. It sounds almost insane, but a lot of times people bring in older versions, you know, by accident or because that's what they used last time, or because they're copying a different project. So it starts with using the most recent version, but then once you have that, you can look inside those repos, actually go through all that manual effort yourself, take a look at the nvd, the National Vulnerability Database, and seeing if there are known vulnerabilities in that code. Or you can use a product and automate that for you. And actually several products will, even before you implement it, give you that information inside your IDE or inside a browser plugin. So literally when you're opening up the repo to look at it on GitHub, it'll just pop up and say, hey, there's known vulnerabilities here.

Basically what happens is a team gets a feature, they're going to work on a new feature in the Sprint. You're saying the first thing they could do then is say, okay, is there an existing library out there I can leverage rather than write brand new code? If so, how secure is it before they even bring it in to start working? Nice.

Ideally they can even look and see if it's got a track record of tons of security vulnerabilities that they always have to remediate. Or if it's, hey, every time they release, they find one a little bit later and they fix it very quickly. You know, it sort of does go towards the quality of the open source community around that component. You know, they're not all created equal. Some are two guys who, you know, went to go solve a problem by themselves. Some are gigantic open source projects with thousands of contributors.

Another thing I see is who's responsible for security. Back in the day, especially with big enterprise, they have a security team, and the security team is a team outside somewhere else that does whatever they do. But on a team's definition of done, they always have security, but no one really tends to know what that means. So who's responsible for security?

Yeah, it sort of reminds me of the functional days here because a lot of times, you know, until the industry got mature enough to include certain quality metrics in their definition of done, it would be like, okay, and it has to be good, right? Functional, it has to work. But you know, performance is a good example of this and it has to perform well. Well, define that. The one big difference is, unlike functional testing or even performance testing to a degree, security never ends. Every component you use, more vulnerabilities pop up eventually. There's never a time where you say, yep, this is secure and we're done.

That changes a little bit about who's responsible for it.

To go back to what we were talking about a second ago. Once you select the component and you include it in your code, the maintenance and the continually monitoring of it to make sure that it is still secure, or when it inevitably has a new vulnerability that you are going to the latest version and making sure you remediating that vulnerability, that becomes almost a DevOps motion by itself in that it's continual and it never really ends. So it's different from quality in that regard where you can say, once this thing works, if we don't change the code, it's going to keep working. You cannot say that with security, you cannot change the code ever. And it will be unsecure at some point.

So I guess when folks check in and code then into their CI CD systems, a code analysis will run and it tells them whether or not they have code coverage. So you're saying on top of that, also in your CICD systems there's an option to do a security scan on code checked in and then alert them before it even gets committed.

Yes. And I guess to go back to the question, that used to be the function of a security team, but the problem there is that they tend to look at it very much like a legal team would. They, they get the code, they run their scans, which are usually big and intensive, and it rules out all the big stuff and then they put their stamp on it and they hand it back to the developers.

Meanwhile, the developers didn't learn anything from any of this. They maybe had to update a few things, but usually it's not that much of an effort on their part. And there's no attention paid to the.

Integrated CICD pipeline to for instance, every time you do a pull request, see what's going in on that and scan any new components. A lot of times the security teams were seasonal or cyclical. Right. You know, once a quarter they'd run a thing twice a year.

That's been shifting because, well, same old story, right. We're releasing more frequently.

That's compounded by the fact that in security, especially new vulnerabilities are found every single day. So it's not enough to just once or twice a year take a look, say, okay, great. Security team says our code's great. If you're shipping every sprint or adding new code every sprint, you need to be able to verify the security of your code every sprint. And you need to keep an inventory of everything you're using so that when new vulnerabilities pop up, you're automatically alerted to it and then remediated as part of your sprint.

So remediation, that's a great point. So one thing that makes people kind of cringe is functional testing is the false positives that happen. And so being new to security, I've been hearing a lot of people say it's kind of similar with security is that you get a lot of alerts and sometimes they're false positive. So how do you get the right amount of security being analyzed when code is checked in, but not doing it in a way that's creating all this extra overhead so teams know really what to focus in on if it's a real issue or not.

So it's interesting because a good chunk of those false positives have to do with the difficulty of deciding kind of which components are what. I know that sounds ridiculous, right? But if you take a look at inside your own code base, you look at a library you're using or a component you've an open source component you've grabbed, you could google it and find the repo, but often there's several repos or they are you forked it yourself and put it in your own private repo and now you're doing something different, matching what the components you have to where they're from and exactly you know what repository they're being used they're sorry, they're from matters a lot. Most of the false positives are misidentified libraries. This gets even more complex with some of the bigger packages which have many libraries inside of them. And suddenly instead of trying to match one thing, you're trying to match seven. But fundamentally a lot of those false positives are just bad matching. It's one of the reasons that inventory matters a lot. Once you've put the effort in to get a good inventory of your code, then it's just a much smaller problem of updating that inventory incrementally every time you add new libraries or packages. And that makes the false positives go far, far fewer because you've got that good, good baseline inside of your inventory.

So once you have that in place, you're monitoring the right things at the right level. Remediation, how do you prioritize? Like, I'm sure things fall into buckets with functional testing. You have just flaky tests or timeouts or actual assertion errors. Where I asserted this and I got that. So that's a real issue. So I know to prioritize based on that. Is security similar?

It is, but it's a little different in that the remediation almost always takes the form of updating something. Not always, but for the most part, it's fair to say that it takes it most of the time. Now the problem there is that code is not independent. Code is dependent on each other. You have a package, maybe there's a library inside that package that matters. Or two components. If you update one, it breaks the functionality and suddenly the piece you're using in another component isn't working correctly. Going through and figuring out those dependencies and what you can update is still a challenge. And sometimes it can be important to weigh the benefits of keeping everything up to date with does that break my actual code? Dependencies are the biggest issue in remediation. That's really what I'm trying to get to.

Since you started in security, how many companies are actually making security a priority Now? I know before it's always been kind of we do security, but no one's actually I hadn't seen really a big push in the security area, but for some reason the past few years I've been hearing more and more about security. Is this another wave you think that's happening in most companies now where security is being pushed up as a higher priority?

Absolutely. And I think that's driven by a few things. But I know our own internal numbers and surveys we've done is somewhere around 60% of companies view it as a top priority. There's still, I want to say there's 15% who said they only handle security after it arises. So a completely reactive approach, which is incredibly old school, just very, very not smart not to downplay anybody who's being only reactive to it. But at that point you're only reacting to a small piece and you're not preventing anything. It's sort of like dealing with cockroaches. If you see one, you have a bigger problem.

So being proactive, which is around 60% of people with a really high top priority matters a lot. But it's, it's a progress, it's a progressive thing where you have to actually ingrain it into the culture. The two reasons, from a business perspective that people are really driving. Security is one regulation keeps getting worse. Everybody wants, you know, you to have the right ISO certifications and there's an impact of security on things like privacy. If, God forbid, personal information gets out, you could be in violation of GDPR or a whole bunch of other fun regulations like the one coming out of California that can have a real monetary problem. The second is customers expect for you to keep their data secure. I don't know if that was as prevalent a problem in the days before these giant interconnected systems that we have now, because not only are you responsible for your own security, you're using other people's components. If you're using open source and you're interacting with a huge amount of other products. So it takes security from being the small problem of I need to lock all my doors to, I live in a hotel and the door has 700 people have a key to the door. And it takes it into making much, much a harder problem to solve.

So how much do you think GDPR has driven security to be pushed more to the forefront? Is that a lead driver you think?

For some, definitely. So I was involved in privacy for a bit and just fundamentally I think it woke up a lot of the corporate culture to the idea of, oh, wait a minute, it's not just a good idea from a PR perspective to, to keep people's stuff private. And I don't want to have a bad reputation. But it turned it into an actual legal problem which companies just handle much differently. Fundamentally, everybody becomes much more risk averse when your problems could actually be legal than if it's just reputation. And there's no way to be private if you're not secure.

So what do you say to someone when someone says security doesn't matter. I just create maybe iPhone games, iPhone apps. So security doesn't. Doesn't impact me.

They're wrong. I wish I had a nicer way to say that. Take just that example, the iPhone game. Presumably you're making money on it, but even if you're not, you're getting tons of personal information in one way or another. You know, whether it's a credit card information which has its own crazy regulations and PCI world, or if it's personal information because you have them entering their name inside of your app.

It becomes not just an issue with you, but all of the partners you trust.

It's safe to say you can trust Apple. But is that open source component that you've just used to render specific animations also siphoning off your private information? Or is it secure enough so that you can't, you know, somebody can backdoor in there and actually get more private information that way, or God forbid, financial information? There's no such thing anymore as an app that is not interconnected to a bunch of other related services. In reality, most applications nowadays are not coded. They're composed of other things.

And if you don't know what those things are that you're using and you don't know how well they're secure, you cannot be secure yourself.

Absolutely. And it's almost like a catch 22. It's awesome we have all this open source code we can use, but it's also not awesome because sometimes I guess people can put in vulnerabilities and security backdoors and things like that. So has open source vulnerabilities been rising because of this?

They've been skyrocketing and for two good reasons. One of them is a good thing. I think as an open source community as a whole, we are really prioritizing making them visible. We want people to know when there's a bug, we want them to know if there's a security issue, and we want them to know how to fix it. There's a downside to that though, which is if I'm a bad actor, the first thing I'm going to do, I'm not going to go find some zero day vulnerability nobody's found anymore. Let's be honest, most bad actors aren't that industrious, or let's just say technically savvy. The first thing I'm going to do is go and look at the gigantic list of public vulnerabilities that are out there and find people who have not prevented them. I mean, if I was a bad actor, I would see the open source community as a gigantic awesome free list of security vulnerabilities I can go attack.

Which shifts the onus onto the people using open source in their code to make sure that they have remediated as many of those public vulnerabilities as possible.

It's sort of not an okay excuse to say, oh, we were using an out of data library. It's literally a vulnerability that's known in the public, known to the bad actors, and has a known solution for the most part.

So not doing it is just the equivalent of leaving all of your doors and windows open and unlocked.

So what are some tools that maybe can help with this? As you know, back in the day with functional automation, I'm all about tools. Are there any tools that people can use to help them find security issues in an automated fashion.

Sure, I mean, there's several, and I don't want to get too shilly here, but obviously White source, we make a couple of tools for this in a few different ways, whether it's IDE plugins or scan inside your CICD pipeline or even after you get it out in the world, keeping your inventory up to date and letting you know when new things happen. But fundamentally, there's a lot of even free tools that are available to help here.

GitHub, for instance, has some great security alerts they just added. That's really, really helpful. You can look inside the actual open source.

Repo and you can see right there, does it have any vulnerabilities? What version do I need to fix those? There's a lot of really good free tools like that.

But they require usually some manual. You have to actually go to GitHub and look at the repo.

And those are good for people just starting out especially. But at some point it becomes like automation. You need to integrate it into your CI CD pipelines. You need to have it as automated as possible.

Nobody will go so far as to automatically update every dependency you have, because that could cause problems. But at minimum, you want a tool that will let you and your developers know, hey, here's an issue. It just happened. There's a new security vulnerability. Here's how that open source community says to fix it.

You need to have the knowledge. So for me, most of the automation around this is about detecting what you have, matching it correctly to the right repos and the right sources, and then giving you the information to actually understand what the problem is and what the right path is towards solution. Similar to automation that way. Right. It doesn't come back and tell you exactly which code to fix, but it does come back and come really, really close if you have good automation.

All right, Jeff, so besides what we've covered so far, is there anything on that report that White Source developers are taking over AppSec report that really caught your attention? Like, I didn't even think of that until after this report came out that you were shocked by or surprised by?

There was a couple. I'll tell you the one that was just flat out shocking to me and I was surprised at this was 30. About a third of our respondents came back and said, my company provides security training that actually helps me code better. And I cannot talk more about that and I cannot endorse that more. I'm shocked it was that high. I'm not sure if you know we ever saw that back in the functional automation days.

Right.

And to get a developer to say that a training helped them code better, yeah, to me that was absolutely shocking. Any company, you know, people who control budgets. Security training is something that has huge benefits and is. I'm not saying that we don't even sell security training. I'm just saying security training focused on actually better coding practices is a huge value, well worth whatever you pay for it. And I'm surprised at the actual quality of it. It's almost universally great. There are a lot of really good experts out there who have a lot of knowledge to share. And on that same vein, the other big trend that I'm really loving about, the ownership of security, revolves around basically people taking that security team that used to sit there and do all the security and turning them into what I would call player coaches, sometimes even embedding them inside development teams to spread out that knowledge of the best practices, the right ways to code, the right ways to scan things that you should be concerned about versus the pieces that you shouldn't. Taking that knowledge and spreading it out instead of centralizing it, just like we did with ops in the movement to DevOps is really taking hold in security and is a very encouraging trend. That security champion, if you will, model is really, really helpful.

Hf any quick thoughts on how teams can help get through all this data that they're getting now from security?

Security alerts, be mindful of the prioritization you have there because at the end of the day you may get a lot of results even if they're all accurate. I've seen people get 3 or 400 results on their first scans where they go, oh no, how am I going to handle all of this? Prioritization does matter and there's a lot of different ways to prioritize it, whether it's based on severity scores for the security or not. But. But if you're just starting out, make sure you have some idea of how you want to prioritize.

Okay, Jeff, before we go, is there one piece of actual security testing advice you can give someone to help them right away with their security efforts and let us know the best way to find, contact you or learn more about White Source.

Sure. So my one number one piece, especially for people starting out is start with knowing what you have. If you don't know that, you cannot figure out how to secure. Sounds basic. It sounds obvious, but you'd be amazed how many people don't even know what's inside of their software. All the different things they're using.

Once you start there, then you can worry about how do I find out the right vulnerabilities for these pieces and how. But if you don't have that inventory, if you don't have that knowledge, you can't.

Mitigate anything.

And in terms of how to get in touch with me, the best way is always going to be through my company website, whitesource.com but feel free to reach out to me. It's just Jeffrey Martin M A r t I nitesource.com thank you Jeff for.

Your security testing awesomeness. The links to everything of value we covered in this episode. Head on over to testguild.com sslikesecurity02 and while you're there, make sure to click on that Try it Today link under the Exclusive Sponsor section to learn all about Micro Focus Fortify, which is the recognized market leader in application security. So make sure to check it out. So that's it for this episode of the Test Guild Security Podcast. I'm Joe and my mission is to help you succeed with creating end to end full stack automation awesomeness. As always, test everything and keep the good. Cheers.

Thanks for listening to the Test Guild Security Podcast. Head on over to testguild.com for full show notes, amazing blog articles, and online testing conferences. Make sure to subscribe to join the Guild and continue your testing journey. This has been a Joe Calantonio production.