Do Your Pipelines Remember with James Rabon

Welcome to the Test Guild Security Podcast where we all get together to learn more about security testing with your host, Joe Calantonio.

Hey, it's Joe. And welcome to another episode of the Test Guild Security Podcast. As you can tell, I'm just messing around with the new intro outro. Let me know what you think because I'm having a hard time finding one that everyone seems to like. Today we'll be talking with James Rabin, a senior Product Manager at Micro Focus, all about his tips on incorporating static analysis tools into your CI CD pipelines. James also shares his vision of the future of using AI to help you with your AI static analysis efforts. This is a quasi interview where I took a session James actually did for last year's online Secure Guild and through the magic of audio editing, sliced it up and created an interview style episode out of it for you. I think you're really going to enjoy this. Check it out.

It's no secret that the number and complexity of applications are growing. But what is alarming is that 90% of security incidents result from exploits against defects in software. The need for better application security has never been higher and Micro Focus Fortify is here to help. Fortify is the recognized market leader in application security and it is the most comprehensive and scalable application security solution and that works with your current development tools and processes. With Fortify, you can start security applications in a single day, including custom code, open source or commercial components and scale as your needs grow with an on premise as a service or hybrid implementation. Check them out, head on over to micro focus.com app security. Hey James, welcome to the guild.

All right, thanks everybody.

All right, James, before we get into it, can you just tell us a little bit more about yourself and at a high level tell us a little bit about what you're going to share with us today.

My name is James Rabin. I'm the current Product Manager for Fortify. The presentation that I wanted to give you today is what I feel is a new angle on static analysis tools and the integration of those tools into the CI CD pipeline. So the title of this is do your pipelines remember? And they must if you want to go fast with static analysis. So we're going to focus on the remembrance side of static analysis and I'll explain in detail what that means and hopefully this will provide some additional information to you to help you effectively scale static analysis programs into CI CD development methodologies.

All right, cool. So James, could you just talk a little bit about cicd, what you mean by that?

Essentially with CI Developers are integrating changes back into a main branch as often as possible. They're validating these changes by creating a build, running a set of automated tests. And by doing that early and often, you avoid the problems that occur when you try to merge everyone's changes back into the release branch at the end. CI does put a greater emphasis on testing automation so that you can effectively move many, many, many commits into main with obviously your development team. CD is slightly less common. It's often CICD is usually rolls off the tongue, but continuous deployment is less.

Everyone's doing CI, not everyone is doing continuous deployment. Obviously there are a number of challenges with continuous deployment, but you need to ensure automation and you have to be quite certain of the quality of your automation testing to be able to deploy your applications automatically.

So, James, what about DevSecOps?

So DevSecOps, that's the correct buzzword I believe we're currently using, but we're just talking about building security testing into the development process. We all know that fixing security issues early in the lifecycle of an application is significantly less than creating or fixing these software vulnerabilities later in the lifecycle. Static analysis tools are probably the most popular to automatically detect software weaknesses or vulnerabilities in code because they can detect issues without actually executing the code. They don't have some of the same drawbacks that dynamic analysis has, where you have to actually have a running application and you know, you're providing a request and then waiting for the response and then doing that, you know, at scale. That often requires very robust testing environments. And so it's not surprising, given the ease, that you can add static analysis tools to a software build that it is. It has remained as popular. It was very popular 10 years ago and it's still popular today.

So what's the challenge then of incorporating static analysis into our CICD DevSecOps type pipelines?

The challenge with static is on the auditing side, right? Static analysis issues. It's a white box test. It knows everything about your code. At build, these results require developer time or security auditor attention. They may often require application security expertise.

They require understanding the security context that an application fits in. The easiest part of this whole thing is adding a static analysis tool into a build. And the hardest part is dealing with all of the results.

As we've talked about, static analysis tools triggered at the build run in the testing phase, right? And we'll get into some of the subtleties here, but that's where it's operating inside the CI CD pipeline.

So, James, can you give some examples of the results of a static analysis report and maybe where things could go.

Wrong if we look at some static analysis results? You know, with static analysis results, the majority of the quality static analysis results come from data flow, in which case, you know, tainted data is followed from an entry point all the way down to a potential dangerous execution point. So these, if you look here, like cross site scripting, persistent, you know, the untrusted data is likely coming from the database and then being sent to the web page itself. Meaning if an attacker can inject that payload into the database, we have a persistent attack that will occur every time that page is loaded. You know, when we talk about static analysis results, a lot of people the first thing that comes to mind is false positives. Even Atlassian, in their own independent assessment of cicd, mentions the value of static tools, but warns against the challenges of false positives. So with a very large data set that I'm about to share with you, I know that less than 5% of all security issues suppressed by a security auditor or developer were actual technical false positives. Meaning that the tool was incorrect. Right. That means that 95% of those issues that were suppressed were due to a lack of context.

Cool. So can you talk a little bit about the metrics that you use to put this session together?

Let's take a look at some of the metrics that I'm using to base my argument on. This is around 20 million total findings, 6 million of these findings were suppressed, and roughly 13 million of those findings were published, meaning they were sent to customers. This is a result of a static auditing service that I have access to the data for. So we have an overall suppression rate of 31% of all static issues, where 5% of those issues were actual technical false positives, and 95% of those were suppressed due to a lack of context. That is a pretty powerful number that I think you may or may not find. I think you may find that surprising. I certainly did. I assumed that the technical false positive rate would be slightly larger than that, but it, but it just isn't in a data set that includes, you know, over 20 million individual findings.

So, James, why are there issues that I don't necessarily care about if they are not necessarily, you know, real technical false positives?

If you look at raw scan results, these are all possible vulnerabilities. And then let's say you're auditing those results. You're saying they're critical, high, medium, low, not an issue. Whatever the case may be, what is the reason why static issue might not be an issue, right? Mitigation's in place.

Codes potentially not reachable. Could be a problem with the scan, right? That might be a problem. Reason organizational choice, right? Risk profiles of applications, right? If you are. If you have an internally facing application that is not exposed to the Internet, you are likely going to audit that application differently than one that is publicly facing, right? So all of these various reasons all lead to issues that an organization or a development team may or may not care about. And it is essential if you want to automate with static analysis in the CICD pipeline that your pipeline remember what these decisions are, right? This context needs to be recorded, it needs to be applied to the results and it needs to do so at scale past and for future scans. It's simply you're not going to be able to run build failure notifications, right? If you haven't built in a quality application context to those findings, you're simply going to fail every build.

So what are some ways that a pipelines can be used to prevent false positives from actually triggering automation?

Hopefully some of you. I'm a big south park fan and south park always had these member berries and they say I member. Do you member? So your pipelines must remember any contextual decision, right? To prevent false positives from triggering automation. You can't build automation around bad data, right? The better quality of your data, the more contextually relevant it is for the application that you're scanning, the better off you're going to be. Branch scans, right? We talked about CICD and builds occurring in a branch and then being tested before they're checked into maintenance. They must have access to that persistence layer of remembrance, right? It's going to do no good to run a static scan that is not merged with the baseline scan that has all of the contextually related decisions because you're going to get bad data, right? There are going to be contextually incorrect findings for that cross site request forgery. Let's think about that vulnerability where you're able to forge a request on someone's behalf. And typically static analysis tools would recommend, you know, including some kind of nonce, right. That are some kind of secret token with that request. Something that makes it really difficult for an attacker to forge a request on someone's behalf. Well, that's only contextually relevant if you think that it's. That an attacker may have some value in forging a request, right? If it's a meeting scheduler widget internal to your application, you probably aren't going to go to the effort to embed tokens with every form request. That's a contextual decision. The tool's not incorrect. You just simply do not care about cross site request forgery in this application.

So complex logic is required to keep track of issues that have been audited and as new issues introduced. Because code is a living breathing thing. What was on line 12 before is now on line 112. So there's a number of technologies that are used by various vendors to do that. And that's a constantly evolving process to ensure that we're not having to re audit issues every time we run a scan. Right? Because we have to have this persistence layer of issues that we don't care about, that we can do that every time. There are filterings and various other things that are used to help spread context. But really ensure that you're doing that well across your entire application portfolio. Every single audit decision is a cost, right? It costs either an application security auditor or a developer something to look at that issue, determine if it's a vulnerability and move on. That has to be persisted and it has to be checked against every single scan, regardless of whether it's a scan of main or in the branch. Right. If we want to find these vulnerabilities early and we don't want to waste developers time with a bunch of garbage, right. We need to query the persistence layer and we need that full context and we need all of our past audit decisions.

So can you go over some examples of static analysis automations?

So you can't automate if you don't remember? Right. So the automation side of static analysis on the surface level seems quite straightforward. These are tools designed to be integrated into the build. They're typically plugins and various other ways that make it really easy to run a scan. But what about failing a build or marking a build as unstable? If you find a vulnerability automatically, as I mentioned before, with bad data or too broad a criteria regarding that data, you're just simply going to fail every build and it's going to take about two builds for a developer to start ignoring the constant instability build notice. Ah, that's just, you know, those false positives are those things we don't care about. So narrowing the criteria down of what you're willing to mark a build as unstable would be a key criteria. Typically organizations might start with a top 10 type vulnerability, something that if exploited, you know, could cause real harm, things like that.

So James, any thoughts on the next generation of static analysis and automation?

We all probably understand a concept of recording audit decisions and then figuring out if it's the same issue that's been audited before and matching that up and then making sure that issue gets suppressed. But we also have filtering that can do various criterias to move issues around or suppress certain issues. But I think there's a concept coming that we can leverage where we have all of these past audit decisions and if we can group those past audit decisions into the correct application context buckets, internally facing applications, externally facing applications, whatever your risk profiles are for your application, we can leverage that data and then using basic machine learning algorithms like random forest, we can provide a confidence value based on past data that in the exact same situations as what you're looking at based on the data set. Right. That you've used to train a classifier, you know, used with random forest, you could potentially predict with context. Right. And that's something that really hasn't been done before. Right. Where you're using I, you know, you heard me say 95% of all of the issues suppressed were due to a lack of context codes, not reachable. I don't care about denial of service, potential vulnerabilities in internally facing applications or ones with little strategic value. Right. These are all contextual decisions. And these data sets, if you have access to quality audited data sets, then we can use that to predict the likelihood of an issue being a true positive with context, which I think is really a game changer with static analysis.

So how would this benefit me? You know, what's the end result?

The result can be better quality data at which to build automation. Right. So it is around persisting and remembering all of our past audit decisions and then scaling that to include application contextually specific decisions in your result sets. We have to improve the quality of the data. And it's, you know, dynamic analysis has some advantages in terms of, you know, significantly less false positives, but it also has a severe disadvantages and significantly more false negatives. It misses significantly more. So if we can take static analysis and we can take these expensive contextual decisions and persist that and use that to provide future predictions, I think we have something really that could really change the game in terms of scale and the actuality of summon being able to integrate static analysis tools into automation in the CICD pipeline, I think machine learning, we're just scratching the surface of what we can do with this. This is a fairly basic example, but it's one that does work quite well. And so there are a number of services, the static analysis as a service businesses, my company and several other companies included of extremely large data sets of quality audit decisions. These are people who audit static analysis results for a living. They should be pretty good at it, and they are. And we can use that data to predict potential vulnerabilities, and we could use your good data, your contextually specific data to predict those issues on top of that. So I think it's something that I find as a topic is very interesting, and hopefully it stimulates you to start asking some questions about how maybe you could do things a little bit differently in a security program that you run.

Thank you James for your static analysis Awesomeness Links to everything value we covered in this episode. Head on over to testguild.com S12 and while you're there, make sure to click on the Try Today link under the Exclusive Sponsor section to learn all about Micro Focus Fortify, which is the recognized market leader in application security. So if this episode has helped you in any way, why not share it with a friend or coworker and together we could spread security testing awesomeness to the masses. So that's it for this episode of the Test Guild Security Podcast. I'm Joe and my mission is to help you succeed with creating end to end full Stack automation awesomeness. As always, test everything and keep the good. Cheers.

Thanks for listening to the Test Guild Security Podcast. Head on over to testguild.com for full show notes, amazing blog articles, and online testing conferences. Don't forget to subscribe to the Guild to continue your testing journey.