Next Generation DevSecOps with Cindy Blake

Welcome to the Test Guild Security Podcast where we all get together to learn more about security testing with your host, Joe Calantonio.

Hey, it's Joe. And welcome to another episode of the Test Guild Security Podcast. Today we'll be talking with Cindy Blake all about next generation DevSecOps as well as some security test tool companies. GitLab recently acquired PeachTech and Fuzzit and want to know how they really fit into the future of DevSecOps? If you don't know, Cindy Blake is the Senior Security advisor evangelist at GitLab and collaborates around best practices for integrating DevSecOps application security solutions with major enterprises, so she really knows her stuff. She also is the author of the new book 10 Steps to Secure Next Gen Software that combines nearly a decade of cybersecurity experience with a background in Lean and software development to simplify the complexities of today's software evolution into pragmatic advice for security programs. Really great stuff. You don't want to miss this episode. Check it out.

It'S no secret that the number and complexity of applications are growing, but what is alarming is that 90% of security incidents result from exploits against defects in software. The need for better application security has never been higher and Micro Focus Fortify is here to help. Fortify is the recognized market leader in application security and it is the most comprehensive and scalable application security solution that works with your current development tools and processes. With Fortify, you can start security applications in a single day, including custom code, open source or commercial components and scale as your needs grow with an on premise as a service or hybrid implementation. Check them out. Head on over to micro focus.com app security.

Hey Cindy, welcome to the Guild.

Thanks Joe, Good to be here.

Great to have you. Before we get into it, Cindy, is there anything I missed in your bio that you want the Guild to know more about?

Well, it is kind of ironic that Fordify is the sponsor of this session since I spent several years at Fordify. But no, I've been at GitLab a couple years and really excited to bring together and bridge the dev side and the security side.

Very nice. So as I mentioned in the pre show, I'm kind of a newbie to security, so and I don't know if this is because I'm a newbie. When I hear git, I automatically think of git. I don't necessarily think of GitLab. So to start off, what is the difference between Git, GitHub and GitLab? Are they all the same?

They're not they started in a similar trajectory, but so GitHub and GitLab both started as Git repositories and that's kind of where the similarity ends. So GitHub, as you may know, was acquired by Microsoft and it's kind of being positioned along with Azure, DevOps for the software development lifecycle management. GitLab is a single tool for that entire software development lifecycle and we're our own standalone company. We've had quite a bit of investment capital infusion and hoping to go public soon.

Very cool. So I'd like to dive into a little bit more about GitLab later on. But first, so what I really wanted to start off with is your new book by O'Reilly, 10 Steps Every CISO Should Take to Secure Next Gen Software. And I really enjoy this book. You have three main levels that you talk about, three main areas. The first one, one is shift on how software is composed and executed. And I guess these are the main shifts you see happening in the next generation type of software for DevSecOps. So I guess before I actually keep talking here, what is the first shift that you see moving people away from maybe the DevOps types, practices they currently know about, towards what you see as more of a new way of doing things?

It's really around how software is composed and that involves a couple things. It involves methodologies, so the agile methodology and DevOps methodology, which calls for minimal viable change, much more iterative. So historically you would take an application and program some major functionality, release and drive that through to production and then deliver it to your customer. Well, today the desire is to take much smaller incremental improvements that can bring business value much more quickly. And so you're biting off smaller pieces and driving those all the way through to production. And what that means is you're iterating much faster. And it's very challenging to do that with traditional application security, where you might be testing the entire application at the end. Because what you want to do in this more iterative environment is test every little code change as they're going. So think of it as what's the saying? How do you swallow an elephant one bite at a time? You know, you want to take off those small bites and do it piece by piece as opposed to waiting to the end. And that way you're more congruent with that methodology. The other piece of software being composed differently is greater use of open source software. So it's much less about reinventing wheels these days, but what's out there that's tried and true that provides a piece of functionality that I can use and pull into my code. And you know, you get the upside there of many people have used it and so it's probably more secure and better than, you know, what some newbie developer could come up with. But at the same time, everybody knows what's under the covers and so do the hackers. So got a little bit more exposure there you need to think about.

Nice. But I really liked about this section as you talked about how people are moving towards cloud and containerization. So how does that impact security? From maybe a traditional way of people may have thought about AppSec to a new way of mindset that they might have to think about now. Now that we're in the cloud. More in the cloud, sure.

So now that we're moving towards more cloud native applications, what that means is containerizing software and having APIs so that those smaller pieces of code can talk to one another. And all of those things, whether it's containers, orchestrators, microservices, bring about a whole new set of attack services. So if you are only testing the application code or you're only doing network security, you're really missing this whole new area of threats. So thinking about container scanning and monitoring the behavior of containers, APIs, making sure that you're testing the APIs, which is where our fuzzing acquisition comes in.

Great. So you did mention fuzzing. Just curious to know, for the folks, I don't know what is fuzzing because I know GitLab recently acquired two companies. We'd probably have to tell or we probably have to get into what the differences are between the two companies because it seems like they both do fuzzing. So what is fuzzing first?

Yeah, so fuzzing finds unknown software defects. So typically when you're doing scanning for security vulnerabilities, it's known problems. Right. There's a database of CWE like known vulnerabilities and you're looking for those things. Fuzzing looks for those things that we don't even know to ask. And so it can find not only security defects, but quality defects as well. And generally there's two types. There's what the industry has referred to as black box and white box. We're coverage guided and protocol fuzzing. And the coverage guided fuzzing is something that is going to find a different set of things than what the protocol fuzzing would find. And so they're very complementary to one another. And we were able to just close on an acquisition of both Fuzz it and Peach that handle those.

So what are the difference between the two? I thought I saw a video from GitLab that talked about Peach is more dynamic compared to Fuzzit. Are there two different types of fuzzing?

So Peach is going to be more useful for the API fuzzing because it's going to look at how the application behaves. And when you think about building into more microservices, you're going to use a lot more APIs to connect the dots between more elementary software elements. And so you need to be able to test the behavior of those APIs in addition to the software. And that's where Peach comes in.

Nice. So, you know, a growing trend in functional test automation is AI. So is there any type of AI that's baked into this fuzzing or do you see that as a trend growing maybe as we move towards this newer generation of DevSecOps?

I think that's a trend. I think that you'll see more AI baked in through security in general going forward because, you know, individuals are really good at picking up on patterns, but nothing beats automation. And if you're trying to scale your application security program, you really want to enable the developers and they're not trained in security, they don't want to become security people. So if you can use AI and automation to apply the same kinds of things that a security person would be looking at, but doing it much earlier in the lifecycle, doing it for the developers, then that's how you can scale. There will always be more developers than there are security people.

Absolutely. And I was watching another YouTube video that you did a presentation on, Kubernetes, and you talked about how we're moving away from this waterfall security type approach to more, I guess you keep mentioning developers, a shift left approach. I know once again shift left is a buzzword, but this is a trend you see as well, is security being moved on into like the Sprint teams, where developers actually have to incorporate security along with their other activities for each Sprint.

Yeah, the difference is, I mean, shift left has been around for a long time. So NIST first did a study back in early 2000s where they looked at the cost of remediating vulnerabilities in dev versus in unit testing versus system testing and versus production. And they were able to show you substantial savings if you can remediate those vulnerabilities early. The challenge has been how to do that. Everybody wants to shift left, but it's been very difficult. In fact, I did a Focus group a little over a year ago with a group of developers and I asked them, you know, do you care about security? And overwhelmingly, yes, they all cared about security. But when I asked them what motivates them to create more secure code, is it a carrot or a stick approach? The answer really overwhelmingly was nobody wants to be, you know, that person that brought their company down. They really took take their jobs very seriously and they feel very responsible and accountable, but they feel like their hands are tied, that they lack the tools to help them find and fix those vulnerabilities. And so anything you can do to empower and enable the developer, they're really ready to embrace it. But it needs to be within their native workflow. Don't ask them to use Fortify or Baricode or one of the tools that was really intended for the security audience. Find a way to bring those scan results to them in their tool like GitLab that they're using.

And once again, as I mentioned the pre show, I've been focusing on functional automation. There's a lot of parallels here. The same thing. We have a popular tool called Selenium that's popular among testers, but developers very hard to get them to use Selenium. So they usually we look to use tools that they're already familiar with, like Cypress and their own language, JavaScript. So it sounds like they're already using GitLab. Then these particular fuzzing and scanning approaches are already built in, so they don't have to change context as they're developing code. Is that correct?

Exactly. And if, you know, security can draft off those coattails, so to speak. And it's kind of come down to a two hors source race for source code management. You know, there's GitLab and there's GitHub bit bucket a distant third. And when you look at those, you know, if they're already using these tools, just go with that flow and build security into that CI process.

Great. So the DevOps shift left type thing you brought up in the second shift that you called out called how software is delivered and managed. And so what are some challenges? What all the challenges do you see in this particular shift? I know you had a cool quote that said security testing is like a square peg in a round hole of DevOps. What do you mean by that? What are some challenges you see getting this push toward delivery and management section of the shift?

Yeah, that really has to do with the difference in workflows. And if you're doing all of your security testing at the End or the testing the entire application. It makes it very difficult to fit that into a workflow where a developer is pushing a small snippet of code, you know, code, small code change all the way through to production. You really want to be able to test just that change and see did that change make a difference in my security risk. And so that's where the square peg, round hole comes from because you have to think outside of the box and think about delivering those results to the developer.

So I used to work for a very large enterprise company and we used Jenkins and all these other open source solutions and it wasn't really robust, but it wasn't a whole and holistic end to end really solution. It was really kind of cut and paste type stuff. So it sounds like GitLab has an end to end type life flow or workflow that developers are in. So would they know about security? Do they have to worry about security? Is this already turned on? Like how do you incorporate security into the developers pipelines as they're developing software, when they're using GitLab, when they're using.

GitLab, our security is baked into the ultimate tier, which is our top tier. The core free version really is intended for the individual developer. As you go up in tiers, it's more for the enterprise users, you know, that have more robust, more mature needs. And ultimate is where you find security. And we baked it into the CI process so that when you commit your code, those scans through, you can enable them through autodevops and set up your policies really with one click and the scans occur at the point of code commit. And the really cool thing is it's multifaceted scanning. So it's dynamic scanning, static scanning, container scanning, dependency scanning and license compliance, as well as secrets detection. And so all of those things happen before the code ever leaves the developer's hands. And that's the dynamic scanning is something that nobody else can really do. And the way we do that is because we're integrated into one tool or one platform. Our CI capability spins up what we call a review app. So it's kind of an ephemeral view of the application with the changes that you've made. And you can use it for user testing. Hey, I made this change. Is this how you thought it was going to work? That kind of thing. But you can also use it, we use it for dynamic scanning to run the DAS scan against it. So again, getting all of those multifaceted results back to the developer and we'll be adding buzz testing to that as well. In fact, I think the first iteration we do everything with iteratively ourselves. The first iteration for one language is, is comes out this month. We Release on the 22nd of every month.

Cool. So, you know, with these scans, how much can first scans catch? And is it just like a sanity type thing? And two, are there any like, are there a lot of false positives that, that would maybe turn off a developer, that they have extra work now because they have all these scans running when they check in code?

Interesting that you say that. So the different scans find different things. So what you'll find in static code analysis is a different set of problems than what you'll find with dynamic scanning. Dynamic scanning requires a running applic and it's looking at how the application is behaving, whereas static code is just looking at the code itself to see if you have vulnerable phrases, if you will. So you really, you know, you need a variety. In the security industry, we call that defense in depth. And you really want to have a variety of scanners in order to find as many vulnerabilities as you can. Now you don't want a lot of false positives because that is noise for the developer. And so how do you deal with that? Well, this again is where that workflow becomes really important. If you're scanning at the very end, everybody's merged all their stuff together. So I've got, you know, 10 developers, changes all merged together and I run my security scan, I find the vulnerabilities. Now as a security person, I've got to go figure out who can fix this, get it on their list of things to do, make sure that, you know, they commit to doing it. I've got to maybe train them on, you know, I want to know who did it so that I can educate them on why this is a problem and how not to repeat it. If I'm doing it in my CI process and I'm doing it on a code commit, I talked about that real tight cause and effect feedback loop. As a developer, I made this change, I created this vulnerability. So, you know, it wasn't Johnny two doors down that created it. It wasn't Sally. It wasn't something I that's been lurking for a while and I just happen to be the unlucky person to step in it. I created the vulnerability so I am best equipped to fix it while I'm still working on my code. So you're less likely to have false positives in terms of things that are findings that you don't know if they're yours to deal with or not. It makes it much more focused and clear. And then in terms of like false positives, you know, is this truly a vulnerability or not? We tend to focus on the OAS top 10, which if you've looked at some of the threat research over the years, Fortify used to do a really great threat report. Verizon does one and they look at, you know, what are the most common exploits for the year and they usually come back to the OWASP top 10. The vast majority of exploits hackers aren't using something new that they dream up. They're going back and using what works and they're going back to things that people typically fall over, like patching and going forward. When you look at cloud native applications, misconfiguration. So, you know, my one quote that I would like to contribute to the industry is that I think misconfigurations will become as important or as exploited as patching and lack of patching in the future. Because in a cloud native environment now you've got all these levers, you've got it for your cloud service or your containers or your kubernetes or other orchestrator and all of those things represent opportunities to make mistakes and misconfigurations.

Absolutely. Now, this may seem like a dumb question, but once the scanner goes off, does a developer know how to fix it? Like, do they already have the expertise to know? Right, the scanner went off, I caused an issue, and based on that report, they'll know what to fix. So they have to go to a different security team to get that info.

Well, it could be a little bit of both. You really want to enable the developer to find and fix things on their own. So at GitLab, we have kept that in mind as the developer is sort of the first user and they're going to use, they're going to find the vulnerabilities while they're still in their pipeline. So while they're running their code and making changes to their code, and we've tried to make it very straightforward, but also have more information that they can click through and drill down into to help them understand the vulnerability and how to fix it. But if they can't fix it, here's the really cool thing is we have crossed silos and pulled people together. So now the security team is looking at the same thing that the developer sees. So if there's vulnerabilities that remain that the developer can't fix, they show up in the security dashboard and there's the developer and the security person can collaborate so they can create an issue in GitLab that is pre populated and ties to the pipeline that was run. It can show the security person exactly what the developer saw and the line of code. All of that information is right there together. And when it gets assigned to someone, let's say it's an issue, but I can't fix it right now. It's going to be in the next Sprint security person has full line of sight visibility into where that issue stands. Has somebody been assigned to it? What Sprint is it in? When should I expect it to be remediated? And so that's something that you just can't do when you've got multiple tools and siloed workflows.

Nice. So, and the last shift that you talk about in your book is how software complies with regulatory requirements once again work for a large healthcare company. How does GitLab help in this step? Does it send alerts, does it get updates on regulatory requirements and it's able to scan for deviations from it.

We're not quite that sophisticated yet. We have lots of common controls that are built into how GitLab itself is secured and how you can deploy it in a secure method. You know, multi factor authentication and access controls and those things which by the way are very important. You had asked I think up front about the remote work. You know, access control becomes super important with remote work. So making sure that only authorized people are making changes to the code. So we've got lots of features that meet those common control. If You Google the GitLab solution compliance, there's a page that shows that kind of maps some of the common industry requirements like ISO or PCI or whatever and what features help with that. But we're also coming out with a dashboard that will map more specifically to, okay, you need to do an ISO report or a PCI report. Here's you know, map to that report, here's your results of what you need to show an auditor. But auditors love GitLab because since it's one platform end to end, you can always see who changed what, where, when. So if a developer has a vulnerability found and dismisses it, they have to, they can write a reason for that. And so an auditor can come back later and see who dismissed it and why. Similarly, if, let's say you've got a critical or high vulnerability and your policy is that those require approval, I can see who approved it and the note around circumstances around why they approved it.

That'S actually a big deal. In regulatory environments having that single source of truth because you don't want to get an auditor in there and show them multiple different systems having them poking around. So one source of truth really could save the day for sure. Awesome. So Cindy, as we move forward 20, 20, 20 and beyond for next generation, it, what are some tools you think folks in the security space really need to be aware of?

You know, if they're doing anything that's cloud native, I think they really need to pay attention to those other attack surfaces. So they need to look at containers, container scanning up front as the developer is coding to make sure that what's put in the container is free of vulnerabilities. And then ongoing monitoring of the containers, such as the east west traffic between containers. You know, in the book I go into an example of somebody that gets in through a misconfiguration and does some lateral hacking to get into other systems. And it's container hopping basically. And so if you look at, we just came out with container network security, which is meant to address that there's other ones out on the market like focus on containers and have for a while. The other thing would be dependency scanning. So as dependencies, third party code, whether it's open source or not, gets used more and more frequently for the ESI building blocks to make sure that you're scanning those dependencies and not just assuming that because a lot of people are using them, that they're safe.

Nice. So, Siddhi, I do like to keep these to about a half hour. We're at a half hour now. I have one last question I ask everyone. But before I ask that last question, is anything you wanted me to ask before I asked that last question, any topic I might have missed that you think is critical that you want folks to know about?

Well, I think just, you know, what is the one single thing that can make the most impact from an application security standpoint? And I would say scan everything. You know, scan often and enable those developers to do it within their native workflow. So that's kind of the secret sauce. And I've seen some analyst reports here recently that would reinforce that point too.

Awesome. And I think in that Kubernetes video you gave a good example. It's almost like airport security. Everyone needs to go through the security, the scan, and then it flags things that pop out as anomalies or things that aren't. Right. So that's what it sounds like, that type of approach.

Almost exactly. I have an image that I like to show of lots of locks on a door and a window wide open. And if you're scanning your mission critical app and you're doing really in depth scans, but you're leaving the other apps wide open and not doing any scans, then you know hackers aren't going to go after your mission critical apps. They're going to find that backdoor that's not locked and then traverse from there. So you need to think about it in a different way. And I like to think of security as an outcome, not a department.

Great advice. Okay Cindy, before we go, is there one piece of actual advice you can give to someone to help them with their security testing efforts? And what's the best way to find contact you or get our hands on your new book?

Sure, the book is available through the O'Reilly Learning Library for free, or you can get by googling GitLab 10 steps and that should come up to a landing page where you can register for that as well. In terms of next steps, I would say, you know, you can download GitLab ultimate and try it free for 30 days. And if you're not doing any application security today, it's a way to get started very quickly and easily. Small companies love us from that standpoint. Large companies love us because it really, really helps with the shift left. So even if they're already using Fortify or something else, having this scanning, this additional layer of like TSA scanning done up front really, really makes a difference.

Thank you Cindy for your security testing awesomeness. Links to everything of value we covered in this episode. Head on over to testguild.com for.

And while you're there, make sure to click on the Try Today link under the exclusive Sponsor section to learn all about Micro Focus Fortify, which is the recognized market leader in application security. So that's it for this episode of the Test Guild Security Podcast. I'm Joe and my mission is to help you succeed with creating end to end full stack testing awesomeness. As always, test everything and keep the good. Cheers.

Thanks for listening to the Test Guild Security Podcast. Head on over to testguild.com for full show notes, amazing blog articles and online testing conferences. Don't forget to subscribe to the Guild to continue your testing journey.