Leveraging AI for Secure DevOps with Peter McKee

Get ready to discover some of the most actionable DevOps techniques and tooling, including performance and reliability for some of the world's smartest engineers. Hey, I'm Joe Colantonio, host of the DevOps Toolchain podcast, and my goal is to help you create DevOps toolchain awesomeness. Hey. So the Secure by Design pledge is pushing software companies towards better security practices. But what does it mean for everybody, especially everyday developers? Stay tuned as Peter breaks it down for you. If you don't know, Peter is the Vice President of Developer Relations and Community at Sonar, so he really knows his stuff. He has over 25 years of experience. Really excited to dive into this topic. You don't want to miss it. Check it out. Hey, if your app is slow, it could be worse than an error. It could be frustrating. And one thing I've Learned over my 25 years in industry is that frustrated users don't last long. But since slow performance isn't sudden, it's hard for standard error monitoring tools to catch. That's why I think you should check out bugsnag, an all in one observability solution that has a way to automatically watch for these issues. Real user monitoring. It checks and reports real user performance data in real time so you can quickly identify lags. Plus you can get the context of where the lags are and how to fix them. Don't rely on frustrated user feedback. Find out for yourself. Go to bugsnag.com and try it for free. No credit card required. Check it out, let me know what you think.

Hey Peter, welcome to the guild.

Yeah, thanks Joe. Glad to be here.

All right, awesome to have you. I guess right off the bat, I'm just curious to know. You're Vice President of Developer Relations and Community at Sonar. What does that mean? I assume you have your pulse on the community, then what's going on in DevOps in general?

Yeah, yeah. So I look after the developer relations team, the speakers that go around the world and talk about clean code and software quality. And then we have a community about around 8 million engineers, pardon me, in the community that love to talk about software quality and look after the code at their companies. Yeah.

Awesome. So like I said, you have over 25 years of experience. You know, DevOps has been around for a while and I know there was a push years ago for DevSecOps. Just trying to curious to get your thoughts on where we stand now. Do most developers know security is important part of DevOps, or is that still something you need to make a distinction with DevSecOps.

Yeah. Hey, that's a great question. It depends, like all good answers, I'll say it depends. But yeah, you know, larger companies, I think it's. There's a big push shift and left getting that security all the way down to the developer. But you know, developers under a lot of pressure to go faster, deliver more features to production and unfortunately security is usually an afterthought. But you know, if you really think about security shifting totally left, you're looking at code. Right. So engineers are the foundation, they're writing that code so they should have that security mindset in place. Yeah, a lot of great tools are coming out too.

Yeah, I love tooling. So we're definitely going to touch into that a little bit. Quick question though. Does it seem like we throw more and more at developers? Is that just the world we live in now? It's like it seems like need to be not only development experts, security experts, performance experts, testing experts. It just seems like there's more and more things that are being thrown on them.

Yeah, exactly, exactly. It's. Everything rolls downhill. Right. And goes all the way to the developer.

I'll keep the language PC but yeah, yeah, it feels like that a lot. Right. It's just more and more pressure. You know, I used to work at a large computer manufacturing company here in central Texas. I won't say the name you probably, your listeners probably guess. But yeah, we had huge QA teams, separate test teams, separate security teams. A lot of that's all gone now. It's a lot of it is all pushed to the developer. You know, you have that cross sectional, you know, folks that have come through security into development, DevOps right into development. So. So you're kind of melding things together but still. Yeah, you have to, you have to know a lot of things other than just writing code. Yeah, it starts to get overwhelming a little bit I think for a lot of developers.

Absolutely. So how do you overcome that overwhelm? How much security does a developer need to know? Do they need to be like a cybersecurity expert? Do they need certifications? Is just you still have maybe a high level security group maybe or like how much security I guess is involved in their day to day activities.

Yeah, I think the two biggest things they need to worry about is definitely the code. Start with the code, scan your code, look for vulnerabilities, hotspots, those type of things. You know, write good defensive code. Right. All those good practices. You know, it's hard, right. Junior developers are coming on board, but they Have a lot of, you know, they're just looking to solve the problems. Right? Oh, it works. But you know, there's probably 50% more code or maybe not that much that you have to do security wise. Right. Checking all your inputs, checking all your outputs, making sure you're not tainting your data as it flows down through the application. Right. But if I think at the end of the day, if the developer really focuses on the code, they're doing a good job. And then the other side of that is kind of post little bit of runtime scanning, static analysis and then runtime analysis. Right. And so they should really understand those things. They should understand their borderlines, their where their code stops and other code starts. Right. And protecting those, those inputs, I think that's the biggest thing. Right. Data coming in is where you're going to get your most issues, right?

Absolutely. So we talked a little bit about DevSecOps, been around for a while. Shift left has been a concept that's been around for a while. So why are there still initiatives going on like Secure by Design Pledge? Why has there been more traction I guess in the security area? Is this just more putting more emphasis on security now more than ever?

Yeah, I don't know. You know, it's probably business leaders, I think three things business leaders want, right. They want their features, high valued features and they want those quick. And they want them secure. Right. And they want to keep technical debt low. And security is a big, big issue. You don't want CIOs, CTOs, CEOs do not want to show up on the nightly news. I know we really don't really maybe not watch nightly news anymore but you know, they don't want to show up on hacker hacker news. Right. For sure. So security is a big, big concern, you know, losing your PII data. I don't know, I just got another letter from some company that they've been breached. Right again. And yeah, I think it's a huge, huge concern. You lose trust with the public. So I think security is top of mind for everything, right? Yes. Deliver good features, good products, but they gotta be secure, can't be hacked or you know, the world just won't use your products. So I think that's where the big emphasis comes from.

Absolutely. So you know, as you said, it starts with code. What does someone need to be aware of? Is it as they're writing code, are there ways that they can be prompted, hey this, you could write this more securely or is it more like the open source packages? They may be Consuming, not knowing. There may be known vulnerabilities in those particular packages.

Yeah, it's both. It's definitely the code first, right. And great tools like sonar, I'm a little bit biased, of course, but your static analyzers, right, will, will sit there in your browser a little bit like Grammarly for the general public as you're writing English help you write better English. We have a linter that helps you write better code, kind of watches your patterns as you write the code. And there's some, we have a really good set of rules, security rules that scan your code as you're writing. But that's not enough, Right. You still have to analyze it in your CICD processes because you have cross file issues, right. A lot of linters just look at one file at a time. But you need that whole view of your code as data gets passed through function to function to function, making sure along those lines it doesn't get tainted or disrupted. But yeah, it really starts at the code and the solid patterns that you write. There's known vulnerabilities in, you know, memory overflows, those type of things, pointers. But even in all the way down to JavaScript, right, we look at and you know, you're not able to manually manipulate memory, but there's still vulnerabilities all over the place. Regular expressions are a big issue. I didn't know this before I came to sonar that. Yeah, you can have security issues in your regular expression. So which kind of blew my mind, right? They're like little compilers in the regular expressions, right? That kind of execute code, you gotta be very, very careful. Yeah. So and then in your third party packages, right? CVEs are great and you have versions, databases that look at, hey, you know, log 4J is a great example, right. You'll look at a specific version there and okay, you're using that in your project. We see the version number, it's linked to a bunch of CVs, but it's very static. It's very static. We kind of had an answer to that called deep Sast. So we'll dive into open source projects, modules, projects and actually scan that code. So we know the known vulnerabilities, but then we can also see the path of your code. So when log4j happened, you might be have that version of log4j in your application, but the code path through there might not even hit it. So if you're a large company that has 5,000 developers and you're sitting back and you're going, how do I attack this with Sonar and deeper sas. You can kind of prioritize. Okay, we, this application, the code is affected because of the module that's there, but our code path doesn't touch it. Right. So let's lower that a little bit on our hit list. But the ones above that we can hit quicker. Yeah, we'll address those quicker. So I think open source is a big issue. It's, it's huge. Huge. You know, open source is used everywhere. I forget the stats on it, but the mind blowing on how powerful it is. But at the same time it is open source and it's developed by folks on the Internets that, you know, you don't know, you don't know the quality and you kind of trust things. Conversely on that you do have a huge user base of the code. So we are pounding on it constantly. But yeah, it's something you have to be very, very mindful. It's so easy to pull in a package from package managers and say, oh yeah, hey I need, I need something to, to, you know, hash my passwords. Right. Oh, here's a package, let me use that. Right. Well, you can have injections in there all the time. You can, you know, it's a big issue. Right. You need to be scanning that code. You need not only to have your, your CVE lookup databases, but you also need to be scanning that code. Right?

Absolutely. So, you know, as a developer, the, it's not good to be context switching. Oh, now I'm doing security, now I'm doing, you know, I'm just trying to come up some features for this. How do you make it so that it integrates with their environment or their, their ide maybe. So I guess first I should ask, I assume everyone knows what sonar is, but maybe you can just give a little bit idea of what is sonar on there. Maybe what you're doing maybe to plug into the developer's ecosystem, maybe to maybe I assume you're doing this to avoid this context switching that a lot of developers have to go through.

Yeah, yeah. So Sonar, we focus on clean code, as we call it code quality. But is it static? Coded analyzer. So we basically take your code, put it into a abstract syntax tree and then we run a bunch of rules against it, looking for vulnerabilities, hotspots, and we do the basic things of consistency. So you know, two tab, two spaces or tabs. Right. And semicolons here, kind of the standard stuff. But then we do things like I was talking about earlier, we call it dbd, which is data bug detection. And that's where we follow your data through some of its code paths, looking for taint analysis is probably the more technical term. And that's all wonderful and great, but if it's not like you said, if it's, if it's an afterthought for the developer, it becomes painful and that, that loop, that internal loop and outer loop becomes wider. Right. So yeah, I think we try with sonar lint, which is our Linux sit, sits in your browser. The browser, your ide. Yeah, sits in your IDE and watches as you code, helps you code better. You could do quick fixes and those type of things. But another thing that's very, very powerful that we have is we call it learn as you code. So our descriptions of our rules are very in depth. We'll even point you out to more references. So I was talking about the regular expression issues, which I didn't know, and I started learning about those from our rules and dug into those as I was coding right in my browser. Right. And so having it right in your browser, I keep saying browser in your IDE.

There are. You can run VS code in your browser, which is super cool. Okay, I don't want to digress, but yeah, staying in your id, closest to that inner loop is where you want to be. But you also need that backup that second check right within your CICD process. I argue for quality first, then functionality. So you know, you can have a well functioning piece of software, but it could be a poor quality. Right. The analogy I give is you could build a house, right? But if it's made of, you know, paper, sure, there's doors that open and close, you have a roof, but soon as it rains, that thing's going to fall down. Right. So the quality is very poor, but it functions well. Right. So I think we should really, really focus on quality first. Quality first, then functionality. Right. And that, that helps by pushing all the way into your ide and it should be integrated into your thought process as you're developing. Right. And not an afterthought, not, not something you bolt on at the end. Right. And I think when we first started shifting left, that's what it was, right. It was like, oh, we're doing the same thing, but let's just make the developer do it. Right. And we all called that shift left, right? And we said, yep, go and let's get rid of all these other folks on the team. We don't need them anymore. Right. And that was not good. That was not good. But I think now we're maturing a little bit and understanding, I think developers coming along. And to your point earlier, there's a lot of knowledge they have to get and so they're integrating that into their sdlc. But without tools, without like sonarlint and others, right. It's. It becomes very difficult. It becomes overwhelming.

100%. So I like how you said quality first. I come from a testing background, a QA background, maybe. I was always in dysfunctional organizations. I assume when someone has a feature that's going to go out the door, that's going to make them money, you know, that's going to be number one priority. Is there still. Do you still have to convince companies that, hey, not only it also needs to be quality because it's not a sunk cost. Because security, if something goes wrong can hurt your brand value. You get hacked, you probably get hit by certain laws nowadays and fines. So how do you, is that a message that's still going on like that you have to take into consideration security?

Yeah, I think less so the engineer, right. I think as engineers we get that right. We want, and I think we lean a little bit on that. Right. And I think that's always been the battle between the business and engineers, right. We want to write good quality, well written code, maintainable, reliable code. And the business is over there cracking the whip. Yeah, yeah, that's great. Go faster, faster, faster, right? And, and we all hold up the technical debt flag and then we say, no, no, no, you're gonna, you know, you're gonna increase your technical debt. It's gonna be harder down the line to go faster. Right. And unfortunately the business leaders go, yep, that's great. But it doesn't, doesn't register, right. I think they think of software as, you know, it's very abstract, right? It's, they say, you know, well, it's just code, just change it. It's not that hard. You know, just move the button here, right. But they don't understand. It's a, it's an iceberg, right. They only see the tip of it, but underneath is, is massive amounts of plumbing. It's, it's a lot like a house. Forgive me, I'll go back to the house. Right? If, you know, if you ever built a house, you know, you look at a model home, you go, great, that's the one I want. You sit down, you design it, they start building and then you show up halfway through and you do a walk through and you're like, actually, can we move the kitchen just a little bit over here? Can we move the sink just a little bit right in your brain you just go just, just move it. What's the big deal? But you don't realize between those walls you have plumbing, you have wiring, you have structures, right. That can't just be moved that easily. It's the same with code, right. It's even though it's abstract, there's plumbing underneath that is tightly, you know, sometimes unfortunately tightly integrated together and decoupling those things take time and it causes, it generates technical debt. And so. But I think business leaders don't really understand that issue of technical debt. Right. It's until three months down the line when you say hey, they go hey, build this. Why it's going to take so long? What's going to take? Well, we've. Because you forced us to go so fast two months ago that we incurred all this technical debt. Now when we build this feature we have to back up some design that we did. Right. Cause we went fast and didn't think ahead as much and we have to rewind that and the. So I think I mostly talk to business leaders and technical leaders. Right. I love my fellow technical leadership. Right. But you get, you battle that line on top, right. And sometimes you get a little bit amnesia and you forget when you are coding yourself of how technical debt can creep in and you want to please the business cause that's how you get your bonuses. Right. And but I think technical debt is a big issue. It's that three legged stool, you know, speed, cost and functionality. Right. If, if you want to go faster, one of those are going to suffer, right. It's really hard to have all three. So I think taking a balanced look at those three and you know in the long run you'll be way better off. But yeah, I think to answer your question, I think developers get it so not you don't have to talk to them as much other than quality first. Right. Functionality second. I think the business leaders is where we have to bring on board a little bit more and help them understand that in the long term this is best for their business.

Love it. So I guess this almost sounds. This works best for the greenfield and new functions. How do developers handle their. I've worked in insurance companies, healthcare insurance. Literally the software is dirty sometimes. Actually one was 40 years old. So how do you handle tech debt then? Like where do you know them when to focus in on your security efforts? Obviously when they're coding that's great. But maybe the code already exists. How do you know where to even begin then like we have tech debt. Where do we start? With security to make sure we even are secure.

Yeah, yeah, this is a big issue. So I help mentor folks that are going from other careers into software engineering, right. And I always tell them, you know, you're going to learn the code, but you kind of learn to code in an isolated area, right? It's all new projects, you understand the code. But guess what, you're going to show up at a company and you're going to be dropped into a 15 year old code base, right? So you have to understand how to read code and all those type of things. So we take approach at at Sonar called clean as you code and happens all the time. I was just at a conference, we are developers in, in Germany and I had this conversation probably 10 times, right? So you turn on a static analyzer and boom, you just get thousands, thousands of issues. And like a good engineer, you turn that shit off. Excuse me, I don't know if I'm allowed to swear on there, but, but you know, you turn that off, right? But that's the wrong approach. So. Yeah. How do you approach it? And so we kind of think about it. We looked at the half life of code and it's about 3.33 years. And we looked at a bunch of open source code, we looked at our own code and we said a line of code is going to stay for about 3.33 years in your code base. And so you start out day one, everything is dirty and then you start cleaning and only focusing on the new code and code that you're refactoring, right? Making sure that is solid clean. And then your SEB1 security issues, SEB2 security issues, really start focusing on those. And as you go, as you're fixing your code and you're pushing into production over time about two to three years, you're at 30, 40%. At the five year mark, you've cleaned about 60% of your code base. And then you always get the question, well what about the remaining 40? And I say leave sleeping dogs lie, right? It might not be maintainable, right. But it's reliable and secure. If it wasn't, you would, you would have issues in it, right? So let it stay for a while and focus on your code as you go. And I think that's a great approach. I think a bad approach is what I see a lot is you turn that on and of course all the senior engineers and mid, you know, go, I don't want to do this, I don't want to sit here and fix these things. Okay. Engineer manager says great, okay, get the new guy, get the junior guy, have him go. He's going to focus on cleaning up all this code. Well that's no offense to you guys starting out right, but that's the worst person to do it, right. They just don't have the experience and knowledge to handle it right. And so I really believe in this. Clean as you code, start with new code, refactor code, make sure that's clean and over time you will touch your whole code base in about five years. Four to five years.

Not great advice. So I can't believe we're 20 minutes into this or more and we still haven't talked about AI. So how does AI change the game then with security? Does it make it easier or does it still make it where you can't just give a junior program or some AI now and say go at it. Like how does that work nowadays?

Yeah, I think there's two sides of that coin, right. I think with Gen AI it's wonderful, right? But it can't do jobs yet, but it can do tasks and it can generate code for you. But it's also based on, you know, the quality of data coming in that these models are being trained on is the quality that's going to come out. And I think that, you know, you have to look at it from the three levels of, let's say of a junior mid level senior engineer, you know that junior engineer. Unfortunately a lot of our Gen AI right now is maybe it's accelerating stack overflow, copying and paste. It's a lot easier to do that. Right. And a junior developer might not understand what exactly that code is doing. So I think it's very, very risky there. You can introduce a lot of security issues for sure. So if you're using Geni Start right now, you should do be doing static code analysis a hundred percent. Right. You need that other backup, a tool that has your back as you go higher as someone like yourself or I. Right. We've seen a lot of code. It's very helpful for me. I can use Gen AI, I can write code, I can modify it really quick, put it into my application and we're good to go. So you know, it's a little bit of a double edged sword there on the, on the code generation. But it's going to get better. It's going to get better and better. On the other side of that is these tools though, you know, AI in general is, is a pattern matching tool, right? Predictive pattern matching tool and I think that's awesome for security, right? It's going to be able to identify issues way better than we can as humans. It's going to take a while. It's not there yet, of course, Right. We still need these wonderful PhD folks in static analysis, right, that have spent their life really studying these things. But I think AI is going to come in and help them better the industry. And it's going to be able to watch our code, it's going to be able to scan it, looking for patterns and predict things, right? So I think it's going to be a big help. Of course we're not there yet. I think we're going to get a big, you know, big push. I hate to say hype because it's a little more than hype, but we're going to, you know, it's all wonderful and great and everybody's going, this is awesome. I think it's going to dip a little bit when everybody goes, eh, that's not that, that's not that great. But then it'll come back, right? If you, if you look back in the early 2000s with the Internet, you know, we had online shopping, right? We had books online, we had delivery services, right? We had all these things and. But everybody was worried, right? Didn't work that great. No one wanted to give a credit card out. Connection speeds were real slow and we didn't have these amazing computers in our pockets, right? And so we had this big hype. Remember you did all the super bowl ads and, you know, just throwing money at startups and you're seeing a little bit of that now. And then that went away a little bit to what, 09, 10, 11. The smartphone was a big contributor that, you know, Internet speeds. And now, I mean, shoot, I don't think there's a day, perhaps maybe an hour that a Amazon box doesn't show up. My front door or, you know, my daughter ordered, you know, doordash the other day, right? A guy shows up and she's like, I'm hungry. And I'm like, what world do we live in, right? You know, it's so much part of our life. And so I think that's AI in general is what we're going to look at. But no one, no one can predict it. My prediction for AI is in. We know it's going to be, it's not going away and it will progress faster than we know, right? We'll be shocked if you just look from ChatGPT 3.3.5 to 4.0, right? The leaps there are huge. So what's going to happen the next three to five years? It's going to be crazy. And it's very, very hard to tell where it will be, but it will be a lot better than we expect and we'll use it in different ways than we can even think about it now, right?

Absolutely. Great stuff, Peter. So also curious to know, we talked a lot about shift left. What about shift right? I've heard about this a while, for a long time. How much do developers have to worry now? When back in my day everything was on bare metal, you had everything in your control. Now you're using third party services in the wild, you're using, you know, cloud environments. Are there other security concerns now that people need to be aware of when they shift? Right. And how can you even prevent that?

Well, that's a great question. I don't know if I have a great answer for you, but I'll try. We can, maybe we'll, maybe we can waddle through it a little bit ourselves. But yeah, I think it's a big issue, Right. I think security professionals will tell you anytime anything is taken out of your control, right. Security wants to, you know, lock everything, close up, lock every door, close every window, tighten it all up, right. But it's hard to scale. So when we're going to the cloud and we're using all these services, there's third party services, you gotta have to be very, very careful, right? If you're not setting up your virtual networks, how they talk across services to each other, even just, you know, not very smart things like putting your keys in the code bases, pushing that up in the GitHub. I just saw a guy on YouTube was showed how to do a search through GitHub to find OpenAI private keys, right? Even in public, I mean in private repos it was, it was kind of scary, right? So I think it's a big concern, right? And we have to be very, very careful. I'm not one for policies and procedures, you know, probably anti those, but yeah, you have to put some rigor, rigor in your practice, in your sdlc. I think it's an issue, right? I think it's. And it's not going away, right? The answer is no, no, no, no air gap it bring everything back in. You know, that'll still, you know, because for legal reasons and medical and all those type of things will still be an issue, right? But companies want to go fast and do more with less and that means using third party services and you have to put moats around your, your technologies and protect things. So yeah, it's a big risk. It's a big risk.

Okay, Peter, before we go, is there one piece of actual advice you can give to someone to help them with their DevOps security efforts and what's the best way to find contact you or learn more about sonar?

Yeah, I think you DevOps folks out there love you to death. Help your engineering teams, help them understand that they need to be static analyzing their code, right? You guys are the gatekeepers, right? Think about functional testing, right? Unit testing, integration testing. There's not a good DevOps engineer that I know that doesn't say, hey, don't worry about unit testing in your CICD process. We'll be okay, right? It's a must have. And I think you have to think about quality, right? Quality first. So integrate a static code analyzer into your CICD processes. Help the engineering teams help write good quality code and then also to the engineer, get a static analyzer. Use it in your ide. It'll help you write better code and that is the foundation that then going forward, you can build on, right? So yeah, that would be my advice. Static, analyze your code, think about quality first. Then functionality and life will be perfect. There's no problems there.

Remember, latency is the silent killer of your app. Don't rely on frustrated user feedback. You can know exactly what's happening and how to fix it with Bugsnag from SmartBear. See it for yourself. Go to Bugsnag.com and try for free. No credit card required. Check it out, let me know what you think.

And for links of everything of value we covered in this DevOps Toolchain show, head on over to testguild.com p11 so that's it for this episode of the DevOps Toolchain Show. I'm Joe. My mission is to help you succeed, creating end to end full stack DevOps toolchain awesomeness. As always, test everything and keep the good. Cheers. Hey, thank you for tuning in. It's incredible to connect with close to 400,000 followers across all our platforms and over 40,000 email subscribers who are at the forefront of automation testing and DevOps. If you haven't yet, join our vibrant community@testguild.com where you become part of our elite circle driving innovation in software testing and automation. And if you're a tool provider or have a service looking to empower our guild with solutions that elevate skills and tackle real world challenges, we're excited to collaborate. Visit test guild.info to explore how we can create transformative experiences together. Let's push the boundaries of what we can achieve.

With lutes and lyres, the bards began their song A tune of knowledge, a melody of code through the air it spread like wildfire through the land Guiding tester showing the secrets to behold.