EP111 How to Solve the Mystery of Application Security in the Cloud?

Security.

Hi there. Welcome to the Cloud Security Podcast by Google. Thanks for joining us today. Your hosts here are myself, Timothy Peacock, the Senior Product Manager for Threat Detection here at Google Cloud, and Anton Chuvakin, a reformed analyst and senior staff in Google Cloud's Office of the CISO. You can find and subscribe to this podcast wherever you get your podcasts, as well as at our website, cloud.withgoogle.com/cloudsecurity/podcast. You can also find us at cloud.google.com/podcasts. If you enjoy our content and want it delivered to you piping hot every Monday morning Pacific time, please do hit your subscribe button. You can rate our podcast if you like our content, and if you don't, drop us a note to know how we could do better. You can follow the show and argue with us on Twitter, twitter.com/cloudsecpodcast. Anton, we're talking about AppSec today. That's kind of a fun topic.

Exactly. And I would say that we have explored the topic of application security in the cloud a few times, but I still feel like the topic deserves more because we still see people who have, you know, somewhat peculiar views of cloud makes their poorly written application secure by magic and a few others. So, it's almost like my impression that AppSec—

And some people have the opposite view. Cloud makes my poorly written application less secure.

Right. Or maybe somebody, I don't know, I haven't met them, but they may exist. The cloud would make their really well-written application not secure, but I think that's a little too silly.

I think so too. So today's guest, I thought, was quite insightful and had quite a lot to say. Any particular highlights that you want to call out before we get right into the content?

To me, the supply chain security— software supply chain security kind of made a cameo appearance. And to me, he had a very, really strong, passionate appeal about that, which frankly, when we had guests that really focused on software supply chain security, they made a case.

They did.

But they never made such a passionate, strong case to care about this. And I think the guest today really did.

It was truly impassioned.

Yes.

It was great. So perhaps with that, listeners, let's not wait one moment more. Please welcome today's guest. Today we are joined by Brandon Evans, InfoSec consultant, certified course instructor, and course author, at the SANS Institute. Brandon, thank you so much for joining us today. I want to pick at your background a little bit. You come from a developer background, so what got you so interested in security to make this your main area of focus?

Well, thanks for having me, first of all. I have been developing applications for a long time, and if you're passionate about programming, you will learn some security concepts by accident. So before I started as a full-time software engineer, I knew the basic AppSec mitigations. Escape special characters from your input before making a SQL query, encode your output before putting it in HTML, make sure that users can only do the things that they're supposed to be authorized for. So I knew all that, but I didn't really understand the damage that could be done to an organization if you didn't get those things right. Now, this changed when I joined the Security Champions program at my previous employer and entered into my first capture the flag event. I hacked this really fun cyber range from Security Innovation that they called Command and Control, and this really opened my eyes to see what is at stake for our industry. Much to my surprise, I won, and I have been doing these ranges ever since. And somehow, I fell backwards into doing security full-time and even teaching it, but this really was what really lit the spark for me.

Hmm, that is fascinating because we keep occasionally— people who are maybe a little bit more cynical than us, Tim, yeah, they do exist— tell us that developers don't care about security, but you went through a pretty short list of basics that I think actually is very reasonable for a developer to know. So if somebody shows up to you and says, Brandon, you're this application security person, what do you tell developers who say they don't care about security? How would you counter it?

Well, I don't think many developers say that, for starters, but I do think there is that perception. And I think that we have really 2 questions to answer about whether or not developers care about security. Do developers care about security? And can developers care about security? The answer to the first question is not really. Why? Because they have no incentive to do so. Most of them are not rewarded by their employer for writing secure code. They're rewarded for shipping features. At worst, they are reprimanded for shipping insecure code. Meanwhile, security is often represented by bureaucrats that get in the way of doing their job while failing to explain why. So if you're a developer who cares about security in that environment, you must either really care about your craft or have much more time on your hands than most people do. Now, the second question is, can developers be made to care about security? And I think they can by changing those incentives. Reward them for taking security courses, celebrate their wins at security competitions, show them how much fun security can be, and you'll be shocked to see how how effective these little efforts can be.

By the way, as a quick side comment to this, when I got into this argument, and again, as you know, as you all know, I'm not an AppSec person, but occasionally I get dragged into these debates. And somebody said, developers don't care about security, but the bureaucrats around them care. And somehow I just felt offended because I felt maybe like I am one of those bureaucrats and I hate bureaucracy. But this is the logic is that people around the developers, maybe they're managers, maybe they're auditors on the side, care, but developers may genuinely not unless something changes, right?

Yeah, I think everybody's different, but I think it really comes down to incentives. And, you know, if security professionals are incentivized to be bureaucrats, then that's what they're going to be. You know, you're not incentivized to be that way because Google has a great security culture and they care about getting things right, at least from my perspective, more so than just checking boxes. But not every environment's like that.

Well, that makes sense. So You know, as we're thinking about this being a cloud security podcast, how do we encourage developers and ops people and everybody else involved in the cloud migration thing for software to think about and encourage the right security controls in the cloud process?

So, I think the first part is to get the incentives right, as mentioned before, but then it's about making it as easy as possible. Developers are generally good about automation. They really don't want to waste their time dealing with very low-hanging fruit security issues, nor do they really want to focus their energy on patching or doing manual labor of any kind. And you'd be surprised to see how receptive they are to working with you on automating scans and tests.

Hmm.

Then you can help them develop secure frameworks that make things secure by default, because I think they generally care about security if you ask them, right? If you ask a developer, do you care about security? They're not going to say no. They're going to say they care about other things more, and they want to make this easier. So if you make things secure by default, it just becomes— everybody wins. So in the context of cloud configuration management, this integrates perfectly with infrastructure and configuration as code. If you're codifying what your cloud environment should look like in code, you can very easily both analyze what your cloud environment looks like and then move to have scans that look for bad configurations or potentially even introduce workflows that automatically create those or fix those configurations.

So, by the way, you said something that I've heard last week from somebody here at Google, and it made me feel a minor kind of a brain explosion. So let me toss it your way because I want to test this idea on you. So you said secure by default, but the context is developers. When I think secure by default, I think somebody's handed a system, he's a user, and he can't screw it up, he can't break it. It's not unbreakable so much, but it's unbreakable by him. Its defaults are secure, and they're not easy to change to insecure defaults. And that's what's in my head first. But this other person, and you apparently a second ago, said secure by default, but for developers. Developers, last I checked, can write code, right? They can program, right? And that means, can you really build a platform that's secure by default for, or perhaps against developers? I don't know.

Well, I guess those are 2 different questions. The first part is about systems, like you mentioned. And the other part is about how you write your code. So with regards to systems, you can put guardrails on your cloud to make sure that bad security settings are not applied, automatically notify people when they get things wrong, and maybe even prevent them from getting those things wrong to begin with. But on the side of code, people typically take the path of least resistance. So what a lot of people do in organizations is they create frameworks that they can use to perform general-purpose operations like database calls or AJAX calls. People don't want to write their own code to handle those things. They want to leverage other people's code, preferably code developed in-house for the reason that I have a feeling we're going to talk about in just a moment, to establish certain standards about how we interface with things. Now, if you build security controls in those libraries, then people will take the path of least resistance, and that path of least resistance will have some basic security controls in there. Make sure to validate input, make sure to escape it when necessary. So, if you can get your developers on these frameworks that are established by architecture teams in conjunction with security, that can go really, really far.

Do you have a catalog of things, or maybe like a quick list of things that we can give to developers So, what are the main issues they have to deal with specifically in the cloud? And of course, please, you're gonna say IAM, but like, I don't wanna put words in your mouth, but you probably would say IAM, right?

I hope he says IAM.

IAM is definitely a part of that. So, I think one way to accomplish this is to have custom policies in IAM. So, for example, we can say, hey, whenever we're interacting with, let's say, storage, at maximum, your applications need these permissions. Here are the permissions we're going to grant, generally speaking. One of those permissions should not be the ability to delete a storage bucket. There's no real reason why an application needs that capability. So that's one of the first ones that comes into mind is custom policies that are reusable within your cloud organization. But in the realm of AppSec, this can really apply to anything that is often performed within an organization. And again, people really don't want to do things manually or reinvent the wheel. If you invent the wheel securely for them, they're going to use your library as often as possible.

So let me ask you a question here. If we're talking about developers, people working on the application, not architects, not admins, but developers, should developers ever be thinking about IAM, or should that be handled by the architecture people and the operations people? Because that, that got me thinking, Anton's question. Maybe Anton asked the wrong question, which he's done before, listeners.

I think that's a great question. I don't think it should be one or the other's exclusive domain because architects are going to come up with, I guess, general best practices and hopefully apply guardrails with organization policy, for example. But the individual developers and operations folks, they're gonna have more context into what they need. So, you may have some instances in which there's a conflict between those 2 organizations, those 2 groups, and then you have to have the understanding to be able to come to those organizations and say, hey, we need an exception. Here's why. Here's why we know that that exception is reasonable. And hopefully, that gets folded back into standards so that the architecture team isn't blocking a whole large amount of organizations who need that permission. But the conversation has to go both ways.

So, you used a word that has gotten my attention, and it's not your fault that the word has gotten my attention. In your mind, Is there a difference between a guardrail and a setting? Or in the cloud, are those things the same thing?

Good question. I typically think of a guardrail as something that prevents someone from applying a bad setting.

Hmm, so it's a meta setting?

Kind of, yeah. For example, there should not be the ability for a developer to grant an application permission to launch other applications. You can put in guardrails that prevent that operation from happening. Or for example, You could prevent someone from launching a function that doesn't require secure transport. So the setting, in that case, is requiring secure transport. The guardrail is preventing someone from launching a system that doesn't require secure transport.

But wait a second. There is also a dimension— when I think of guardrails, I think of them as being invariant or inviolable. Like, setting, I can set, I can unset. It's called a setting. I can set, I can unset, right? Guardrail is something unless I have a dynamite, a chainsaw, I'm not going to break through that. Sorry, I'm feeling violent today. Maybe I should stop. The point is that the guardrails imply hard limits, right?

Interesting.

Guardrails are inviolable. Settings are not.

I would say that a guardrail should be enforced consistently or it's not really a guardrail. It becomes a guideline.

I'd agree with that as well.

Ah, pirate code. That's right. We needed a pirate code quote. Of course. It's not a guardrail. It's more of a guideline. I love it.

So listeners, you might not realize this given Brandon's wonderful answer here, but this is completely off track where we wanted to be and inspired entirely by an inside the organization fight that I had last week. So to get us back on track, Brandon, and I'm sorry for the side quest here, we want to talk about supply chain security. Do you think of supply chain security as part of AppSec? Is it different? Is it part of it? And how important is it? And to make the question just a little bit longer, Is this a realistic thing for the average org to be able to care about?

Yeah, so this is one of my favorite topics. Oh, good.

Ours too. It scares Anton more than anything.

It does scare Anton a little bit, yeah.

It definitely keeps me up at night. So I think that part of the supply chain attack vector matters to developers. So we talk about supply chain in terms of using software from third parties in our organization, but specifically the part that developers need to care about is third-party software packages. I talked about this extensively on SANS's recent free ebook, Cloud Security: Making Cloud Environments a Safe Place, which Anton also had a segment in. So in there, we found that nearly 70,000 packages on npmjs.com, which is the central repository for Node.js JavaScript packages, depend on a single package. We also found that the third most popular package depends on 46 packages, and 35,000 packages depend on that package. So even small applications depend on a tremendous amount of code published by strangers on the internet, and developers need to care about that.

Can I please hide under my desk now?

Yes, it's terrifying. And you talked about one example of this in your recent episode on Log4j, but it's not just about accidental vulnerabilities in third-party code. It's also about purposeful hijacking of this code by malicious insiders or hackers who compromised these package repositories. So these attacks get even more dangerous in the cloud because that makes it really easy to pivot, because if you compromise a virtual machine with IAM credentials, those credentials can be used to pivot throughout the entire cloud account, potentially leading to an entire compromise. And the worst part of this, in my opinion, is that this is somewhat inevitable. The alternative to using third-party code is to develop everything in-house.

Badly.

Can you imagine?

Yeah, badly too.

Yeah.

Are you going to create your own programming language? What about your own operating system? Might as well develop your own CPU at that point. Can we really trust the power grid? Yeah. So how far down do we go? And I think we need to find the right balance. And as an industry, we have failed to do so.

But is this a problem solved by above their pay grade, by quote unquote bureaucrats on the side from developers? Or is this something that developers and their dev managers and maybe like mid-level leaders would care about? Like at this point, I'm puzzled myself. If I'm using the code from the internet, I'm going to be scared now.

This is a tremendously difficult problem to solve. I think that a lot of people need to get very serious about this. I think the really only thing that security professionals can do is use software component analysis. And all that really does is look at the components that your application uses and determines if those packages have known vulnerabilities. But that's not going to determine whether or not those packages in your specific application are going to make your application vulnerable. And that is something that not only do developers get involved with, but I would like to see QA get involved with. Imagine if QA could do some basic security abuse case testing, where they put in bad inputs and see what happens. I think that we are severely underutilizing a workforce that does testing for a living when we have so much of a need for security testing. So I think that this is a security problem, a management problem, an application security problem, a cloud security problem, because They're really hard to separate. And maybe even QA can get on board because this is a problem that is just terrifying and everyone needs to get involved with.

I mean, this is a solid motivational speech. I think it motivated me to actually be more mindful of this. I don't want to treat it as a kind of more esoteric topic, but I think that I may well be drafted to be an evangelist for that topic now because I feel like it's— I think Brandon made a really good appeal that I haven't heard before. Well, I haven't heard such a good appeal before to this, to care about this.

Yeah, that was great.

Oh, thank you. That means a lot coming from you.

Yeah. So now that we are appropriately scared, let me try to switch to one other favorite subject that actually is both mine and Tim's subject, threat detection. And in this case, logging. In the past, I've seen people attempt to create logging libraries and frameworks and tools and give them to app developers and say, Hey, developers, you want to produce useful telemetry for your security brothers and sisters? Use this library. Is this the best we can do? Like, how do we deputize or draft developers to help us in DNR to do our jobs?

If you're wondering, listeners, logs are Anton's favorite subject. Me, threat detection.

Well, they're together, right?

That's a dangerous topic. We're not going to have a pillow fight now. We'll have it next time.

How do you really separate the two, right? How do you do threat detection? detection without proper logging.

Good question.

So, I think you're probably both on the same page more than you may agree in this conversation. But I think part of the problem with the approach that you just mentioned is that this was a one-way conversation. The security team created this framework without talking to the development team in this hypothetical scenario and threw it over the fence. In reality, you should have a conversation between the security teams and the development teams, or else you're not going to meet the requirements that the development teams need in order to use this library. You should talk with them, explain what you want to accomplish, explain what data points are most valuable, and then work with development leaders. Have your security professionals work with your development leaders to create coding frameworks like the ones that establish guardrails in the AppSec world, but now to ship logs by default that you care about. And the security teams not only shouldn't write it themselves, oftentimes they can't write it themselves, or they'll take a really long time to write them, especially for all of the languages that they need to support. So make sure that these teams work together is really the short answer for almost every problem.

It's almost like you're suggesting some form of DevSecOps.

Yes.

Do we have to drink? It's too early here in California to drink, but I think that it's— I think you are suggesting the developers, ops, and security working together, right?

I think everybody should work together in general. I mean, learn how to play nicely with each other, but make sure— this is the one caveat I'm going to mention— DevSecOps does not mean give all of the work to the developers in operations and security and then QA. We just keep giving responsibilities to developers, and that's not what this should look like. It should look like collaboration, not throwing work over the fence.

Yep, that makes a lot of sense. So, on this note of kumbaya, perhaps, do you have suggested reading and one tip for our developer friends when it comes to improving cloud security?

Yeah, this is a perfect transition. So, I'm admittedly not a big reader, so if I recommend an actual physical book, you know I really like it. And if you want to understand how DevOps works and how to get security into the equation, I highly recommend reading The Phoenix Project and its sequel, The Unicorn Project. Not only is it very informative, but it's a genuinely good novel about a realistic organization going through transformation to DevOps. Then, of course, I would be remiss not to plug sans.org/cloud, where you can find a bunch of free resources like blog posts, white papers, posters, cheat sheets, and more, some of which that I helped create. And finally, if I had to give one tip on cloud security and app security, it would be to get your hands dirty. It's really hard to teach folks how to use technology securely if you've never used them in your life. Sign up for a cloud account with a free trial, launch some infrastructure, install some open source software, try to have fun with it, suffer with how difficult it can be, and build some empathy, which is something that we often overlook in our line of work.

Well, Brandon, an episode of wonderfully crisp answers, wonderfully deep insight. Couldn't thank you more for joining us today.

Really appreciate you for having me.

Perfect. Thank you very much. And again, I love the energy too, because like I got motivated to like AppSec and, you know, I'm not an AppSec person. Next time, what? Somebody will motivate him to love IAM? I don't know.

Let's have another conversation in the future. Maybe I can help out with that as well. Yep.

Awesome. Well, hey, thank you so much. And really, thank you for the diversion on guardrails versus settings. I really enjoyed that too.

Hey, all these terminologies, all these terms, we are working with something that's simply not scientific. So I try to come up with terms that are descriptive to me. I might be contradicting every other security professional's definitions, but hey, I think it makes sense, and hopefully it made sense the listeners as well.

And now we are at time. Thank you very much for listening, and of course, for subscribing. You can find this podcast at Google Podcasts, Apple Podcasts, Spotify, or wherever else you get your podcasts. Also, you can find us at our website, cloud.withgoogle.com/cloudsecurity/podcast. Please subscribe so that you don't miss episodes. You can follow us on Twitter, twitter.com/cloudsecpodcast. Your hosts are also on Twitter, @Anton_Chuvakin and @DavidMalan. Tweet at us, email us, argue with us, and if you like or hate what we hear, we can invite you to the next episode. See you on the next Cloud Security Podcast episode.

Thank you, everybody.