EP165 Your Cloud Is Not a Pet - Decoding 'Shifting Left' for Cloud Security

Security.

Hi there. Welcome to the Cloud Security Podcast by Google. Thanks for joining us today. Your hosts here are myself, Tim Peacock, a Senior PM in Google SecOps, and Anton Chuvakin, a reformed analyst and senior staff in Google Cloud's Office of the CISO. You can find and subscribe to this podcast wherever you get your podcasts, as well as at our website, cloud.google.com/podcasts. If you enjoy our content and want it delivered to you piping hot every Monday, please do hit that subscribe button. You can follow the show, argue with us and the rest of our listeners all on the Google Cloud Security Community page. You can find the link to that, the show, and our LinkedIn, which you should also follow for special video bonus episodes, all in the show notes. Anton, this is another one in our Cloud Security Voices series. Really excited to be doing these. And today we're talking about a fascinating backstory, the fascinating challenges of deploying successfully at scale. Where did we get these ideas about successful deployments? Where does that all come from?

I think it's actually, to me nowadays at least, it fits in the meta theme of doing security the cloudy way. Because it's been kind of an elusive topic for a lot of people. We tell them, hey, you lift and shifters, you're doing security as if you're in the '90s, as if you have a data center.

What does it mean to do it in the 2020s then?

Exactly, exactly. But people rarely explain what it really means. And this episode offers a glimpse of that type of what does cloudy security or cloud style or modern security, I don't know, don't have a cool term about it, really means. And there are definitely fun elements of that.

I think the cool term is just cloud-native operations. I think that's what we get a really solid look at in this episode. And so without any further ado, let's welcome today's guest. Anton, I'm pretty excited about today's content, not only because we're taking part in this new initiative, but also because we're gonna get into some really genuinely interesting terrain that mirrors closely some stuff you and I were talking about. This is the first time where we've had an episode come after talking about stuff as a work context rather than the other way around. So this is fun for me.

Right, correct. And I think we're gonna hit some fun topics. I'm still reeling from all the definitions of policy as code, so I'm hoping our guest would have his own fun definition 40 CS Code. Definitely looking forward to that one.

Yeah, I think that's a good thing to look forward to. And we've got a lot of other great content. So let's dive right into it. With that, I'm delighted to introduce today's guest, Ahmad Robinson, a cloud security architect here at Google. Ahmad, you recently gave an awesome Black Hat webinar where you talk about pets versus cattle when it comes to cloud operations. Because I associate this phrase, pets versus cattle, with the Kubernetes world. Could you explain what you mean when it comes to cloud for this?

Definitely. And thank you so much, Anton and Tim, for having me on. This concept— well, number one, I have to give awesome credit to my teammate and friend Elliot Abraham for coining this term in terms of security. So he brought it over into the security world. But definitely the first thing that you think of is Kubernetes, and you think about VMs versus containers and microservices and things of that nature, right? But it's really just this concept of, hey, we're doing something that is very inefficient, by far, probably manually changes. And all of these things lead to security vulnerabilities for the most part. So how can we change that from having a manual mindset, something that— having a mindset where you're not scaling, you know, in the best way? And how can we make sure that all of that stuff also not documented, also to the point where you have all these different silos who are doing different things? So it's difficult when it comes time to audit. It's difficult to enforce policy, all these different things, right? So the whole notion of that is That is a pet. You are essentially treating your environment as a pet. So you're taking very good care of it by making changes that are in production, like a virtual machine, say that you needed to get some type of a patch onto it, load it immediately, or you need to load some security tool on it. Well, you load it onto one individual tool, or you go around manually clicking on 3 different virtual machines instead of rolling that out as, you know, something that is baked internally into, like, a golden image so that you can just deploy it across your entire infrastructure. So it's really just that mindset, and now we're taking that into security.

Okay, so this is a nuanced and an interesting take on cloud security. So what in your past has led you to these insights?

Gosh, well, where do I start? I was, you know, born in Cinderella— no, I'm just playing. In my past, you know, really when it comes to security and, you know, this particular career field, it really came down to, to be honest with you, just a bunch of waking up in the middle of the night because you are, you know, you have pager duty or whatever the case is, and you have to get up and you have a phone call, and that's because something has gone down, or it's because of some type of security risk that has now sprung itself within your organization. So it's really just a matter of, you know, and I brought this up in the past, but we used to have this server that went down all the time at one organization I worked with. And, you know, we knew it by name. It was like DFW-23 or whatever it was. And every time it went down, we would always manually have to make changes. And we knew where all the little nuances were, but we still had to go in almost on a biweekly basis and make changes and adapt and bring it back online. And then even from a security standpoint, that particular server was very unique versus everything else that was in our entire data center. So when we had to roll new software out or when there were code changes, even from a dev perspective, everything was very unique for that environment. So, you know, when it came to inspections and auditing and all of those things, you know, it became very troublesome for the auditor to get a handle on what the heck are you guys actually doing in this environment, or what is your standard template for deploying certain workloads out there? So, so that's really what, what sprung this up. And that's really what, in my background, the main thing that brings this on. But I've also had some very interesting pets as well that I love. So I love talking about anytime I can bring pets or anything from my life into a work environment, I love it.

So tell us more about your background and your journey to Google.

My background. So that's what I was going to start with. You know, I was going to start with where I was born and all of that stuff, right? So, but, you know, where I really like to start is when it comes to the Air Force. So I joined the Air Force. Long, long time ago. I actually won't even say the year, but, uh, I got thrown into this environment where I was the only airman, like everyone above me. And what that means is I was the only one below like E-5 in an environment. So I was around very mature people. And then on top of that, I was thrown into an environment where there were a lot of contractors around and all of them were doing things around technology. And at that time I was very young and I just knew that I was very interested in technology and I just needed something. I didn't know exactly what it was. But particularly, they threw me into security. I started working with things like ASAs, and I started working with— that's where I learned Solaris. That's where I learned, you know, a lot of the first networking concepts. Eventually, I went on to get my CCNA and all those different things. So from my background, that's really where it starts. And then from there, it just became a journey of— by the way, networking is always and will always be— I will never forget my network brothers and sisters that are out there— networking is always my first love. I am a CCIE. Although today it's probably expired, but at the same time, I am a CCIE. I've had that for a while, as well. But security, as you all know, first of all, it goes with everything, but especially hand in hand with networking, especially when you're doing a lot in terms of firewalling and VPNs and things of that nature, right? So I've always been hand in hand with them. So, you know, several years ago, I decided to make the transition over to security, and then voilà, here I am at this point. So it's been— seems like a long journey, but at the same time, it's actually been pretty short. And it's crazy just how my life has been and just being thrust into it. But the biggest thing to know is that It doesn't take a ton. And that's really what I want a lot of folks to understand, because a lot of people see, you know, myself or they see some of my colleagues and they really think that it takes a lot to get into, you know, this career and all of that. As long as you have the interest and the passion is there, all of the information is— we're in this day and age, all the information is out there. You can definitely seek it out, gain different types of experiences and bring your own flavor, your own personality into this career.

Yeah, I love that.

We wouldn't hold the CCIE against you.

Okay. Well, thank you. It was definitely, uh, It's been a while ago, but yeah, I miss those days a little bit, if I'm being honest with you.

How has your diverse background sort of contributed to the team? We'd love to hear more about that.

Oh, my diverse background. So, number one, I'm going to start off the technical path here. And that is just, I think, and this is just from feedback of customers I've worked with over the years throughout the country and things of that nature, it's, it kind of probably starts with my personality. And I am who I am because of the background that I have. So, I've joked about it before, but, you know, born and raised in Southern Illinois and, you know, just bringing, I guess, a little bit of the Midwest into it, but also just a little bit of energy. I found earlier on in my career, especially being at certain locations in the military— in the military, things are very serious for a reason because of what the mission is. But then, once I crossed over and I left the military, I'm like, really? Our whole job here is just— at a lot of these businesses is to make money or whatever, right? So, it's like, hey, let's just have fun. This is technology. Let's have a good time.

Let's bring energy.

Let's be vibrant.

So, I think that diversity, just bringing that, I try to bring that into every conversation that I have is no matter how serious it is, though, let's try to, when you bring out that joy and release those endorphins in your brain, it just opens up so many more possibilities. And from a technical perspective, my background in networking and my background specifically with, you know, the military and different organizations that I've worked with really helps me bring that side of it to it, which I know a lot of folks come from that side, but really helps me understand, okay, from a networking perspective, that was what I consider the foundation. You really can't have, even though the most important crown jewel of everything is, you know, business, our business applications, is the software. That is the actual, you know, what brings in the revenue and things of that nature, right? But at the same time, the foundation of that, I feel like, is the network itself. So now that I started from the network and I moved up to security, and nowadays I'm learning more about development and things within Kubernetes and the serverless world, Knative and all of that. As we get into that world of it, it really has allowed me to apply a foundation to something, and I can speak almost— I'm not going to give myself too much credit here, but I can speak almost to the entire stack depending on who's in the room and who I'm speaking to.

I think this perspective, you would really like our guest from a couple of weeks ago, Marty Rausch. I mean, his network emphasis was really strong, and I think there's a great story there. I want to shift gears a little bit away from the network. And pick more at this cattle versus pets thing. You know, we hear about shifting left. Could you talk about shifting left in the context of security? And really what I'm curious about is not— I think a lot of our listeners are familiar with the concept or familiar with the concept of security. I'm kind of curious to hear your perspective on where people are struggling with that or getting it wrong.

Yeah, the biggest thing when it comes to where people are getting shifting left wrong, honestly, a lot of the issues when it comes to shifting left is really just a lack of communications between teams, a lack of collaboration. There's also a lack of involvement of security teams. So it's difficult to shift left when security teams aren't involved, for example, in a dev pipeline. And that dev pipeline can end, you know, once that application is dropped off. It can end in a virtual machine, it can end in a container, it can end in serverless. It doesn't matter. A lot of those times, those developers, you know, kind of leave the security team out of the conversations. And then, and there's also slow and probably cumbersome security processes that are out there. I've met with different customers and You know, they've talked about how the entire security pipeline, you know, to get something deployed or they just want to test something out, they have to go out and they have to fill out manually, you know, some different forms that are out there. They have to submit those. Those have to be manually, you know, checked and then approved. And perhaps that gets sent over to a different team and there's different time zones that you're working in and all of that. So it's really just a matter of how can we make that more efficient? That's really what's holding up a lot of that shift left mentality is that security, I think they've been hearing this buzz term for a while and they want to do it and they get all motivated. But then they're also not, you know, we're not communicating across different silos as they've turned out to be, but across different teams to bring them together so that we can collaborate and actually get something to where we are operating on a common platform, which is the biggest thing. You know, when we talk about shift and left, the biggest thing from a technical perspective is now you're on one common platform. So for example, if I'm talking about infrastructure as code, infrastructure as code allows everyone to collaborate and see exactly what's going on inside our, you know, air quotes data center, even though data center now has transformed to cloud, multi-cloud, hybrid cloud, all of that. But it allows everyone in that one common language to see exactly what's going on, exactly what the configuration is, what the policies are, and all of that. Versus if we were to go, you know, 15 years ago, I couldn't go to one single spot to see exactly what was in my data center. I had SAN storage, I had servers, I had networking that had its own coding language or configuration. All these different tools use their own language. So now, From a technical perspective, common language with infrastructure as code. And then from a human perspective, it's communication and collaboration in terms of let's get together and let's find out how we can all get on the same page. You that are in development, you want to just deploy something as fast as possible. You want to use your creativity, all of that, right? Whereas me, you know, in security, I want to obviously protect that as much as I can, but still give you the right freedom. So how can we bring that together? And it's just opening up that communication.

By the way, do you also see a mistake when people shift left and they forget right? Occasionally, I've seen teams that kind of embrace shift left, but then they really neglected the detection response, the runtime stuff. It's almost like they went left too far, and they forgot that right existed. Have you seen that at all?

That is definitely the case. So, for some that are very far thinkers, and they just take it and they run with it, you know, to the left, they run with it to the left, they can kind of forget some of those tools that are to the right. So, for example, Even, you know, tools like SOAR tools that are out there and all that, you can still integrate those and keep those inside your environment. Or let me go through and describe a specific use case. Let's say that we shift left. And when it comes to, let's just use a virtual machine image. So let's go back to the pets versus cattle mentality. A pet mentality or a paddle— see, I almost messed up there, Tim.

See, I messed people up with that.

A paddle mentality there would be if I deploy virtual machines, you know, individually and I'm choosing each individual image. And then again, just as I was mentioning before, every time I need to make a change, whether it be from an application perspective or be from a tooling perspective, I need a new logging agent, whatever it is, I go individually to those VMs and then I, I make those changes. Very bad thing because now we're not standardized across our entire environment, all of that, right? So now if we shift that to a cattle mentality, that's where we leverage a custom image pipeline. So we create— we leverage tools like Packer or Cloud Build, actually those 2 together, for example, where Cloud Build can orchestrate and leverage Packer to build an image in your environment. And then now that becomes your golden image, and it's not just a single golden image, but it also has your standard operating environment for all of your different use cases across your different teams. So then those images get deployed into that environment. And now once they're deployed and they're actually running, what do I do? Do I just— so we have a tool here at Google Cloud called VM Manager. VM Manager is responsible for things like OS inventory. It's responsible for OS policy configuration, things of that nature, right? So because let's just say, for example, if I needed to roll out a new patch, If I rolled out that new patch through my custom image pipeline, does that now mean I don't need to use VM Manager to ensure that there is a new patch that is uploaded onto my operating systems? No, you want to come at it from both sides of the angle there. So I want to shift left and leverage that custom image pipeline to bake in the right security tooling. But then also I want to come at it from a right perspective and say, hey, while you're deployed, I need to continuously monitor and make sure that this is up to date, up to our policy standards. And then if not, we want to remediate that. There can be some type of way through break-glass scenarios or all these different ways ways that you can go around your pipeline. So you just want to ensure that if worst case someone does, we're still doing the right thing. So hopefully that answers the question there.

I think that's a great answer. It was a thoughtful answer that covered a lot of ground. I think one more thing on what Anton was saying, just because it's a hobbyhorse of mine, I've seen a lot of the cloud third-party vendors really neglect not just the scanning side but really the threat side as well. And I wouldn't want to see security teams get into a world where Sure, they've got great shift-left coverage, but they're not seeing the threat actors once they're in. So, I think as people are thinking about cloud, shift-left solves one set of problems in concert with other approaches, but it doesn't have a lot to say about the right-hand side of security. And you can't really operate just with preventive things.

Funny enough, just before this recording session, somebody said they're going to launch a startup and their slogan would be shift-right.

Mm-hmm.

Because to really reenergize the cloud-focused people who maybe shifted left too far, to remind them that the DNR and the right-time stuff still exists, like, it's not gone. It's not like you fix it in the pipeline and it's fixed forever. So, this is kind of fascinating, a fascinating discussion. But let me try to touch on the infrastructure as code bit because you mentioned it a few times, right? So, A lot of people who are kind of very outspoken in the cloud, they understand cloud, they talk about infrastructure code and policy as code. And this occasionally confuses people who are still coming off the data centers, still coming off their CCIE certification classes, right?

Hey.

So, what's your approach? Well, you've transcended your routers and switches. Clearly, there's no debate about that. There's no debate. But some people have not. And how do you explain infrastructure as code, as policy as code, to people who are still in the data center mindset? This has been a pain in our podcast over the episodes. A lot of people are still not quite sure what to do with all the modern stack stuff. How do you teach them policy as code?

It's a good question.

Yeah. So, first of all, I didn't transcend. All of that stuff is in me. All of the old CCIE stuff is in me. So, I definitely don't want to leave Like I said, my fellow brothers and sisters out there are out of it. But when it comes to policy as code, it's really just a matter of, you know, so ways that I've kind of described it in the past is, you know, this is just really your way of checking, you know, as folks maybe are entering a club, you know, you're trying to have a good time. You got all of these infrastructure admins that are out there. You got all of these, your network admins, your firewall admins, all of them. They're all trying to enter this club. And this club is your enterprise, right? And what you've now done is you've placed a barrier around that club. And that is basically that infrastructure as code. You placed a barrier around that says, hey, at the very least, you know, you got to come through this door. And that's especially not— that's not just infrastructure as code, by the way, for listeners out there. That's also infrastructure as code with a GitOps-style pipeline.

Mm-hmm.

So you want to make sure that all folks are entering through that door. Policy as code are the bouncers. They are the bouncers that are at that door that are going to check those IDs. They're going to frisk you and do all of that stuff to make sure everyone's just having a good time once they get in there and that you're the right folks that are getting in there. They're also the ones that are also going to make sure that you're matching that guest list that is there, right? So that's really what it comes down to as policy as code. That's ultimately what you're trying to do there. There's 2 separate things, but they work in concert together. And I think the policy as code aspect, especially when you take advantage— so for example, I do demos where I show my GitHub environment interacting with, of course, my Google Cloud environment. But then I'm leveraging Open Policy Agent, or OPA, to enforce certain things like ensuring that I have labels attached to all of my different workloads or enforcing, you know, different naming conventions and things of that nature. So it's just a matter of making sure those little checks and balances are there. Now that's me using it for very simplistic use cases, but expand that out to just ensuring and setting constraints or boundaries. And we have internal constraints here at Google Cloud, and not to bounce around too much, but we have org policies and things of that nature that you can leverage with Google Cloud as well. But if you need to go beyond what org policies can do, or just like I said, little nuances for different workloads and resources that you're deploying in your environment. That's where that policy as code comes into play, and it becomes your checks and balances. And then you could show that in concert with the infrastructure as code, your policies, and then also shifting right and showing your compliance in your current environment. Showing those 3 things to an auditor really paints a very clear picture of what is going on. Here is exactly what's deployed. Here's my desired state for my entire environment in infrastructure as code. Here is what we're preventing, how we're you know, kind of squeezing that out here, and here's exactly what we're looking for when it comes to that environment. And then now, I leverage a tool like Security Command Center to verify that I am, you know, compliant in PCI DSS or whatever the case may be. So leveraging those 3 tools becomes very powerful for, you know, your QSAs or your auditors that are out there.

Hey, Ahmad, what was that last tool you mentioned? Security Command Center? What's that?

Ah, no, you're doing it, Tim. Security Command Center. I don't know. Is this a tool I just read about prior to joining this? No. Awesome tool there.

Listeners, the joke is I'm one of the PMs for Security Command Center, so that's near and dear to my heart. But unfortunately, as much as this conversation is near and dear to my heart for all the reasons, we're about at time. We have to wrap up. So I want to ask 2 of our traditional closing questions. First, do you have a recommended tip to improve people's cattle versus pets mentality in cloud, of course, other than using Security Command Center? And then 2, do you have recommended reading for people?

Oh, recommended tips. So my recommended tip, I'm going to start with the simple one that I alluded to earlier, which is start the conversations. I know that that sounds very simple and it's probably overused, so I don't even know if it's going to fully take effect, but listen because it's coming from me. Um, you'll start the conversation, get in a room or get on a call with different folks from other teams that are across the organization. and all of the teams that make these things work and find out what exactly those teams need and how we can work together. Because that's, again, it's always the people side of it that hinders everything. The technology has caught up. It has been caught up for several years now. So, it's just a matter of making the people side catch up. And then, when it comes to recommended reading, that's a tough one. But, you know, given that we're in the age of AI and, you know, you don't want to play those games where you drink every time you hear those 2 letters there. But I would just say, We've talked about pets versus cattle mentality, but I would encourage you to read, like, the blog post. Once you get that pets versus cattle mentality and you've solidified that in your organization, the next step is looking forward to the future. What's going to change, you know, moving forward? So just some interesting, maybe not anything you need to do today, but some interesting reading could be Security AI Workbench. We have a blog post that's out there on that. Check that out. And it's something that, you know, can just get your brain working towards a way of how AI is going to change security moving forward. Yeah.

Ahmad, I love that tip, especially because so often on the show, we like to joke, you know, CISOs don't need another tool. They need a family therapist to improve their relationship with other teams. And I don't know a therapist on the planet who doesn't start with, well, have you tried communicating about it? So that's— maybe your future career is not, you know, Air Force guy, not network guy, not cloud guy, but family therapist for CISOs. So with that, Ahmad, Aditi, listeners, thank you so much for joining us today. Thank you.

Thanks for having us.

Thank you.

And now we are at time. Thank you very much for listening, and of course, for subscribing. You can find this podcast at Google Podcasts, Apple Podcasts, Spotify, or wherever else you get your podcasts. Also, you can find us at our website, cloud.withgoogle.com/cloudsecurity/podcast. Please subscribe so that you don't miss episodes. You can follow us on Twitter, twitter.com/cloudsecpodcast. Your hosts are also on Twitter, @antonchuvakin and @tinafernandez. @Anton_Chevaikian and _TimPico. Tweet at us, email us, argue with us, and if you like or hate what we hear, we can invite you to the next episode. See you on the next Cloud Security Podcast episode.

Security.