EP80 CISO Walks Into the Cloud: Frustrations, Successes, Lessons ... And Does the Risk Change?

Security.

Hi there. Welcome to the Cloud Security Podcast by Google. Thanks for joining us today. Your hosts here, as ever, are myself, Timothy Peacock, the product manager for threat detection here at Google Cloud, and Anton Chuvakin, our reformed analyst and senior staff in Google Cloud's Office of the CISO. You can find and subscribe to this podcast wherever podcasts are available, as well as at our website, cloud.withgoogle.com/cloudsecurity/podcast. If you enjoy our content and want it delivered to you piping hot every Monday, please do hit that subscribe button. You can follow the show and argue with your hosts on Twitter as well, twitter.com/cloudsecpodcast. Anton, today we are kicking off the first in what is, I believe, a many-part series, it's a number too big for me to think about, of talking to CISOs on the show about their journey to cloud. It's not quite Journey to the West, but Journey to Cloud.

I think it's really more about their first, usually scary encounter with the cloud. I think your definition just lacked drama a little bit. I feel like we are aiming for CISO meets cloud. You won't believe what happened next.

I mean, what happens next?

Okay.

Maybe that's a little too cliché. Fine.

No, no, it's great. It's great. We should go for clickbait. That's a good goal around here. So.

Very respectable.

Very respectful of us. But, but today's guest, David, I think is really wonderful because he tells such a lucid story of it not being just a technology thing, not just a people thing, not just a process thing, but really weaving those 3 together.

Yes.

Yes. And with that, let's not take any more time from David. Please welcome today's guest. We are delighted to welcome today's guest, David Stone, a key member of our Office of the CISO here at Google. David, we're talking about your journey as a CISO in your past life when you were migrating to cloud. Could you give us the 30-second overview of, let's start with what triggered your org's migration to cloud? I imagine one day, like, your CIO was struck by a lightning bolt that said you must migrate to cloud. Is that how it went? That's what I always picture anyway.

Well, yeah, the lightning bolt was pretty dramatic too.

What was the lightning bolt?

You know, as they got struck by it. But yeah, so it all really came from the business one day and the CIO in concert. looking at cloud as a real means for us to migrate and really up our game across the technology board. But it also was really stemmed from a couple things, right? We had legacy hardware, legacy software, and business drivers that were really looking at how do we make this more efficient along the way. So as we were looking at that solution set, we looked at the cloud as a real means to really accomplish our business goals across the board and really drive transformation for our organization into the next level.

So when we hear this, when I hear this, I'm always curious, was security team brought in at step 1, step 2, or step 37? Or they read about it in the news? How did it happen in your case? Or like, you know, feel free to obscure some details, but ultimately we do want to know how often security team gets invited into the cloud party.

Yeah, so in our case, we were really, really entwined with the business. So part of our function was making sure that we were out in the business every single day, asking questions, really involved. So we actually really got involved in this early on. When the CIO got brought in, we got brought in with the technology team right from the onset of, hey, we're thinking about moving to cloud. We need you as a partner in this journey, and we need to make sure that we do it in a way that, you know, we have security embedded in and quite frankly, that we don't hit the front page of the newspaper, right? We make sure that we're deploying this in a secure and repeatable way. So we really got brought into the front of that planning cycle, the front of the business objectives, and even to the resourcing And making sure that we had the right budget to really go after security in the right way as we went through the transformational journey.

So any more insights on the planning? Because I think that I get the broad picture, and I think I'm happy to hear that your case was security team not reading it in the newspaper, on CNN, but like being involved in planning. But like, how was this planning roughly ongoing? Like, what did you do? What did they do? Sorry if it's a vague question. We are kind of fresh. testing this, but I'm kind of curious, how was this planning organized?

Yeah, so that's a great question. So one of the things with our organization, we were immature in the cloud when we were looking at this, right? So we really thought about—

Most people are.

Yeah, most people are. I mean, honestly, as I'm more in the office of the CISO, I see a lot of people coming from the same places with limited understanding of the cloud and kind of the journey and the things there. So we brought in actually a partner to help us out with the planning and the journey of this to originally, how do we frame this? How do we look at this? Along with our internal project management staff and everything else, we have big enterprise project management resources that we brought in that do their typical things along that. But it also required us collaborating with a partner in this aspect, collaborating with our internal security teams, and collaborating with the business on what is the objective, right? What are we trying to achieve here? In our case, you know, we were looking at kind of a wholesale data center exit from one of our business units. So we had to look at the various angles of that and how did we want to migrate, reduce our technical debt, reduce our cyber debt, and really make this hopefully a frictionless environment for all those people that were going to be using it in the future in terms of how do we get security cooked into this and be kind of in the background, if you will, as much as possible while still innovating and driving the business goals that we really wanted to achieve.

Okay. So, there's 3 parts I want to pick at there. One, what did the collab look like? How did you make the collab technically and tactically successful?

Yeah. So number one thing is collaboration. Overcommunicate, overcommunicate, overcommunicate. What you think your assumptions are are not the assumptions probably at the end of the day, right? So we really had to bring in like the key stakeholders from the business and really dive deep into it, right? Sometimes they had a limited understanding of technology. Sometimes they had a limited understanding of security and vice versa. Sometimes we had a limited understanding of what the business was, right? What they were trying to achieve at the end of the day.

You're telling me security didn't understand the business? No way.

I know. Put on your shocked face. Yeah.

Shocked Pikachu. Okay, so the collab is like, question your assumptions, overcommunicate, stay in sync. I want to pick at also the story about reducing debt. A lot of the time, Anton and I speculate, oh, when people come to the cloud, do they think about opportunities to get rid of X, Y, and Z? Was that what you were doing there? Were you thinking, how can I stop doing this thing? Or were you just like, how do we get there?

Oh yeah, we were first and foremost, how do we stop doing this, right? So if you think of any data center or any large-scale migrations, we probably had quite a little bit of tech debt that was built up over the years, right? Really good-intentioned ideas and things, but eventually, they were kind of put together in various different ways that at the end, you kind of scratch your head a little bit and you say, okay, is there an opportunity here to reduce all this, I'll say, debt or over-engineering in some aspects and really just streamline how we do it? But you really got to start in that journey, and it'll go to kind of my second point, is from a security team, we really wanted to hit 2 security objectives. Let's not just focus on the way we did it yesterday, but what are we actually trying to achieve here? What is the goal from security? Let's put that in a story format, in a format that the customer can understand the business, and then we can put it into a technical, how the heck do we get things done in order to get to that goal and to that objective? But really looking at it from that perspective. And I think the third key point is organizationally, it requires you to think through how you're organized. Most security teams, if I look at them, you know, they've been organized over the course of many years to kind of align to security frameworks, right? I've got my GRC team, I've got the security architecture team, I've got the vulnerability team, I've got data protection, I've got security operations. I've got all these teams that have built up their domain knowledge, but sometimes they don't work together, especially in large organizations, to all get to that same objective. And it really requires you bringing all those people internal to security together, educating them on what the cloud is, what our objectives are, and then how do we get there together as a team and really pulling those folks in at the onset of that to get to those security objectives. And it really plays into that resourcing and how do we actually plan the work? Because if you don't have that upfront, you can't build in the time and the capabilities necessary to get there and really understand how do we need to get to the business to that objective. And knowing that we may not look organizationally like we do today tomorrow, right? But that's okay because we're going to have more time and energy to go hunt bad guys and do really interesting security work if we eliminate some of this technical debt and cyber debt that we built up over the years.

This reminds me of an awesome tweet I saw the other day. It was, it was 2 panels. The first panel was, who wants to see the org change? And everybody had their hand up. And then the second panel was, who wants to change? And nobody had their hand up.

Nobody.

It's always the other guy. Oh, that's right.

So when you were working the org through this, how did you, as the leader of the organization, manage not only the technical aspects of change, which surely were complicated, but how did you get people to change as you went through this?

So honestly, that is the hardest part. Technology has far outpaced the ability for any large-scale organization to adapt and adopt it in a full-scale manner, right? organizational change is the hardest part, in my opinion. Getting people to change, I would say technology is easy, people are hard, and organizations are harder, right? In order to get to that point, and it's a lot of really well-intentioned people trying to get to the same area, but we need to figure out how do we bring them all together. And it really aligns on the mission statement, right? And I found that effective if we can be clear and concise about where we're trying to head to, what our goals are, and what your role is going forward, right, in the organization and how you're going to be able to Grow, grow your skill set, be able to really adapt to the new things coming. And by the way, all that stuff that you moan and groan about on a daily basis, if we can get some of that out of the way so you can go do the fun stuff, let's go focus on that and let's figure out how we lay the foundations so that we can all get there together and do some really neat stuff and really drive the industry forward in the future.

So you give them a concise definition of what's going to happen next and show them the shiny object. in your hand. The— here's where we're going to go on the hunting. That makes sense.

That's right. And their path, right? That's a real key important piece is like, this is your journey path, this is your learning path. And I haven't met a security professional yet that doesn't love to learn, right? They wouldn't go into this profession, I think, without the onset of, hey, how do I learn and how do I be better? Just because they have to learn on a daily basis, right? That's what security people do, right? You have to understand a lot of different domains and learn very effectively across it. So absolutely, you got it.

So it's interesting how These remind me of a very common line we mentioned in the podcast. They migrated the technology, but they did not migrate the thinking. So it sounds like David is outlining some of the elements of actually migrating the thinking. So to me, this is like I'm jumping from joy. The audience probably does not see that, but—

He has an audience. He's jumping up and down in his chair. It's adorable.

Some of this can kind of frame out the principles or the ideas or the approaches. I don't know what the right word is, where you are changing the thinking. And that's how you do it. You don't just say, This server, go up. Because that's how you get lift and shift, and that's how you get all your mistakes copied to a cloud provider. But I think this whole planning upfront thing and a few other items, what do you want to achieve versus what do you want to copy, are some of the elements. And I think I love it. It's probably going to be useful guidance to our clients.

Absolutely. And that doesn't go without challenges, right? There are some pieces of the organization, like you said, that are going to pull their hand down of who wants to change, right? And you've got to address that. And really, those are the folks that you got to spend a lot of time with, right, to understand, you know, what's the benefit to them, what's really going to come out of this, and how they can really play a future role in really shaping this and making it beneficial for them. But like I mentioned before, organizational change is hard, and I wish there was a one-size-fits-all, but there's not, right? Each organization is a little bit different in their dynamics, whether they're highly regulated or not, whether they have to adhere to certain things. Fantastic, but it's really about the security objectives for the security team. So focusing people back on that, saying, we've been here, we've done that, we can change again, right? Let's do this better. Yeah.

So we asked about the hardest part, rather you mentioned the hardest part, but what about anything that was like the most surprising? And of course, as security professionals, our minds perhaps right away go to bad surprises, but maybe there were good surprises. So what was the most surprising part of the migration?

You know, I think the most surprising part for me as a security leader was trying to help the business truly understand the value of security embedded in the ability to really influence and make it a differentiator within our product sets. And then vice versa, the security team understanding the business side of it and the context that they were coming from in order to really integrate our security to be frictionless. I think it surprised me how much effort, and as much as I like to say that we were trying our best to be there, We weren't there yet. And it was obvious through the discussions and the communication that, hey, we still had miles to go between the two in order to get everybody on the same page on basic definitions. And then let's move over to the cloud definition of how do we do it in the cloud, right? So it was amazing to me, and even though we had put a lot of time and effort into that, and I think it really goes to the education and the opportunity to collaborate, that you can never overdo those things, right? Because you will always touch a new piece of the business or a new stakeholder that you maybe haven't had the opportunity to introduce to and the concepts and them on board with, why is this thing important? Why is security so important to me? Why do I care? And how will this help me make money at the end of the day from a business perspective? It was really surprising to me, as much time and effort that we put into security training, awareness, and all the right things, that those conversations kept recurring across it. And it was a good conversation because I think it made us stronger as a team going forward within it. But, you know, out of that too, there were some hard moments within that and lessons learned, right? Obviously, when you lay down cultures in new organizations, those are hard things to foster and bring up. It's really hard in getting people out of somewhat of a racy mindset to, no, no, how are we really going to operate this thing on day 2? Like, okay, I've got an access request that comes in from an end user, a developer, a business user. How are we going to service this to make sure that it's adequate, it doesn't jeopardize anything within production, and that we can maintain this on day 2 operations? Getting people wrapped around solid concepts and how we're actually going to do this was, I think, one of the surprising things to me of how much time and effort we had to put into how we operate the dang thing at the end of the day, or how we operate the environment, right? Because a lot of organizations don't put that up at the front, but how we operate really dictates a lot of things about how we're gonna deploy the technology, how we're going to maintain the technology, and then what's gonna be ultimately effective. Because if you don't do it for day 2 operations, it's just gonna fall flat on its face. We've designed a really great system, it's fantastic in concept, but nobody can operate it. Well, that security control is no good because it's going to fail on day number 2 and day number 3. So, how do we really make sure that everyone's on board with what their day-to-day responsibility and how we're going to make it better for them? I think that was a surprising piece for me, not only in the cloud, but even on-prem, right? We had some maturity to do on-prem, and that kind of naturally migrated into our cloud, and then how do we make it better for the future as well?

Well, that's really interesting. So one thing that I think some organizations struggle with technically is how do they validate their security controls in the cloud? So as you were into day 2, into day 3, what were you doing to make sure your controls were, well, working as you meant them to be?

Yeah, I think this is the layered defense in depth kind of concept and approach here as we looked at it. And there was a couple aspects to this.

Did you have a debate about whether defense in depth survives in the cloud? Wasn't there somebody who was trying to convince us I mean, I'm pretty sure it does, but just a fun side conversation.

We also had the security onion versus garlic debate with Seth. That was a fun episode.

Yeah.

So really, like when you were verifying controls in the cloud, what did you do?

Yeah, absolutely. And I think that we'll debate on whether defense in depth is dead or if it's a layered onion or however you want to do it. Belts and suspenders. We could probably come up with a couple other good things.

I call it security lasagna. Many layers of cheesy goodness.

See, I love lasagna, so I like that analogy. Just because it made me hungry. See, people love lasagna. So really, it was around a couple layers, right, of that lasagna, if we will. You know, first around how do we do organizational configuration management, right? Because a lot of controls in the cloud are really around organizational policies and controls. How do we have a tight handle around configuration management of those? Then we move up into the stack into things like infrastructure as code, right? How does our infrastructure as code pipeline Really make sure that we validate controls. We've got golden images that we always pull off of that are hardened. How do we deploy them safely in a secure blueprint that we've pre-approved and make sure that our developers are using the common ingredients to make that lasagna, right? And then as we move up the stack into that application development, how do we make sure our CI/CD pipeline from an application perspective has security controls embedded into it at the right stages across that? And then you layer on top of that one, you know, kind of the cheese on top of it, right, is the security operations function, right, of how do you make sure that you're operating that environment and what you thought you deployed with all that fantastic technology up front and with all the pipelines is actually what got deployed into the operating environment in a way that was consistent with the right configurations. And I think the key here is you're never going to get it all right on day one, but if you set the foundation right and you get the big stuff out of the way and you iterate time and time again, that's what makes great security teams even better. The ability to iterate, the ability to learn, and the ability to quickly adapt. And it's really about getting the right controls or the right hooks in the right places so you can slowly tighten those down and you can make sure that it's even better tomorrow than it is today.

Okay. So there's a few things in there that I love. First, listeners, that last point about iterative improvement of our security systems, that's really one of the core tenets that Anton and John Stone lay out In their ASO paper. And I think what's so interesting is we have opportunities for both major and minor loops of improvement in security, particularly as we go to cloud and automate things. Second, I love the analogy of SOC as the cheese on top, but I will tell you, if your SOCs smell like cheese, you need to wash more often. But then finally, this piece about developers using tooling correctly, I think that's really interesting because so much of what we do in my product is help security teams enforce correct usage. And so I'm curious, what were you doing to make sure that your product teams, your engineering teams, the people building the business were actually doing the right thing? How did you do that tactically?

So tactically, there's a couple of steps. One, it always starts, I think, with developer education and awareness, right? They got to understand why they're doing something in order to get behind it as well, right? Just like the security team had to get behind the security objectives, the development team and the application teams have to understand the why in order to invest the time and effort and not just see security as a roadblock down there and why are they preventing us from doing things. So, I think the why is so important with that education and culture shift. Tactically speaking, how we did that is we slowly went through that with our development teams and worked through what would work and what would not work in terms of policy and compliance as code, right, and security as code. And how do we slowly migrate to a great place over time in order to get our entire inventory of code up to snuff. And we did it within different control checks, right? As we went up the stack within our development pipeline, we would enforce more and more controls as we went up, ultimately getting to the point where we didn't even want to get them out of the development environment unless they really hit most of the prod security controls. Because guess what? Every time you go back down to the dev environment, it costs more time and money. So, we really wanted them to get all of those into the pipeline sooner. And by involving them in the conversation and telling them, this is our game plan, and, hey, guess what, if you go into prod, you might get kicked back to dev, they started saying, well, why wouldn't we just make sure that all of our code complies down in the dev environment and be done with this thing? It's like, that's a great idea. We're really glad you came up with it, right? So, you know, it's one of those things, right, where you gotta get the people understanding the why and what the objective is. And it's amazing, really smart people come to really great conclusions on their own, right? And we all work as one team to get there. Really good stuff.

So, one of the fun things that tends to draw listeners is, of course, the mentions of security operations, and not in a cheesy context. So, I know it's kind of one particular angle on the SOC, but in many cases, what we've seen is that even if the security team or some part of the security team is invited for the whole cloud migration show, SOC is invited very late. So, how did you deal with the whole detection response expansion of responsibilities? Just curious how, in your scenario, in your first encounter with cloud, how did you deal with your SOC or DNR team or however you named them?

Oh, they were right up there front and center, right? Because we wanted to be outcome-driven, and the best way to do it is go to our SOC and say, hey, what are the threats you're seeing in the environment right now? Could you start with the threat intel of like, what's our biggest threats in the cloud? What's our biggest threats overall? And let's make sure that we build in our defenses in order to address those biggest threats that are coming after us. Having them involved early was so crucial. It was just mind-blowingly crucial because there are aspects of, okay, I need to log out of these particular scenarios, these particular areas in order to address the overall threat use case. Give me visibility to it and then be able to detect and then respond to something when it does happen or if it does happen, right? Probably more a matter of when and how do we respond from a forensic standpoint in order to be stood up and be ready to rock and roll so that we're not losing valuable times on configuration of IR, right?

So let me pick at something there. You mentioned threat use cases.

Yeah.

Did you come up with a new threat model for cloud? And did you worry about the same threats once you were operating in cloud as you did on-prem? This is a perennial debate Anton and I have.

So yeah, this is our other favorite question.

Yes. Oh, I love the question.

What changed when you went to cloud for your threat models?

I think the actors didn't really change, right? The people coming after us didn't change. They changed tactics, right? Based off what works and what doesn't work. So in a typical on-prem environment, you have to worry about different layers of the stack, right? From all the way down to infrastructure even further. In a lot of those cases, the cloud providers have a lot of that figured out already for you, right? They provide really good security. And here at Google Cloud, it's fantastic, right? All the way down the stack.

Hardware root of trust with Titan chips, et cetera. Yeah, it's cool stuff.

Absolutely. Zero trust deployed across, you know, end-to-end encryption, all the fantastic stuff that, you know, most—

So some threats go away in cloud. But did you have new threats you worried about?

That's exactly it. So obviously the attacker is going to go after the weak point, right? So that just shifts the amount of attacks to, okay, things like user error, configuration errors, API security, data protection. You know, you got your application stack layer. All of those things that were still prevalent in the on-prem environment just become way more important on cloud because now they become the very weakest link to that stack from an attacker point of view. And they're always going to go after the people and the process piece of that. because they know that that is the weakest link in our technology stack. So as we shifted there, we really looked at, okay, we want to make sure we protect our data, fore and foremost. We know that configuration management is an issue for most organizations, right? Getting that right. So let's figure that piece out and then let's figure out access. Let's figure out how do we lock down access to the furthest degree possible and keep tightening those screws as much as we humanly can with Things like the concept of zero trust, right? But that's a long journey to get there. So how do we do practical things in incremental steps to get us to a zero trust model? And then how do we then take a look at that and then figure out internally how do we monitor and get full visibility into the environment? And then we can start focusing in on the more threats down that model as you progressively get deeper and deeper into the stack. And like I mentioned before, you can't get it all in one swipe. The important piece is to make sure you have the controls and the hooks in place that you can slowly tighten that over time without a major amount of effort, right? It's slow incremental changes that you can easily back out so that you can make sure that you're balancing business operations and security all within that cycle.

Perfect. That's actually useful. I think I— we have a whole bunch of other technical questions, but in the interest of time, we'll just ask one strategic question. And it's a trick question.

No.

Is moving to cloud a net risk reduction? And of course, if David says no, I would say Phil Venables thinks yes. So, just be real careful answering, but can moving to cloud be a net risk reduction?

Anton, it's really not nice to set people up to disagree with their boss.

Ah, whatevs. You know, we're having fun.

Is moving to cloud a net risk reduction? Yes, it should be. And the reason why I say that is that, you know, configured with the right blueprints, with the right thoughtfulness, with all of those things, it absolutely is a huge risk reduction. Where we see, I think, people getting into trouble is not having the forethought, jumping straight into the technology and deploying things in a wild, wild west scenario with no controls and no governance around it, right? And again, the attackers go to the big open area that says open sign, come hack me here, right? So if you leave all that open for them, obviously they're going to waltz in the front door. But for most organizations that we see take a thoughtful approach from an organization operations and a technology approach, It's a huge risk reduction because now you've eliminated a lot of the vendor-in-the-middle risk. You've increased your access posture from a zero-trust model, and now you, you're really focusing in on that top level of threat of, okay, now how do I secure my applications and really get that bar raised across the entire spectrum? So it reduces the attack surface for an attacker to come after you, and it raises the bar and your focus on those other areas where you can make huge improvements by focusing your energy and efforts. So, absolutely across the board. And then I'd be really bad if I didn't mention visibility, like having the ability to see things that you never were able to see in your on-premise environment and have the ability to look at those and really bring up your capabilities as well as your visibility is a huge benefit. And I think not a whole lot of people recognize that that doesn't just go for security, it goes for like, hey, resource management, funding. I can now see how much this thing really cost me, and let's do cost optimization from a FinOps perspective, from a security perspective, and really look at our controls and how effective they are versus how much they cost us. It's a fantastic visibility mind shift, not only on the technology side, but from an organizational perspective.

Oh, that's an awesome answer. So we're now at time, and if you've listened to the show, David, which I think you have, you know what's coming next. First, do you have recommended reading? And second, Do you have one tip for helping other CISOs go to the cloud?

Oh, recommended reading. There's been so much really good stuff on this. I'm not sure I can top any of it. I would say that I think one of my kind of favorite mindset books is definitely the SRE book from Google, right? Of just how do you stay there? How do you learn and how do you adapt over time? Right? I think it was—

You have to put a GSU in the swear jar for recommending that one.

That's right. I'll take some of that as well.

That's right.

I think that that's it. And I'd say one big tip is really at the end of the day, you want to make sure that the security foundations are in place, but you're also adaptable to where the business needs to go, and you recognize and you're able to adapt that approach so that it works for everybody. Because just like, you know, that configuration management where if you can't operate it on day 2, it's no good, the same is true of the business. If the business can't operate with the security controls you put in place, it's time to rethink, right? What gets both of you to the security objective you want to achieve at the right pace?

So get the foundations right, but don't set them in stone. I think that's a great answer. David Stone, thank you so much for joining us today.

Thank you.

And now we are at time. Thank you very much for listening and of course for subscribing. You can find this podcast at Google Podcasts, Apple Podcasts, Spotify, or wherever else you get your podcasts. Also, you can find us at our website, cloud.withgoogle.com/cloudsecurity/podcast. Please subscribe so that you don't miss episodes. You can follow us on Twitter, twitter.com/cloudsecpodcast. Your hosts are also on Twitter at Anton_Chuvakin and _TimPico. Tweet at us, email us, argue with us, and if you like or hate what we hear, we can invite you to the next episode. See you on the next Cloud Security Podcast episode.

Security.

Security.