EP105 Security Architect View: Cloud Migration Successes, Failures and Lessons

Security warning.

Hi there. Welcome to the Cloud Security Podcast by Google. Thanks for joining us today. Your hosts here are myself, Anton Chuvakin, a reformed analyst and senior staff at Google Cloud's Office of the CISO, and Timothy Peacock, a senior product manager for threat detection here in Cloud. You can find and subscribe to this podcast wherever podcasts are sold and distributed, as well as their website, cloud.withgoogle.com/cloudsecurity/podcast. If you enjoy our content and want it delivered to you piping hot every Monday, please do hit that subscribe button. Please rate our podcast if that option's available in your app. Of course, we only want you to rate it if you love it, love it, love it. You can follow the show and argue with us on Twitter, as well, twitter.com/cloudtechpodcast. You can also use our LinkedIn page. So today's episode is about cloud, and it's also about lessons and, dare I say, some failure.

I think it was about people.

It was about people for sure. And I would say that it was more about people than about failure. It was certainly about lessons and kind of how cloud migration is ultimately a people exercise first.

Yeah, yeah, yeah. It's about how you fail at moving people to cloud, not how you fail at moving technology to cloud.

Yeah, kind of both.

Okay. We spend a lot of time on technology actually, listeners. This was one of those fun ones where we got to talk about whether firewalls still have a place in cloud.

That's true. And we're also going to talk to somebody who has a very direct, very hands-on, and very lengthy experience moving people to modern architectures, and somebody with really a lot of depth in architecture. This, to me, is quite exciting.

Yes, that depth of experience really comes through in her answers. And perhaps with that, let's not spend any more time. Let's get right into it. I'm delighted to introduce today's guest. We're joined today— and this is, I got to say, one of the coolest cool titles we've had on the show yet— Michelle Chiburka, Security Bene Gesserit and Cloud Security Advocate. Michelle, we're here to talk a little bit about cloud migrations and some of their failures, you know, because on the show we like to talk about how things go wrong as much as how they go right. So tell us about some of your favorite cloud migration fails.

Well, without going into too many details, of course, we want to maintain privacy for people.

No, change the names to protect the guilty.

Yes.

And the innocent, right? I think the first one that is in general a bad change approach is the big bang, right?

Mm-hmm.

Where everybody thinks they're going to do this massive, complete cloud migration on this really unrealistic timeline because the executive board or the CEO said, no, we're gonna have to have it done by this date because we need to exit out of data centers. I think that's one of my favorites. And it just, It seems to happen everywhere. I don't know what you've seen, Anton, but that's my sort of take on it. And then inevitably, you know, somebody— I used to love those because somebody would tell me the date that some consultant told them to, you know, it was going to take, or that somebody arbitrarily set. And I'd say, well, you can say it. That doesn't mean it's going to happen.

Does it also drag people towards lift and shift, correct? Because from what I've heard, if you go date-first, then not changing anything in your security stack, in your application stack, in your networking is— sounds like not just a good idea, it's the only way, right? If you have to do it by X date, you're not refactoring, you're not re-engineering, you're not re-architecting, right?

Absolutely. But even worse, okay, let's be honest here, you're not really always going to be able to refactor everything. That's just, you know, it's not cost-effective. It's not— you may have a monolith, and there's nothing inherently bad with a monolith, but it doesn't always work very well in cloud. But let's get realistic. What really happens is that people cut a lot of corners. They don't just do lift and shift, they do messy lift and shift where it's just a train wreck. And they cut corners on things like security groups and policies and things like that, or policy as code. And then you've got an even bigger problem.

Interesting. So this reminds me, recently I got into a minor Twitter fight over Does cloud inherently reduce technical debt? And the argument—

Minor Twitter fight. That was a big one.

And I missed it.

It was very gentle.

It was a good one. It was a good one.

It was a good one, yes. But the idea is that the original idea was that cloud reduces technical debt because cloud forces you to change with the cloud. But somebody else brought up the point that Michel just brought, namely that if you do things in a hurry, you generate technical debt.

Well, sure, if you do things the wrong way, you get the wrong result. That's not rocket surgery.

You can probably get a PhD because if you're that smart with that wisdom, because I don't think people know that.

Well, it's, you know, what bothers me is that people forget. They think that this is just a technical implementation and a cloud transformation is an organizational change issue. I'm going to avoid the term change management because that can become confusing and people think about change management systems. So we'll talk about change leadership. And I feel what I've often seen is that you don't have change leadership, and they just dump it. Organizations won't look at the human problem of change, of nobody likes big bangs, right? Really, the statistics and the literature shows that the best change efforts are in organizations that support continuous change, where you have a combination of emergent change, and where you have process fairness and communication. And that idea from the original HBR article about fair process, right, where you communicate, you, you use the 3 E's of engagement, explanation, and expectation clarity. But, you know, I don't want to get into organizational science here. But really, it's a social problem in a lot of ways. And when you try to do this big bang with unrealistic timelines, you will get people who will dig their heels in because they feel like you didn't ask them. And now not only are they going to cut corners and be sloppy because they don't think you care about them, and they don't think you asked for their opinion, they're just going You know, it's going to get really ugly.

Doesn't Tim have a line about family counseling that is like super appropriate now?

Yeah, yeah, yeah.

So, we pretty much should just use it because we used it only maybe 10, 20 times in the previous episodes.

This is my line, listeners. You've probably heard this before. It sounds, Michelle, like what you're saying is CISOs preparing for a cloud migration don't need a piece of technology. They need a family counselor.

Paul Gibbons has a great book on change, The Science of Organizational Change. And I think that's accurate. When, you know, I recently did a literature review on this because it's fascinating to see how the human element can just destroy your DevOps transformation, your cloud migration, any of those transformational initiatives that you're trying to make happen and you need to make happen. They tell you all the time you need to make sure you have the right resources, the right people who have the skill set to make that happen. But I assert you also need change leaders.

Hmm.

When you're interviewing and you're trying to find the right people who are change agents, you should be asking them questions like, how do you make this happen? What do you think of servant leadership? I mean, I hate to go down this rabbit hole, but I feel like it's accurate because I've seen this happen way too many times. You know, it's the same thing about, you know, where they, you know, they mandate Kubernetes. No, we're going to do everything microservices. And you have monoliths, you have teams that aren't set up to really collaborate well, and that doesn't work very well. Cloud native does not work very well unless you have collaboration and cooperation and the ability to have that cross-sector communication to operationalize it in a proper way.

So, is this one of those cases where we've got leaders in an org who are in an org because they like technology and they've succeeded at the technology side of things, and all of a sudden they've got to do the opposite, which is the people side of things, and that's causing them to come up short?

Sure, sure.

Yeah.

So, wait, I'm somewhat confused by one point. So, if you push a hard date, and if you go fast, you don't change enough. And that's wrong because you copy data centers to the cloud, you end up with increasing levels of, you know, cost goes up and all that. But you're also saying that changing too much is also bad. Because if I say, I'm going to re-architect everything, I'm going to rewrite, I'm going to go to Kubernetes, I'm going to go to everything modern. If not changing enough is bad, how can changing too much be bad? I'm lost.

You need the Goldilocks of change.

Oh, thank you, Tim. I really need that wisdom.

Well, that's— I don't mind because it's really, I sort of see a cloud transformation in some ways like a Kobayashi Maru, and you have to hack the simulator, right? You have to, in a way. And I've been frustrated, but I think organization, you have to identify, is your organization cloud-friendly? What does that mean, right?

Wait a minute, cloud-native, cloud-focused, I want that. Cloud-friendly.

What does it mean?

It means that the people in your organization, in your technology organization, and even the people using your tech, right? They have to be amenable to a certain level of change. It's that idea, that Nassim Taleb idea of antifragile, right?

Mm-hmm.

Is your organization— does it embrace continuous change? Is it antifragile? Because if it's brittle at all and resistant to change, how are you going to do a cloud migration? And especially a big bang cloud migration. I think that's an important aspect to evaluate and assess when you're getting ready for this kind of effort and why the people element is so important.

Now, that's really interesting.

So, that makes sense. And I think to me, it's kind of interesting from the actionability point of view. It's not something philosophical, like, be more like cloud native. Like, I don't know what it is if I'm a bank in Kansas. But this to me is a lot more clear. So, let me try to shift gears a tiny bit from kind of these overall migration challenges to more process failures? Because the idea is that, yes, people scope it wrong. People sometimes go with security that's copied from on-premise. But what are your favorite cloud security process failures while migrating, if it makes sense?

Yeah, I think where security ends up being a side-channel activity, that's one of my favorites. And it's because— and you're going to get mad at me because I'm going to say it's a human problem again— where the security people got left out way at the end, right? You bolted— you tried to bolt it on way at the end. We're going to run, run, run ahead. And we're not going to bother to really have a conversation with the security people because they didn't think about moving it upstream. Right. And yes, I'm avoiding the term shift left. I'm using the term moving upstream because it's, to me, that's a more understandable way of looking at the process. Right. But every time, security gets left behind. And so, you get a security team that digs its heels in because they feel like they weren't consulted and included at the beginning. And they also, you may not have people who understand not only just modern software development processes, maybe they're very compliance heavy, maybe they are low on a DevOps security maturity level where the security team doesn't have a lot automated and in with the development pipeline. But then they also, you have a lot of security controls, traditional legacy ones that are on-premise. And those aren't even going to work. And you haven't done any vendor scoring or any discussions about how you're going to get new controls or use the integrated controls in the cloud provider. You haven't had that discussion at all. And now you're like, we're ready. And security goes, whoa, Nelly, no, we're not. And guess what? You have to be PCI DSS compliant. So, bah, bah, bah, you failed.

I really like that answer, in particular because you talked about on-premise security controls not working in cloud. Could we just spend a second talking about some of those? What fails when you try to take it to the cloud?

Yeah, firewalls.

What are your favorite non-working controls? What are the non-portable controls? That's a good one, right?

Okay, firewalls and WAFs, right? Where you try to use— where somebody might say, I've talked to people and they go, well, wait, that vendor has a virtual version of the firewall.

Ah, so 2014.

Hello, 1998 called. They want their firewall back.

Actually, you know, it's funny, that takes us all the way back to episode 3 of the podcast where we agreed that firewalls could live out their lives in little cloud buckets and that would be enough.

You know, I'm a former firewall engineer and administrator, but you learn pretty quickly that in a virtualized environment and especially in cloud, there are native features, native security features, and that's why it's so important to engage You know, security people, security experts really early on. Invite them to the party.

So, wait a minute. Sorry for interrupting on the firewall side. Firewall is obviously a big passion of people still. I got into a Twitter fight. Yes, sorry.

You just want to bring up all your Twitter fights, right?

So, I got into a Twitter fight over that very topic. I've expressed the same view, is that dragging your firewall appliances to cloud in the form of virtual machines is stupid. You have cloud-native controls from Google Cloud, from other clouds that do The gist of the strongest pushback I got was that there was a particular feature or 2 or 3 in their little Cisco, Check Point, Palo Alto, whatever, name your favorite brand, that they got to have for some reason. And that Google, in their mind, stupidly didn't build this feature into the firewall stack in the cloud. So they have to drag the appliances. I, not being a firewall person, couldn't really answer it. And I couldn't argue about the specific features. Can we dedicate 3 minutes to this topic? 2 minutes? 1 minute?

Yeah, sure. First of all, yes, all next-generation firewalls, they have a lot of different features that you're not going to have, right? But is it really about the exact 1-to-1 match, or can you find ways to achieve that in different ways? Can you use different techniques? And that's why you have the discussion way early. And my favorite collaboration tool, and you're probably going to laugh about this, I recommend it all the time. A control mapping exercise. I go way down and take the control mapping matrix. Like, you can use the prioritized approach one from PCI DSS. You can use the Cloud Security Alliance cloud control matrix. And I add a new column that says technical control. And I sit down and then I negotiate. Okay, here's the actual requirement. How could we do that? So it's more of like you step back and you do a yes and. Instead of a no, no, no. Now, that takes— it takes sophisticated people or people with certain, you know, layer 8 skills who want to do that, you know, but you're not going to get anywhere unless you do have those conversations. And that's why it's so important to start early with your security team. The result, if you don't do that, is a lot of latency. You know, I've been in conversations where, you know, a team can't figure out why they're getting such latency and they're like, no, no, no, we have to use this different service in cloud, or we have to use this, you know, we have to re-architect it this way. And I went, wait a second, I'm looking at your diagram and you have a virtualized version of a firewall here. Do you think that might be the problem with your latency? Have you tested it without that in line? Have you tried to use a more cloud-native feature, something that's native to the cloud provider? You don't want to be there in the 11th hour or where you suddenly have— I've been in environments that during the height of COVID when they were getting hammered a lot in traffic, That's when they were trying to figure it out. And I think that's a terrible time to do that, right? Because let's face it, cloud, the benefit of cloud is to get you to churn your applications, to get more of it out there, to be able to scale, you know, horizontally really quickly. So, you need velocity, you need coverage. It's not about perfect coverage, it's about maximizing your ability to provide that coverage. I think that's, it's a reset in the way we think about the way we deliver services. They're more fungible and that they're more disposable, right? The nature of cloud.

So I think this is an awesome example of a lift and shift technical failure. The firewall that introduces latency, the failure to understand what's going on there. I think lift and shift is almost always wrong. What do you think about lift and shift? I— and listeners, Michelle is like rocking her head back and forth, taking her time to think about the words here. Anton is looking confused. What do we think about lift and shift?

I'm an ex-architect.

You're an ex-architect.

Right, I'm an ex-architect. So, I was an architect for— an enterprise architect, security architect for many years. So, I'm going to give you the typical architect answer, which is, it depends.

Oh.

And I think that's why That's why unrealistic timelines bother me, right? It certainly bothers an architect where they're all going, no, no, no, we have to build our visions. We have to draw this out. We have to take time to look at it and do gap analyses, right? I'm not sure we can get crazy in that, but I think that's why it's so important to have those conversations, right? And to make sure you have the right stakeholders involved when you set those timelines and when you design things. For example, what's the business value in refactoring a complete monolith? There are plenty of organizations that still have mainframes. Why? Because mainframes just, they work, they're fast, and to take yourself off a mainframe is really expensive. Is the juice worth the squeeze? I think that's what you have to ask yourself.

But so wait, my original thinking on this was that lift and shift is not always wrong because as long as it's a step one, not step only, Because if you're gradually moving, that's actually acceptable step 1. But when I brought this up in some other, not Twitter fight, some other conversation, people said, yes, but people say it's gonna be step 1, but then 3 years later, they've never taken step 2. So how it connects to your point about timelines. So how do you not get stuck in this step 1 being step only?

Change appetite. Remember how I said the thing about continuous change in cloud? Is your organization cloud-friendly? And when I say cloud-friendly, that is not a technical term. technology assessment, right? Cloud-friendly is how willing is your organization to accept continuous change? Because really, that's what embracing cloud means. You are now embracing the notion that you don't have that idea, you know, you don't have pets anymore. You're not going to go, you know, keep it forever. It's not going to stay up for 365 days. You have embraced the idea of snowflakes.

Mm-hmm.

and continuous change. As long as you can get your organization there and you can set these milestones through OKRs and everything, then it's acceptable. And that because you should do that, right? You can't do everything at once. It's just not possible. Even if you were refactoring and even if your business was willing to embrace a complete refactoring of everything, you're not going to be able to do it all. You could hire an army. I don't think anybody can afford that. You know, you're not even gonna find the resources to do that. So I think that's acceptable as long as the context is right. Does that make sense?

It does, but you're kind of correlating success in this with change appetite. And if they ain't got any change appetite, then you're saying they won't have success?

I think that a company's success might be limited. I think that's fair to say. For example, if an organization isn't DevOps, if they don't have pipelines and automation, do you think they'll be very successful with cloud?

No, for sure. It's very clear. But so, wait. So, okay, fine. Let's imagine this situation. You're an architect. You meet somebody who is telling you about security cloud migration, and they say, yes, as a first phase, we're going to copy most of our systems. We're going to copy most of our security controls. We did some assessment. We know where to put the duct tape. And then we'll move on, and then we'll build this plan. So, you say, okay, fine, as long as it's step one. In 2 years, you talk to them, and they're in the same exact spot. What do you tell them then? You open your mouth and say what?

Well, honestly, I don't think I would have to tell them anything because you and I both know that if you do lift and shift and you leave it that way for too long, get a wheelbarrow filled with cash because it's a consumption-based cost model. Yes, they did. And it's not going to work out well for you, right? You're going to have too much compute that you use. You're going to have a lot of inefficiency, things running too long. You will have overspent. And the bottom line, your books will tell you, your finances will, you're probably, your CFO is gonna say, what did you do?

Okay, but they're there in a year and they should move, but they're not moving. So you're saying architect, it's not architect's place to give advice in this, in this case, they should go talk to a lawyer or to a CFO. A CFO would tell them—

No, no, no, that's not what I'm saying. I'm saying that I think an incentive is financial. I think an incentive could be to go into the billing console, start pulling numbers, and demonstrating inefficiencies. I mean, I think that's a compelling case, right? You've done lift and shift. You think it's okay. Look at the finances, how your— what your spend is, what your cloud spend is. I would also assert that lift and shift is probably not as secure.

Right.

Yes.

I think if you're not using the native elements of cloud, the security features and capabilities of cloud, if you haven't identified those, then You're likely not going to be as secure because you're, you know, let's say you're— I've seen this before— you're backhauling logs or you're backhauling some packet captures sporadically to maybe an on-premise DLP or IDS. You're— and it's not contextualized. You're probably— it's probably not as secure and you're not seeing things that you need to see as quickly. So I think I would be analyzing your security events and incidents as well. That would also be a good driver to demonstrate demonstrate that it probably isn't the best idea to just stay at lift and shift.

Hmm. And I like that you suggested the cloud-native security tools there. Have you thought much about the problem of cloud-particular native security tools versus third-party multi-cloud security tools?

Ooh, can of worms opening.

I know, and we've only got a few minutes left to open that can of worms, but I just thought I might pop the top and see what comes out.

Like we talked about how there might be levels to a cloud migration instead of a big bang, depending on how cloud-friendly you are. This is another reason to sort of involve your security team and have discussions about multi-cloud, because I think people think that multi-cloud is like super great at first. And I would tend to agree with you in an abstract and theoretically, you're not going to do multi-cloud probably right out of the gate. Why? Because you've just magnified your complexity times probably 3 or 4, however many you want to have, right? So maybe you don't want to jump to that multi-cloud security tool right away. Maybe you decide, oh, I'm going to use just the cloud features at first. And then as I mature, as my consumption of cloud matures, then I start looking at enhancements and additional tools as I adopt you know, more features in cloud, or I do something more, you know, I adopt some abstraction layers, maybe a portal, something like that. I mean, I think it's in, you need to consider consumable bites of the adoption, right?

Uh-huh.

I don't think you're going to jump in right away to a multi-cloud security tool like that. I don't know that that's really practical, and it might be too complicated, and then just, You get cognitive overload, you get change exhaustion. I've read that there's actually a limit to the amount of change that people can take, and it's in units. There are measurement units. So, you have to consider that. How much can—

What's the measurement unit for change?

It's a really obscure paper. I could certainly send it to you, but it's per person. There's a certain number of change units that each person can tolerate.

Wow.

I know.

Yeah, I want that.

Somebody did this in a study. It was great.

Listeners, we'll have that in the show notes for you if you really are interested. in that one, because I am.

It's pretty nerdy if you're into change literature. I, I recommend it.

Michelle, I can promise you one thing about our listeners. They're all nerds.

Well, great.

So with that, and with having insulted our entire listener base, uh, it's time to ask our traditional closing questions. First, aside from the paper that quantizes change, do you have recommended reading? And then 2, do you have a weird tip for improving people's cloud migration security outcomes.

And by the way, we're going to put Dune, the book, in the resources anyway. So don't say Dune. Yeah, you're not skipping Dune.

That's a given. Okay. So you're going to— I'm going to make this a theme because again, I think any kind of digital transformation is a social issue. It's a community issue of your organization. And so I think anything you can read in that sense, Paul Gibbons has some great books on change leadership I highly recommend. The traditional servant leadership books, you know, if you haven't, Greenleaf is the father of servant leadership. He actually came from a technical background. There you go. And he discovered that people were important to deal with.

What a shocking discovery.

I know, right?

No way, right?

Any of the DevOps, like State of DevOps reports, all of this talks about the work that DORA does is actually portable and usable in the realm of cloud migrations, right? Because it talks about these issues of automation of culture, people, process, technology, and culture. And then there's the Accelerate book, right? I think this is all good recommended reading, but the more you can read about change and change leadership and understand what you're in for and how you're going to connect with your community of technologists to have them feel engaged with the effort, to feel like they have input, then you have— you've created a collaborative effort and you're more likely to be successful.

That's an awesome answer. And then does that also overlap with your tip for improving these migrations? Create a collaborative effort?

Yeah, I know I sound like, you know, we're all going to hold hands and sing Kumbaya here, right? But that's really what a digital transformation is about. And most importantly, that's how security can be the most successful because there are way too many tendrils in cloud that security is very complicated in cloud, you know, and then you've got the additional factor of you will still have some on-premise stuff. You'll have developers and everybody, you know, accessing from their environment, you know, which may be an office, it may be remote. Now you've got a huge amount of complexity. If you don't know how to talk to people and if you don't know how to create relationship and maintain it, you're in trouble.

Well, Michelle, I couldn't think of a more fitting set of answers for somebody who identifies herself as a security Bene Gesserit. I think this is just perfect. I mean, making digital transformation a people issue is the most Bene Gesserit thing I've heard all day, maybe all year. So, Michelle, thank you so much for joining us on the show today.

No, it was my pleasure. It was really a joy being here and talking about this. Thank you.

And now we are at time. Thank you very much for listening and of course for subscribing. You can find this podcast at Google Podcasts, Apple Podcasts, Spotify, or wherever else you get your podcasts. Also, you can find us at our website, cloud.withgoogle.com/cloudsecurity/podcast. Please subscribe so that you don't miss episodes. You can follow us on Twitter, twitter.com/cloudsecpodcast. Your hosts are also on Twitter @Anton_Chuvakin and @_TimPico. Tweet at us, email us, argue with us, and if you like or hate what you hear, We can invite you to the next episode. See you on the next Cloud Security Podcast episode.

Security.