EP210 Cloud Security Surprises: Real Stories, Real Lessons, Real "Oh No!" Moments

Secure running.

Hi there. Welcome to the Cloud Security Podcast by Google. Thanks for joining us today. Your hosts, as ever, are myself, Tim Peacock, the Senior PM for Risk Analytics in Google SecOps, and of course, Anton Chuvakin, a reformed analyst and senior staff in Google Cloud's Office of the CISO. You can find and subscribe to our podcast wherever you get your shows, as well as at our website, cloud.google.com/podcasts. If you enjoy our content and want it delivered to you piping hot every Monday, please do hit that subscribe button. You can follow the show, argue with us and the rest of our Cloud Security Podcast listeners all on the Google Cloud Security Community page. Anton, we are talking about failures and not just my own failure as a podcast host this morning.

I think you're exaggerating that, but I think we are talking about lessons from more cloud security failures than mistakes. But here's the thing. I think that people have made enough mistakes to be able to sort of see patterns and see traces of some deeper truth under the mistakes. And I feel like this episode isn't just like a standard, here are the mistakes other people made, here are the mistakes you probably will make, but it's deeper than that.

It is deeper than that. And I think at times it verges into something more along the lines of human nature and the nature of human interaction with technology and organizations. In that way, it actually— you might not see this comment coming. It reminds me of Kelly's episode on chaos engineering because so much of what she was getting at is human nature and human failure in technology.

Yes. And I think that's an excellent example because I felt like while we are not trying to push the technology aside and say, oh, those pesky humans, it's their fault, but ultimately that part comes up every time if you discuss adoption of new technology, whether it's cloud or GenAI or some random future tech. I think that part is— maybe it doesn't repeat the mistakes, but it rhymes.

Yeah. I mean, I think the critical bit is adoption of technology doesn't happen in a vacuum. It's done by human beings, and human beings haven't changed. And so maybe with that weird philosophical preamble, let's turn things over to today's guest. Today we are joined by Or Brockman, a cybersecurity strategic customer engineer in Google Cloud's Solutions Consulting. We're really excited to have Or join us. Welcome to the show. I want to start off talking about engagements that have gone wrong. I think, you know, you've done so many customer engagements over the years and you've got all these good stories. I was hoping you could tell us about one where you like lifted the hood up off the car and just went, oh my God, that's what's under there. Holy cow, what's going on here?

Yeah, so interestingly enough, these mistakes are repetitive and there's one that I see all the time. So one story that particularly, I remember pretty well, was a bank in Europe, and the CISO was contacting my team, the Security Professional Services team, after they experienced a security breach. And the request was pretty simple. We just need a security posture review from your team. And usually when we get these requests, we do 2 things. So firstly, we interview the teams. We try to understand how they operate and how they cooperate and how they think. But secondly, we log into their environments, enter their dashboards, and we just snoop around. We check their folders, their projects, their assets, their permissions. Basically, we try to leave no stone unturned. And that's when, you know, we see things, right?

Uh-oh.

So yeah, exactly. And during our interviews, the CISO, you know, he was, he was always repeating this mantra: in my bank, security is Everyone's responsibility. Now, I don't know. I feel a bit ambivalent when I hear these statements because you know we do believe it should be in everyone's hearts and minds. But you know sometimes behind it there is missing accountability and missing ownership of security protocols, and that's exactly what happened to this bank under the hood. So what happened in this case was that. the security team put an organizational policy in place to disable the creation of publicly accessible buckets. That's great. We love that, right? But later on, the landing zone administration team, they had to do some debugging and obviously they disabled this policy temporarily, of course, you know, famous last words. But meanwhile, the application team, they were busy deploying a Terraform code that they found online. It was super long. They never read through it. And obviously, as you might have guessed it, this code basically created a publicly accessible bucket which later contained sensitive customer data. Now, luckily enough, they had the Security Command Center which alerted. There was an—

Oh, no way.

Yes, it actually created an alert and open bucket here, please. Take care. However, this is really, you know, this is where things went really sideways. The SecOps team reviewed the alert, and they assumed that the application team was on top of it.

Huh.

So no one took action, except obviously for a group of hackers who just, you know, stumbled upon that data. And well, let's just say they didn't keep it to themselves, right? It ended up in the dark web.

Ah, one of those. So let me interrupt very briefly. Because I think that security is everyone's responsibility has sort of two sides, maybe the light side and the dark side. And the dark side is the cliché side when it's like, assume that everybody would care, everybody would do things. But I also see that the dark side of like, that means it's nobody's, is kind of interesting. It's a little bit like a random company pronounces that security is the first priority for them. And everybody knows they're lying because very clearly that staying in business is the first priority for the business, right? I mean, of course, there are some situations when security is a precondition for that. But it's kind of a similar cliché where I say, oh, it's everyone's responsibility. And moreover, it's number one for all of us. And it's like, no, you're clearly saying it in bad faith, right? So that's the source of nervousness is that if I hear somebody lie, I start thinking, What else are they lying?

Yeah, I also like that story because it, well, 2 things about it. One, it's always nice to hear that my old team found something, but 2, it's a really good illustration of it's not enough to buy the tool. You actually have to use the tool and look at its output.

Absolutely, absolutely. You know, and it really, security should be in everyone's hearts and minds, but still that does not mean you don't need, you must have crystal clear ownership, you know, accountability, inter-teams communication, Otherwise, you end up in a situation where security is everyone's responsibility suddenly becomes security is no one's responsibility. And that's exactly what happened to this bank.

Yeah. And it feels like also the turn— I'm going to try to make a metaphor, very uncooked metaphor. So, it feels like this particular ditch is very easy to get into. It's almost like you're driving on a very slippery road and whoops, you're in a ditch. And the ditch is Nobody's responsible.

Exactly. Exactly.

No, Tim's not happy. I mean, we're on video, so you can see him kind of making a slight cringe face. Okay, fine. Make a better one. But the point is that I can see this good intent, but I also see frequent bad results.

Yes.

I wish I had the ability to make as much side-eye as my corgi puppy does. He has the best side-eye. So listeners, viewers, I'll leave you with that thought of a corgi puppy right now. And maybe Anton, you can ask the next question.

Okay. And I think the next question, since we are on the topic of mistakes, is not It's not just the bad mistakes, hot mistakes, fun mistakes, but it's the mistakes that actually you keep seeing the most often, like the common mistakes when you kind of like see somebody do this and you say, no, no, not this one again. So anything that's like super repetitive. I feel like there's value in this to listeners, to their cloud teams, to their bosses, to their bosses' bosses sometimes, because if everybody who Orr deals with makes the mistake, maybe they do too. And they can maybe fix it before it becomes a disaster.

Okay, so here's a super common mistake. And, you know, it's lesson learned. Buying tools cannot compensate for the lack of well-defined processes. Okay? It cannot. And I understand it because, you know, it's a shiny new tool. And we like buying things. And we like, we're all like tech geeks, we like exploring new technologies, and it's fun. But you know what? Buying that tool cannot compensate for the lack of well-defined processes. And I have a really good example for that. We had a customer, and that was a vehicle manufacturer, and they were facing lots of problems, right? They had compliance issues, they had misconfigurations, and their solution to it, or what they believed it to be, better said, was purchasing a new CNAPP tool. cloud-native application protection platform. And then they contacted us, my team, the security PSO team, and they asked for our help to architect that new product. And we got to work. And, you know, once again, we were snooping around their environment. And what we saw was just disaster. It was terrible. We saw open ports, we saw publicly accessible storage buckets, we saw public IP addresses on so many hosts. You know, all of those big no-nos. And all of these were anomalies. And these were not the standard policies. These were just numerous security exceptions. So, we started interviewing the team to understand, and we explicitly asked them, so, how is an exception request made, and who approves it, and based on what?

Yeah.

And how is it documented? And you know what, most importantly, when does this get revisited and revised potentially? And their answer was that there is no formal process for requesting exceptions. So, it was kind of like a situation where teams were making changes without any oversight or any accountability. And really, that discovery was just a turning point in the entire engagement because Suddenly, even the customer realized that this new cool technology, this cool new product, the CNAPP tool, it's not the problem nor the solution. And their real challenge was the lack of well-defined processes. And in this case, it was the security exception process.

So, I'm going to act really weird right now. And I'm going to act really weird—

Just right now?

Just right now. Yeah. This is going to be timeboxed to about maybe 5 minutes. This is kind of my favorite example, and I've been giving similar examples a long time ago, since the days of ITGRC and people buying SIEM and not building a process. And I have this tendency, if I make the same argument many times, at some point, I have to second-guess myself. And with this particular case, even though I usually almost always serve on the same side of the argument that Oriznon, I want to try to jump on the opposite side. Somebody brought the following counterargument to me, that if you have some kind of a tool that requires heavy process, it is our responsibility as a startup founder, because startup founder was giving this argument, to build a tool that is so much better that it compensates for process deficiencies.

Huh.

So, he said, Anton, you're kind of right, but you're also wrong. If we innovate hard enough and we think hard enough and we build hard enough, then our new generation tool actually covers what used to be paper process in the old world. So, your argument is not valid. I was briefly speechless. Briefly, very briefly. And I don't recall what I responded. It was like last year and stuff. But it stuck in my head. that it's sort of not insane to think that a sufficiently advanced tool can, in fact, cover, or can probably cover for the deficiencies in process. What is our collective take on that? Maybe we should ask interview Tim on that as well. He built plenty of tools.

So, in my personal opinion, even the most innovative, cutting-edge AI, machine learning, and all these fancy acronyms tools, You know, if you don't have clear communication, defined roles, lack of strategic direction, these tools will not be able to compensate. And we usually turn to these tools because we're scared of dealing with the real problem. And it's much more complex to deal with the real problem. And it's much more boring maybe to sit and document your workflows and your bottlenecks and figure out what's going wrong. And it's much fancier and nicer and more exciting to buy a new AI machine learning UBA-based tool. So I think we just turned to that because it's exciting and it's fun, but it won't cut it. You know, it won't. The tools cannot compensate for the lack of well-defined processes. And I don't think you can ever convince me otherwise.

I think that it's a reasonable take to double down. I would double down as well. I probably did double down when somebody asked me because I've seen too many examples when the process exists in areas that cannot possibly be covered by a tool, like human-to-human communication.

Right.

But wouldn't this same startup founder, at this point kind of metaphorical, can say, well, if I have a better ticketing, a better API calling, a better approval process that's automated, and LLM looks at past approvals and blah, blah, blah, it would cover for these weaknesses? I think that If you double down, they may also double down and say, nah, yeah, we can automate some of that.

I think I'm going to be really boring and take a leaf out of Anton's book and say, on one hand, I could see where that startup founder is right. And on the other hand, I can see where Aur is right. I totally agree that it is human nature to think, oh, I have a problem. I could wave my magic wand and hopefully this problem goes away because I spent a lot of money And wow, what a whiz I am for doing that. And on the other hand, it's also human nature to get deeply entrenched in a process which no longer serves a function because you have people that you like who are doing it, and you probably could get rid of them with a properly functioning tool. So I can see both sides here, and I guess, listeners, I'm sorry to not take a stronger stance. I'll take a Catherine move and say both are right.

But wait, that's actually a reasonable take because—

Ah, yes, reasonable take.

I win. We switched from it is stupid to think tool fixes process weaknesses to who knows, maybe. I think that's a position shift already.

Who knows, maybe. I'll take it.

Maybe it can alleviate the symptoms of, you know, of like not defining the process as well. So it can alleviate the bad, bad symptoms by innovative technology, but the root cause is still there, unfortunately. And it will surface again one day.

I do want to pick at this though, because Anton has probably been giving this tool versus process advice since I was in, I don't know, elementary school.

No, no, don't go there. I'm not that old.

How do you get people to really— because in your role, you must deal with this all the time. So, how do you get people to really think hard about it and do it right? Like, how do you change minds on this one, Art?

So, I think that eventually, when you treat security as a burden, when you look at these as, you know, just a checklist, when you're looking at it as something that is, you know, time-consuming and resource-intensive and ineffective, you're doing it wrong. It shouldn't be that complicated. And there are so many good ways to achieve that. So, take, for example, threat modeling, right? So oftentimes we see customers that do understand the importance of doing threat modeling, but it's just that their approach to it is anything but simple. We would interview the teams and they would say, oh yeah, threat modeling, let me find that 80 pages long document we wrote about it once and never, no one ever reads it, right? And here's STRIDE and OWASP and PAS and all these different frameworks. Formats and standards, but eventually, because they treat it as a burden rather than a benefit, they just don't do it right, and they're missing the entire purpose of threat modeling. So how should they be doing it, and how should they be treating security in general and specifically threat modeling? I think it should be done intuitively. I think it should be kept. Simple. I think it should be like a ritual, just like, for example, if you were to imagine standing at a crosswalk, an extreme rush, and you're waiting for the light to change green and you see the cars passing by really fast, but you instinctively start assessing risks in your head. What will happen to me if I cross at a red light? What happens if the car doesn't see me? Right? So without even realizing, you're conducting a very important security principle of threat modeling. You're identifying potential threats, you're assessing them, and you're deciding on a course of action to mitigate them. So it should be kept simple. You know, one of the founders of this theory, Adam Shostak, once said, 4 questions: What are we working on? What can go wrong? What are we going to do about it? And did we do a good job? We don't need 8 pages long documentation. We don't need complex diagrams. We just need practicing these 4 questions, practicing this form of thinking, practicing a security mindset. And I think that will cut it at most times.

I think that TM, threat modeling, is particularly vulnerable to people writing a long, painful doc and never touching it. I think that I'm not an AppSec person. I keep telling everybody for the last 10 years that I don't do AppSec. I don't know it. But threat modeling is fascinating enough to me, so I delved into it. I read the same Adam Shostak book, which we'll feature in resources. But I feel like turning TM into compliance is one of the worst things you can do. Or turning anything into compliance is one of the worst things you can do.

Oh.

No, Anton, you had the Tim Dirks episode. You had your mind expanded.

Okay. It shrunk back. Sorry. Sorry, Tim.

But he's right. He's right. You know, it's time to forget about compliance-oriented control lists and shift focus to continuous and simple threat modeling. That's what I think.

Well, I totally agree with you that practice is really important for this stuff, and that if you can get into the habit of thinking a particular way, it makes a huge difference. For example, I make my puppy sit at every crosswalk.

Nice.

Yeah. He doesn't think about cars, though.

Yeah, so that he—

But so that he doesn't run out in the street. So, he always sits before crossing.

Practice makes perfection.

So, let us ask one more question before we start thinking about closing the show. Well, the episode, not the show, please.

Whoa.

So, what about turning mistakes into advice? And from all the mistakes you've seen people make, and from all the— across industries, across company sizes, So, if somebody, you know, picks on you, maybe not necessarily at 3 AM, sorry, Tim, no 3 AM points this time, what is the advice I would give, you would give, sorry, not I would give, every company to avoid the mistakes you've seen? What has the biggest impact on security posture? Sorry, I'm rambling this time, but you got the point, right?

I got it. So, I think that if you would ask me what's the number one, recommendation I would give, lesson learned, it is that any customer must start breaking down the silos in their teams. And we see that. I think that's the number one mistake we see.

Mm-hmm.

And within our customer base, we see silos between security teams and the development teams. It's just, you know, mind-blowing how often we see it. It's just that the security teams And the development teams operate in their own separate worlds, and they treat each other as a burden. They don't like each other. Security teams, they just feel like they're constantly playing catch-up. And developers, they just look at security teams as a roadblock to their progress. And these silos, they can be, you know, crucial. They lead to friction, to miscommunication, And eventually, they even lead to security vulnerabilities. So, number one advice I would say is that any organization, any customer must try to create a culture of shared understanding and trust between security teams and developer teams, infrastructure teams, DevOps teams, etc. They must think about how to encourage security teams to learn about the developer's role and vice versa. And there are so many ways, you know, like hackathons or trainings and joint problem solving and even just a fun night out.

Yeah.

So number one recommendation for all of the customers and all of the people listening out there, please break down your inter-team silos.

I think that is such good advice, and I've seen that become actionable with something as simple as like pizza and beers in the break room for the security team and the developers to get together.

There's nothing pizza and beer cannot solve.

Right?

Yeah, exactly.

Except maybe open ports on buckets.

Well, I guess they would never appear. They won't be closed by pizza and beer. But if you have pizza and beer, the next batch would never be open. But let me drill down into this a little bit because you've presented a really good second part of the break silos. Because at times, not in this case, I've seen people give this advice as kind of what I would call drive-by advice. They would drive by and say, break silos. And then they realized that there are 50 years of organizational history that gave birth to the silos. And some very intelligent consultant passing by and saying, break silos, and leaving won't do it.

Yeah.

I think we've covered the pizza and beer, lunches, other things. We've had this same problem even within security operations when, say, SOC versus developers versus Detection engineers, there are even silos within security. And sometimes, I've noticed that people appreciate the advice, but they really trip over the how. So, can we go through a few more ideas on top of having lunches, maybe having job rotations, maybe having pizza and beer? What else can work to actually break silos?

How about a cool hackathon where the security team and the dev team would have to cooperate in order to win the grand prize? And, you know, by developing something that is secure by nature or secure in all of the other formats that we know, how about doing some kind of a cross-training? How about, you know, there's all kinds of like champion programs that you can do. It's just mind-blowing to me that during our interviews, when we interview the customer teams, that's the first time they meet each other.

Oh, wow.

How could that be? Right?

That's really grim.

That is grim.

Yeah.

I know.

This really leads to so many security vulnerabilities. So people, people foremost.

Oh man. So we've, we've really put tools at the bottom of our stack of people, process, and technology today. Very sad day for tool vendors, but maybe a good day for human beings. Or I hate to do it because this has been a ton of fun despite my inability to do anything right today. Could you recommend reading for our listeners?

Anything. It can be a fiction book.

Yeah. Something so that people are not left empty-handed. Your blog, your Could be fiction, could be poetry. We've had people recommend Mary Shelley. We've had people all over the place.

I think everything is allowed except for my blog.

Yeah, Anton's blog, actually, you have to put a dollar in the swear jar.

Oh, that's a really good question. I'll have to think about it. But actually, I really like Stephanie Wong's LinkedIn posts where she's writing all about leaving your comfort zone and how that could feel sometimes as something that is scary, but But actually, the more you feel it, it means you're progressing. So, if I'll have to recommend something, I think it will be reading her blog posts. They're really good. They're LinkedIn posts.

I love that recommendation. And I think that's the same Stephanie Wong that's been on the show, isn't it?

Yes, that's the same.

Really?

Yes.

Yeah, yeah, yeah. She joined us in 2021.

She interviewed us on our show.

She did that for the AMA episode, but she was also on in 2021 when we had the, like, 8-guest extravaganza.

Wow.

Oh, wow. Wow.

I'm such a fan.

No, we're never doing that again. But, Or, we would be happy to have you back. And listeners, thank you so much for joining us today.

Thank you for having me.

And now we are at time. Thank you very much for listening, and of course, for subscribing. You can find this podcast at Google Podcasts, Apple Podcasts, Spotify, or wherever else you get your podcasts. Also, you can find us at our website, cloud.withgoogle.com/cloudsecurity/podcast. Please subscribe so that you don't miss episodes. You can follow us on Twitter, twitter.com/cloudsecpodcast. Your hosts are also on Twitter at Anton_Chuvaikin and _TimPico. Tweet at us, email us, argue with us, and if you like or hate what we hear, we can invite you to the next episode. See you on the next Cloud Security Podcast episode. Thank you, Ronnie.